How Does Policy Evaluation Shape New Flock Products?

Flock’s goal is to build a responsible public safety infrastructure; one that solves crime and shapes communities for the better, with privacy protections and civil liberties in mind. That mission requires deep intentionality and thoughtfulness behind the changes we make to our products. Each new feature, product launch, and new business vertical must, at the highest level, improve safety for communities and be built with responsible compliance and ethical oversight. 

Several years ago, after being influenced by Professor Barry Friedman’s book Unwarranted and the work of NYU’s Policing Project,  we created a structured, formal Policy Evaluation Process to decide what we build, how we build it, and what guardrails need to be in place before we ship anything new. 

This process has evolved over time and likely will continue to do so. Compliance and ethics at Flock are never static, but an evolving, improving project that the entire company, across Engineering, Product, Policy, Legal, and Strategy, is constantly engaged in.  

This blog walks through how our Product Evaluation Process works today, who is involved, and what it has meant for real product decisions.

Why does policy evaluation exist for Flock product changes?

The central purpose of Flock’s Product Evaluation is straightforward: before we launch any new product, partnership, integration, or major feature, we ask four gating questions:

• Is it legal?
• Is it aligned with our ethical creed and commitments to civil liberties?
• Is it compliant with rapidly evolving privacy and AI regulations?
• Is it secure for the communities and agencies that rely on us?
  ‍

To answer those questions consistently, we often perform a formal evaluation, particularly for:

• Any new product
• Any product change that substantively changes or adds to our current capabilities
• Any new partnership or integration that materially alters what Flock does or what our customers can do
  ‍

Minor updates, like aesthetic tweaks or UI refreshes, generally do not go through the full process. But if a change touches AI, data processing, highly regulated concepts, or meaningfully expands customer capabilities, it triggers the formal Policy Evaluation.

Over the past few years, this has evolved from a smaller, informal practice into a much more robust, company-wide discipline. We have expanded the feature set that goes through evaluation, added new stakeholders (including dedicated privacy and compliance counsel and a Chief Information Security Officer), and made sure product teams now proactively bring questions to the table.

Where does policy evaluation fit into the product lifecycle?

Flock’s multi-step product development process begins with customer discovery and R&D, and ends with go-to-market. But between those stages, policy evaluation is a critical hurdle in the decision whether to bring a product to market. 

For new products, the evaluation happens at the end of the Product Validation phase, once the team has researched the idea and is deciding whether this is truly something we want to build and bring to market. The evaluation typically happens as follows:

1. Product Manager submits project information

This includes:

• A description of the proposed product or feature
• How it changes what data is collected by Flock, what type of data, how data can be used and by whom, and how it can be shared compared to the status quo
• Whether the proposal uses AI or machine learning, and if so, how
• The target customer segments 
• Key contacts, confidentiality needs, security/compliance requirements, potential upside/market size, and desired launch timeline
  ‍

2. Policy and regulatory evaluation
   ‍

The Policy team’s section is designed to answer two gating questions:

(a) Is this legal? and (b) Is this proposal in line with our ethical creed?

They look at:

• Current law and regulation: How is this technology treated by regulation today? Is there a trend toward tighter regulation or prohibition in the future?
• Privacy rights: Could this product infringe on privacy or other civil liberties? If so, can those risks be mitigated, and how?
• Objectivity: Does the product capture objective evidence, as opposed to subjective or speculative inferences?
• Misuse and abuse: In what ways could this product be misused, and how likely is that misuse? Are there ways to minimize or eliminate those risks?
• Accuracy and impact: What is the potential for inaccurate results? What would the impact of an error be, and what safeguards can reduce both the frequency and the harm of inaccuracies?
  ‍

The result is a clear summary of any major policy risks and recommendations to consider before proceeding.

3. Privacy and data protection evaluation
   ‍

When applicable, Privacy Counsel may be pulled in to review how the product interacts with evolving U.S. state privacy laws and FTC enforcement expectations. They assess:

• Data scope and sensitivity: What categories of data are collected, processed, or stored? Which sensitivity level do they fall into — from public information to sensitive personal information and Criminal Justice Information (CJIS)?
• Purpose and legal basis: Why is this data needed? Is the processing necessary to achieve that purpose?
• Data minimization and retention: What is the minimum dataset required, and how long is it retained? What deletion policy applies?
• Consumer rights: How will access, deletion, correction, and opt-out requests be supported, including “Do Not Sell/Share” and “Limit Use of Sensitive Personal Information” rights where applicable?
• Third-party sharing: Which vendors or processors have access, and are proper data protection agreements and contractual clauses in place?
  ‍

As with Policy, this ends in a conclusion summarizing key risks and recommendations.

4. AI-specific evaluation
   ‍

When a product uses AI or machine learning, we may also assess it against AI accountability laws, anti-discrimination guidance, and sector-specific rules. We may look at questions like:

• AI purpose and function: What type of AI is being used (classification, prediction, generative, recommendation), and is it generative or deterministic?
• Training data: What datasets were used? Were synthetic or anonymized datasets used? Was bias testing performed pre-deployment?
• Fairness and bias: Has the system been evaluated for discriminatory impact across protected classes? How are bias risks monitored after deployment?
• Transparency and explainability: Can we explain system decisions in plain language? Is there a model card or equivalent documentation?
• Human oversight: Can human users review or override AI-driven outputs? Are there clear escalation paths for errors or harmful outcomes?
  ‍

Again, this culminates in a risk and recommendations summary.

5. Security and Trust & Safety evaluation
   ‍

Our Trust & Safety and Security teams are often pulled in to evaluate whether the product aligns with the FTC Safeguards Rule, NIST Cybersecurity Framework, SOC II Type 2 guidelines, and common breach notification laws. They would focus on questions like:

• Access controls: Who can access which data, and is access role-based and fully logged?
• Encryption and storage: Is data encrypted in transit and at rest? Where exactly is it stored?
• Security testing: Has the system undergone penetration testing or vulnerability scanning? Is there a documented incident response plan, and who owns it?
• Vendor security: Are security requirements built into vendor contracts? Do vendors provide recent SOC II, ISO 27001, or equivalent assurance?

Who makes the final decisions before Flock creates a new product or feature?

Flock’s Product Steering Committee makes the final decisions.  

Once the evaluations are complete, the teams flag any potential risks and recommended mitigations. After initial reviews, the product team and the evaluating teams align on adjusting requirements, adding safeguards, or refining features until they reach consensus on what the most responsible version of the product looks like.

When a decision needs executive attention, it is escalated to the Product Steering Committee, which is comprised of C-Suite decision-makers from Policy, Legal, and Product, plus a principal engineer who can weigh in on technical feasibility.

The Steering Committee has the authority to make final decisions based on the legal and policy recommendations.

What this looks like in practice: real-life product examples

The process is only meaningful if it tangibly shapes the products we ship. One concrete example indicative of how the Policy Evaluation has functioned at Flock is our Distress Detection feature on audio alerts.

Flock’s Audio Detection sets itself apart from traditional gunshot detection by adding additional audio signatures indicative of crimes or dangerous situations. Over the past few years, we have added audio signatures like screeching tires (indicative of illegal sideshows) and vehicle crash detection to the product. Last year, we also began scoping the possibility of layering in what we called Distress Detection, designed to detect prolonged screaming that might indicate a safety issue in a parking lot or outdoor space. 

Because this feature involves audio, it raised obvious and important questions around wiretap laws and expectations of privacy. 

The evaluation process looked like this:

• What data is involved: Product submitted documentation of what audio would be captured, at what decibel levels, and in what environments. This included provisions on how long the data would be stored/how it would be stored.
• Policy research: Legal and Privacy teams examined current regulations and case law on wiretap and privacy laws. They concluded that, at the decibel levels and outdoor contexts involved, there was no reasonable expectation of privacy under existing law.
• Product-level safeguards: To layer in additional safeguards for privacy protection and to prevent misuse, Privacy counsel recommended device-level controls over which devices can generate distress alerts, so agencies can avoid unwanted alerts in areas where loud but non-emergency noise is expected. It was also decided that the feature would be opt-in for agencies to enable; agencies must affirmatively choose to turn it on and configure it for their environment.

From policy on paper to safeguards in code

This work is ongoing. Laws change, public expectations evolve, and technology advances. Flock must remain nimble as the landscape of our industry evolves, and the Policy Evaluation is a part of that. We have a broad mandate – to build products that make communities safer – but that doesn’t mean we should always build every tool that might solve crime or enforce public safety. 

The Policy Evaluation process ensures that everything we ship is not only within the lines of the law but also maintains privacy protections for ordinary people, reduces the risk of misuse or disparate impact, and adds to a system that communities can trust over the long term.