Avoiding risks in the AI use in law enforcement activities

The European experience

In law enforcement, AI has been used in the EU since at least the mid-2010s, including for surveillance in public places during certain¹ events (e.g., the G20 summit in Hamburg). The limits of such interference with people's rights² have raised concerns, including among MEPs.³

In May 2024, the EU passed the AI Act, which legitimizes the possibility for already constant surveillance in real time and in any public space of more than 6,000 people who are wanted on a European warrant for crimes.

The Act classifies a number of AI systems used in law enforcement activities as “high- risk”, which implies increased requirements for such systems: to ensure automatic recording of events throughout the life cycle of an AI system; to apply a risk management system, including testing, etc. The Act also provides for the use of a risk management system, including testing, etc.

For example, the use of remote biometric identification of suspects in real time in public places (tracking by AI on surveillance cameras) is prohibited in law enforcement activities but is allowed with the prior authorization of a judicial body for some particularly dangerous crimes (terrorism, human trafficking, drug trafficking, etc.). The list of crimes is quite broad.

Law enforcement profiling of individuals in criminal investigations is also allowed, but human oversight of AI is required.

Thus, the AI use in criminal investigations is possible in the EU subject to certain requirements for high-risk AI systems. Thus, the Law recognizes the risks of discrimination and violation of fundamental rights in the use of AI systems (e.g., safety - not to fear arrest just because an AI camera on the street signaled a match with a criminal). To mitigate such risks, the Law introduces requirements for high-risk AI - data quality, risk analysis, human monitoring of work and decisions, and the ability to fine-tune AI systems.

Whether compliance with these requirements will be sufficient to avoid creating distortions is questionable. For example, if the system is trained even on representative data showing that, for example, thefts are committed by people from the same country or of the same gender, then risks of discrimination are created the AI system, trained on such data, may point to these people as suspects in the first place.

The EU has attempted to balance the risks and benefits of AI by imposing special requirements on high-risk AI for law⁴ enforcement. However, Austria, for example, considers these measures insufficient to guarantee the rights of citizens; in its opinion, law enforcement officers are given too much⁵ power.

It is worth noting that in 2018, the⁶ Council of Europe developed and adopted ethical principles for the AI use in judicial⁷ systems, which formed the basis for the regulation of this area in the EU. In particular, the list of principles includes requirements for the quality and security of systems, user awareness, and others.

The US experience

In the US, there is no AI regulation in law enforcement. This creates problems for the police in respecting the rights of suspects and complicates their defense, for example, in 2020, a man was arrested on suspicion of theft because facial recognition technology showed he matched a photo of a criminal, but he did not⁸ commit theft.

Vermont and Virginia bills (January 2024), categorize AI systems that significantly impact access to criminal justice as high risk. This means there are special requirements for risk management and for the data on which the AI is trained.

The approach of these states differs from that of the EU on two main issues:

1) It is not prohibited to the AI use, for example, to assess a person's risk of committing crime based on profiling alone.

2) No human oversight of AI is required, but it is required that those subject to AI-based decision-making be necessarily briefed on the use of such a system and its purpose, and, in Vermont, the way it works.

As a result, the transparency of AI systems, which affects the suspects' right to protection, is ensured by general requirements for high-risk AI (on the quality of the data used, risk management) as well as the suspect's right to obtain information about the operation of the AI.

China’s experience

There is no specific regulation of the AI use in law enforcement. However, in 2022, China's Supreme Court issued similar principles to the Council of Europe on the AI use in judicial⁹ activities. At the same time, China has laid down special principles such as:

– AI should not substitute for a judge in making decisions.

– The responsibility for the decision lies with the decision maker, not the AI.

– The right of the user (judge, defendant, etc.) to reject AI products when making judgments.

Russia’s experience

There is no special regulation of the AI use in criminal proceedings in Russia. At the same time, among the main functions of the Ministry of Internal Affairs of Russia is the task of introducing AI into the information and¹⁰ analytical system, however, there are no methodological recommendations on the use of these services in the public domain. The closed nature of such information makes it difficult to analyze and challenge decisions made using AI in the Ministry of Internal Affairs system, including the defense of suspects and defendants in criminal cases.

In 2023, a person was charged with murders based on the neural network's decision that he resembled the sketch of the criminal by 55% (the case against Alexander Tsvetkov) . Cases, the circumstances of the prosecution of which are like the case of Alexander Tsvetkov, are not available in open sources, but in the context of inaccessibility of data on the AI use in the system of the Ministry of Internal Affairs, this may mean the absence of not such cases, but their publicity. Taking into account the existing nature of the judicial and legal system (election of preventive measures, low proportion of acquittals), this aggravates the risks of prosecution of innocent people, and leads to the dilution of responsibility for the decisions taken during the investigation. The possibility of bringing charges because of a procedure with a non-transparent AI mechanism makes it difficult to establish who is responsible for the prosecution and defense of the accused.

It is possible in Russia:

1. Establish requirements for AI systems intended for use in crime investigation regarding the data used, including for training and testing, risk management, transparency, human control.

2. Provide legal guarantees that a decision generated with AI input cannot be made without human assistance. On the one hand, this means that a human is ultimately responsible for any decision, while on the other hand, holding them accountable is a matter of law enforcement.

From the monitoring issue No. 5, May 2024. Download the full issue (PDF) · issue page at the Gaidar Institute