AI in healthcare services

The US experience

In June, California (USA) considered a¹ bill on utilization of AI in patient communication. Earlier in the US (2023-2024), initiates to regulate the utilization of AI in healthcare services were considered. The following approaches can be highlighted:

1) AI qualify as a practitioner:

− Eligible to prescribe drugs. Qualify AI as a practitioner if (1) authorized by the

State involved and (2) approved, cleared,² or authorized by FDA (Federal US Bill³

США ).

2) Limiting the use of AI:

− Prohibit the use of AI in making certain decisions regarding healthcare (together with insurance coverage and public assistance) solely based on the results⁴ generated by the AI (Georgia State Bill ).

Any such decision must be meaningfully considered by the individual. It is planned to set up rules for this purpose.

− Prohibit healthcare facilities from substituting independent evaluations by licensed patient care professionals for AI recommendations or decisions (State of⁵

Illinois). For example, if a nurse has identified procedures for a surgery patient's recovery, the facility may not order her to change them based on the

AI's recommendations.

Some States do not specifically regulate AI systems in healthcare services, but they are recognized as “high risk” (Vermont, Virginia,⁶ Colorado ). These 3 States require developers and users of high-risk AI systems to take measures to avoid algorithmic discrimination in human access to healthcare services, and users to notify about the utilization of AI in the provision of services. There is an obligation for the developers to educate users about the ability and methods of monitoring.

In California, June 2024 clarifies the obligation of healthcare facilities that the use of generative AI to create written or verbal communications about patients' health⁷ conditions :

− Inform that the message is generated by AI and that such message has been verified by the human service provider.

− Brief the patient how to contact the person providing the service.

In addition, Massachusetts is considering⁸ a Bill to use AI in mental health starting in 2023:

− Licensed professional in the field must be approved by the licensing authority for the use of AI.

− AI needs to be constantly monitored by a specialist.

− The patient provides written informed consent to receive treatment from the specialist who will utilize AI. Obtaining written consent from the patient is a prerequisite for any medical intervention.

However, when using AI, consent is required after being informed about how the AI.

The EU experience

The EU signed the AI Act into law in June⁹ 2024 recognizing usage of AI as high risk:

1) To ensure the safety of a medical¹⁰ device used, for example, for in vitro.

2) When authorities assess a person's eligibility to access public health services (e.g., through health insurance).

3) To assess and classify emergency calls, such as prioritizing referrals to emergency first aid services and obtaining medical assistance.

High-risk AI has special requirements including requirements to analyze risks, provide human control over AI, etc.

The EU approach is like that of the US states of Vermont, Virginia, and Colorado, but there are differences. The EU requires human control when using high-risk AI (in the above states there is no such requirement), however, there is no obligation to inform people about the AI use. This is due to a desire to remove any liability from healthcare consumers. Unlike Colorado, the EU does not have an established ability to require human review of an adverse AI decision. From technology development perspective, Colorado's approach has advantages: revised decisions provide feedback and can be used to improve AI performance.

The experience of China

In 2022, National Health Commission of the People's Republic of China stated that existing AI in the medical field lacks data and transparency of algorithm performance for¹¹ medical care. It is unclear how to determine liability for patient harm caused by AI. Therefore, China has set restrictions on the use of AI in¹² medicine:

1) A medical institution cannot utilize AI by impersonating a practitioner or substituting a practitioner qualified to provide diagnostic and treatment services in person (Art. 13).

2) Prescriptions for medicines must be written by the attending practitioner, the use of AI or other ways of automatic prescription writing is strictly prohibited (Art. 21).

Russia’s experience

Russia has 2 experimental legal modes for testing AI medical technologies and has adopted GOST R 59921.2-2021 "Artificial Intelligence Systems in Bedside Medicine".

However, there is no specific legal regulation of the use of AI systems in healthcare. Nevertheless, AI can be used as part of software in a medical device (Order of the Ministry of Health of Russia dated 06.06.2012 No. 4n). Such software is categorized as high-risk software with special conditions of use and licensing. The rules for the registration of such systems are defined by the Decree of the Government of the Russian Federation No. 1416 dated 27.12.2012.

To apply AI in medical practice in Russia, it is necessary to introduce risk identification and management systems, as well as to establish a requirement for human control over decisions made by AI. This will enable the use of AI in medicine, for example, the use of AI for diagnosis can reduce medical expenses by up to 50%. At the same time, human control will reduce the risks of errors in diagnosis or prescribing medication.

From the monitoring issue No. 6, June 2024. Download the full issue (PDF) · issue page at the Gaidar Institute