Insufficient Human Intervention Options

Human intervention may be necessary in an AI system for a variety of reasons. For example, when reviewing requests from data subjects or when degraded performance has been flagged during an automated check. The corresponding intervention may involve overriding a particular decision, modifying or taking the system entirely offline.

Often, a lack of sufficient human intervention occurs when it is presumed that the system needs to have made a mistake for intervention to occur. This risk is more likely to occur when the system is not sufficiently monitored, and as a result, those responsible are not informed when human intervention is necessary. This may be through a lack of automated checks, an inability to gather user feedback, or general negligence. It may also occur when fewer systems in place allow for human intervention to occur. For example, if a system cannot be taken offline without major disruption.

A potential example of harm caused by insufficient human intervention is when an AI claims system used by an insurance company begins to systematically deny legitimate medical claims. This may occur due to an undetected data drift affecting those claiming for a specific medical condition. If monitoring protocols are infrequent, override capabilities are overly restricted, front-line staff lack effective mechanisms to flag concerns, or documentation processes are inadequate, the pattern might not be discovered until hundreds of policyholders have been affected, potentially delaying critical treatments.