Leaking Personal Data

Generative AI systems may inadvertently or maliciously disclose personal information through their outputs. This includes individual's personal data (e.g. PII) or information that is considered personally sensitive (e.g., health information). Such information can be integrated into the system from training datasets or integrated retrieval systems. These disclosures can occur unintentionally through regular system interactions or be deliberately elicited by adversarial queries.

Because of the opaque and probabilistic nature of LLMs, it is often difficult to guarantee that private information has not been memorized or can’t be extracted. Even systems that implement fine-tuning or retrieval-based augmentation (RAG) risk exposing source material unless protective mechanisms are enforced throughout the training and inference pipelines.

- AIID Incident 1070: Serviceaide AI Platform Implicated in Health Data Exposure Affecting 483,000 Catholic Health Patients