Leaking Proprietary Data

Generative AI systems may inadvertently or maliciously disclose proprietary information through their outputs. This includes proprietary or confidential enterprise information. Such information can be integrated into the system from training datasets or integrated retrieval systems. These disclosures can occur unintentionally through regular system interactions or be deliberately elicited by adversarial queries.

Because of the opaque and probabilistic nature of LLMs, it is often difficult to guarantee that private information has not been memorized or can’t be extracted. Even systems that implement fine-tuning or retrieval-based augmentation (RAG) risk exposing source material unless protective mechanisms are enforced throughout the training and inference pipelines.

An organization can use proprietary data to train or fine-tune a model to address customer issues in detail. However, without proper protections the model may expose private information in a public application.