Harmful and Inappropriate Content

Generative models are capable of producing inappropriate content in various formats, including text, images, audio, and video. The consequences can range from minor user discomfort to serious harm, such as legal liability or psychological distress. This capability stems from the models’ exposure to broad, often unfiltered internet data during training.

Many models undergo alignment steps such as fine-tuning or reinforcement learning from human feedback to reduce unsafe behavior, but these methods are not foolproof. Inappropriate content may still appear, particularly in edge cases or under adversarial prompting.
Importantly, what qualifies as “harmful” or “inappropriate” must be defined in the context of the application and its intended users. For example, guidance on “how to shoplift” may be inappropriate in general-purpose chat, but useful to include (as a counterexample) in an educational or security setting, like how to prevent it. Definitions of harmful content should be co-developed with diverse stakeholders
to include legal experts, affected user groups, and domain specialists to ensure cultural and contextual alignment.

The context of the prompt also matters. A model that produces harmful output in response to an simple input (e.g., “how should I do laundry?”) presents a much higher severity and likelihood of harm than one triggered only by obvious boundary-pushing prompts (e.g., “what are some curse words?” or “how do I commit an illegal act?”).

- AIID Incident 1026: Multiple LLMs Allegedly Endorsed Suicide as a Viable Option During Non-Adversarial Mental Health Venting Session - Substack user @interruptingtea reports that during a non-adversarial venting session involving suicidal ideation, multiple large language models (Claude, GPT, and DeepSeek) responded in ways that allegedly normalized or endorsed suicide as a viable option.