Malicious Use

Malicious Use refers to the exploitation of AI systems to cause harm to individuals or organizations. Organizations must recognize that AI's capabilities can be leveraged for various nefarious purposes, often by exploiting allowed uses at scale or by circumventing existing safeguards through "prohibited uses."

Common types of malicious AI use that pose direct risks to organizations include:

- Misinformation Campaigns: AI can generate and amplify false narratives at scale, manipulate public opinion, or impersonate legitimate news sources. For example, AI-generated false news articles or deepfake videos could spread false rumors about a company's financial health, damaging its stock value or reputation.
- Enhanced Social Engineering Attacks (including Phishing): AI can create highly convincing and personalized phishing messages, deepfake impersonations for fraud, or engage in real-time interactions via chatbots to manipulate employees or customers.
- Criminal Advice Generation: AI systems can be prompted to generate instructions for theft, fraud, evasion, or other illegal activities, which could be used to target an organization's assets or operations.
- Automated Spam and Malware Generation: AI can rapidly automate the creation of sophisticated spam and malware, bypass traditional security filters, discover system vulnerabilities, or create code to facilitate cyberattacks against an organization's infrastructure.

These misuse categories often overlap. Deepfakes may be embedded in phishing attacks, spam may be used to spread misinformation, and generative tools may power social engineering at scale. The convergence of these threats creates a rapidly evolving and interconnected risk landscape that is difficult to monitor and counteract using conventional safeguards.

- AIID Incident 965: Phishers Allegedly Using AI-Generated Video of YouTube CEO Neal Mohan to Target Creators: Scammers are reportedly using an AI-generated deepfake of YouTube CEO Neal Mohan to steal user credentials. The fake video announces false changes to YouTube’s monetization policy, and it then tricks creators into clicking malicious links or downloading malware. The scam spreads through private videos and emails from a fake YouTube address. It exploits platform features to appear legitimate. Victims risk losing account access or exposing sensitive data.
- AIID Incident 766: Trump Shares AI-Generated Images Falsely Suggesting Taylor Swift Endorsement: Donald Trump shared AI-generated images on social media that falsely depicted Taylor Swift endorsing him for the upcoming election. The images, which included Swift dressed as Uncle Sam and fans wearing “Swifties for Trump” shirts, were shared despite being labeled as satire.