Model Evasion Attack

In Model Evasion attacks, malicious agents craft adversarial data to circumvent a system. For example, spam emails or malware can be designed to look inconspicuous and not trigger a negative label. The adversarial examples are derived programmatically by finding the decision boundary between classes (e.g., adding a large number of "not spam-like" words) or by adding in "noise" that will confuse the model.

These attacks often happen when an actor can test the system many times and figure out what inputs work. Stopping them means limiting access, checking for strange behavior, and testing how easily the model can be fooled.

- Proof of Concept from Image Classification: This study shows that imperceptible changes to images can cause image models to apply different labels.