What is Adversarial Attacks
Ways to fool a model with specially selected input data that looks normal to a human but confuses the algorithm.
Definition
Adversarial attacks show that AI can make mistakes not only by chance, but also under deliberate pressure. A small change in image, text, audio, or query can cause the model to produce incorrect results, reveal unnecessary information, or break rules.
Example
You can add almost imperceptible noise to an image of a road sign, and the computer vision model will begin to recognize it as another sign.
Why it matters
The term is important for everyone who implements AI into a product: the model must be tested not only on ordinary examples, but also on attempts to deliberately bypass it.
How it works
The attacker looks for weaknesses in the model: sensitivity to noise, unusual wording, conflicting instructions, or borderline examples.
Where it is used
- model safety
- moderation check
- protection against bypass of AI systems
Limitations
It is difficult to completely eliminate such attacks. Defensive methods can reduce quality on regular data or create a false sense of security without regular testing.