AI System Manipulation and Exploitation Practices
The prohibited AI practices content likely addresses specific manipulation and exploitation techniques that are banned, which represents a distinct regulatory concern beyond general prohibition.
manipulation exploitation subliminal techniques vulnerability exploitation psychological manipulation behavioral manipulation deceptive practices harmful manipulation
Overview
Legal Framework
The prohibition of AI systems involving manipulative or exploitative practices is governed by Article 5(1)(a) of the AI Act, which is detailed by Recitals 29 and 110. The law bans the placing on the market, putting into service, or use of an AI system with the objective or effect of materially distorting a person's behavior through subliminal, manipulative, or deceptive techniques. This prohibition applies when such distortion causes or is likely to cause significant harm. Recital 29 clarifies that the harm includes adverse impacts on physical or psychological health, while Recital 110 extends the concern to systemic risks from general-purpose AI models, such as undermining democratic processes or public security.
Practical Application
The prohibition targets practices that subvert human autonomy and free choice. The key interpretation from the Recitals is the dual focus on intent (the objective to distort) and outcome (the effect of distortion). A system can be banned even absent malicious intent if its operation has a materially distortive effect leading to significant harm. The concept of "material distortion" is linked to exploitation of vulnerabilities related to age, disability, or specific circumstances. While direct case law on this AI-specific provision is nascent, data protection principles on fairness and manipulation, as referenced in cases like PROMUSICAE, inform the understanding of when technical data processing becomes an unfair manipulation of an individual.
Key Considerations
- Conduct a "Manipulation Risk Assessment": Scrutinize AI system design, especially user interfaces and decision pathways, for techniques that subliminally or deceitfully nudge users toward harmful decisions they would not otherwise make.
- Document Design Objectives: Maintain clear evidence that the system's purpose and foreseeable effects are not to materially distort behavior, as liability can attach to both the intended objective and the actual effect.
- Assess Systemic Risk for GPAI: Providers of general-purpose AI models must evaluate if their model's capabilities could enable downstream manipulative practices at scale, constituting a systemic risk under Recital 110.
Laws (11)
Case Law (1)
Guidance (8)
Richtsnoeren 3/2019 inzake de verwerking van persoonsgegevens door middel van videoapparatuur
guidelines cameratoezicht
Richtsnoeren 3/2018 over het territoriale toepassingsgebied van de AVG (artikel 3)
guidelines territoriaal toepassingsgebied AVG
Guidelines 4/2019 on Article 25 Data Protection by Design and by Default Version 2.0 Adopted on 20 October 2020
Guidelines on data protection by design and by default
Guidelines 3/2019 on processing of personal data through video devices
Guidelines on processing of personal data through video devices
Guidelines 8/2020 on the targeting of social media users
Guidelines on the targeting of social media users
Richtsnoeren 4/2019 inzake artikel 25 Gegevensbescherming door ontwerp en door standaardinstellingen
guidelines privacy by design en default
Richtsnoeren 8/2020 betreffende de targeting van gebruikers van sociale media
guidelines targeting gebruikers sociale media
Richtsnoeren 3/2022 betreffende het herkennen en vermijden van misleidende ontwerppatronen in de interfaces van socialemediaplatforms
guidelines misleidende ontwerppatronen
Deze richtsnoeren bieden praktische aanbevelingen aan aanbieders van sociale media als verwerkingsverantwoordelijken van sociale media, ontwerpers en gebruikers van socialemediaplatforms, over het beoordelen en vermijden van zogenaamde 'misleidende ontwerp patronen' in de interfaces van sociale media die inbreuk maken op de vereisten van de AVG. Daartoe beveelt de EDPB aan dat verwerkingsverantwoordelijken gebruikmaken van interdisciplinaire teams, bestaande uit onder meer ontwerpers, func...
News (3)
Beware: Government Using Image Manipulation for Propaganda
U.S. Homeland Security Secretary Kristi Noem last week posted a photo of the arrest of Nekima Levy Armstrong, one of three activists who had entered a St. Paul, Minn. church to confront a pastor who also serves as acting field director of the St Paul Immigration and Customs Enforcement (ICE) office. A short while later, the White House posted the same photo – except that version had been digitally altered to darken Armstrong’s skin and rearrange her facial features to make it appear she was sobb
Manipulation by Algorithms. Exploring the Triangle of Unfair Commercial Practice, Data Protection, and Privacy Law
Machine learning can be used to optimize sales practices in consumer markets, but it also raises concerns about manipulation. According to this article, in order to mitigate these risks, we need to understand how unfair commercial practice, data protection, and privacy law interact.
On dark patterns and manipulation of website publishers by CMPs
This paper studies the installation and configuration process of consent pop-ups and their potential effects on website publishers' decision making. It finds that Consent Management Providers often violate the law, and that configuration options may lead to non-compliance.