\n<\/p>\n
In dystopian fiction, like Orwell\u2019s 1984<\/em><\/a>, thought criminals get punished for thinking against the state. In such unwanted scenarios, a thought is consciously manifested in people\u2019s behaviour (e.g., ideas written in a diary); and that behaviour is observed by humans (e.g., the thought-police-officers, who would see that diary and catch the thought-criminal).<\/p>\n Today, AI-emotion-inferring is taking Orwellian surveillance from the humanly observable to the AI-legible; and from thinking to pre<\/em>-thinking, the pre- or even unconscious.<\/p>\n Emotional surveillance via AI is aimed at interpreting AI-readable data and attributing emotions to them (Bouchagiar<\/a>). As such, it may entail the black-box-processing of humanly invisible data \u2013 like facial micro-patterns that can solely be read by AI \u2013 and information consciously or unconsciously disclosed by a person (e.g., fear arising prior to conscious thinking).<\/p>\n This post argues that law enforcement\u2019s emotional surveillance via AI has the potential to be outrageously excessive where it targets humanly invisible and pre- or unconscious emotional data. The post proposes the recognition of the right to opacity of such data, with the objective of making EU law catch up to a world, where the line between inner and outer life may be technologically erased.\u00a0<\/p>\n Before proceeding to the discussion, it is worth distinguishing between manifested\/conscious and unmanifested\/unconscious emotions.<\/p>\n On the one hand, manifested emotions are and should be regulated. For instance, judges, deciding on the length of a sentence, may take into account frustration, as disclosed during the commission of a crime. Moreover, the law is, in principle, concerned with conscious, voluntary emotional states. For example, criminal liability requires a voluntary action and a guilty mind (e.g., Weigend<\/a>; Chiao<\/a>).<\/p>\n On the other hand, unmanifested emotions have traditionally remained and arguably should remain untouched by regulation. No person is punished for an emotion that s\/he felt but never expressed. Similar can be the case with unconscious emotions: for example, one cannot get mistreated for an act committed, whilst in a state of unconsciousness (here, there may be exceptions, where, e.g., one knowingly causes her\/his own state of unconsciousness, such as voluntarily taking drugs before driving).\u00a0<\/p>\n Having drawn a clear line between the untouchable inner and the regulatable outer, let us look at the EU\u2019s permissive approach to emotional surveillance.\u00a0<\/p>\n The audacity to permit the possibly un<\/em>reliable inference of an un<\/em>defined mental state for law enforcement goals<\/strong><\/p>\n In the past, the access of the police to invisible pre- or unconscious emotional data was impossible. There was, therefore, no need to ban or even regulate this impossibility. Now that such access is technologically possible, we would expect regulators to prohibit it outright. This absolute prohibition would be desirable, because AI may be misused for the establishment of dystopian thought-control-like regimes; and<\/em> because we, humans, have not yet clearly defined what emotions are (Posner<\/a>); and<\/em> because AI-emotion-inferring may be unreliable (e.g., in view of cultural\/individual variability<\/a>, <\/sup>\u00a0lab-vs-field<\/a> gaps or bias<\/a>).<\/p>\n In fact, in its amendments<\/a> to the European Commission\u2019s initial proposal<\/a> for what would become the AI Act<\/a>, the European Parliament (EP) had recommended an unconditional ban on emotion-inferring for law enforcement goals. The logic behind the ban was straightforward: there are major abuse-risks, given \u2018serious concerns about the scientific basis of AI systems aiming to detect emotions<\/em>\u2019, including (1) cultural, individual and contextual variability, (2) \u2018limited reliability (emotion categories are neither reliably expressed through, nor unequivocally associated with, a common set of physical or physiological movements)<\/em>\u2019, (3) \u2018lack of specificity (physical or physiological expressions do not perfectly match emotion categories)<\/em>\u2019 and (4) \u2018limited generalisability (the effects of context and culture are not sufficiently considered)<\/em>\u2019 (see Amendments<\/a>, recital 26c (new); Article 5(1)(dc new)).<\/p>\n Notably, this ban could be supported by the EP\u2019s proposed general principles that would apply to all AI practices (Amendments<\/a>, Article 4a), e.g.: \u2018human agency and oversight<\/em>\u2019, requiring that AI be solely used to serve the people and respect dignity (Article 4a(a)); \u2018privacy and data governance<\/em>\u2019, demanding that any AI be compliant with the right to privacy (Article 4a(c)); \u2018transparency<\/em>\u2019, meaning that all AI-tools must be explainable and inform humans on their interaction with AI and its possible impact (Article 4a(d)); and \u2018diversity, non-discrimination and fairness<\/em>\u2019, under which every AI-use should avoid undue discrimination\/bias (Article 4a(e)).<\/p>\n Problematically, the final text of the AI Act<\/a> completely ignored these major abuse-risks that may arise especially in the area of law enforcement, as pointed out by the EP (Amendments<\/a>, recital 26c (new)). Moreover, the AI Act<\/a>\u2019s final version accepted only few of the EP\u2019s unreliability concerns (e.g., context-dependence, cultural\/individual-variability or error-rates); but this, in areas other than law enforcement (i.e., the workplace and education; see recital 44 of the AI Act<\/a>). The EP\u2019s general principles, that would govern all AI-practices, were eventually reduced to certain obligations and requirements solely applicable to \u2018high-risk\u2019 AI-uses (Chapter III of the AI Act<\/a>).<\/p>\n What the above means for law enforcement is that, without defining \u2018emotions\u2019 and after accepting unreliability concerns, the AI Act<\/a> sees AI-emotion-recognition as \u2018high-risk\u2019 (Article 6(2) and Annex III, para 1(c)). It follows that designers or deployers of AI systems must meet certain obligations\/requirements (Articles 8-27 AI Act<\/a>), which are, however, limited, accompanied by exceptions favouring uses by the police (e.g., transparency obligations to inform on exposure to emotion recognition do not, in principle, apply to criminal investigations; see , Article 50(3) AI Act<\/a>).<\/p>\n Overall, it is questionable whether the AI Act<\/a>\u2019s conformity or other assessments could sufficiently protect us from police-abuses that the EP had intended to avoid: misusing the potentially unreliable inference of what we have not yet clearly defined; and this, for law enforcement goals. This was an audacious choice from the part of the EU legislature because it potentially leaves our humanly invisible and pre- or unconscious emotional information in the hands of any \u2018petty officer<\/em>\u2019, who may, then, play with our rights and freedoms (see Olmstead<\/em><\/a>, para 474).\u00a0\u00a0<\/p>\n Given that what dystopian imagination required a totalitarian state to achieve, the AI Act gave us through polite exemptions and gentle risk-classifications, it is worth exploring how existing laws could set out a categorical ban, whose un<\/em>conditionality would be guaranteed by the recognition of a human right to opacity.<\/p>\n \u00a0Bans are desirable, but not enough\u00a0\u00a0<\/strong><\/p>\n If we accepted that, as Warren and Brandeis (p. 213)<\/a> argued within their discussion on the right to be let alone, the \u2018principle which protects <\/em>(\u2026) emotions <\/em>(\u2026) is the right to privacy<\/em>\u2019, then the regulation of emotional surveillance would be primarily a matter of privacy law.<\/p>\n In the EU, we have the General Data Protection Regulation (\u2018GDPR<\/a>\u2019) and the Law Enforcement Directive (\u2018LED<\/a>\u2019). But they say nothing about emotions. Still, the European approach to the concept of \u2018personal data\u2019 is likely broad enough (Purtova<\/a>) to cover emotions as inferred<\/em> via AI. In fact, inferred emotions (e.g., happiness) can, with the aid of additional information (e.g., biometrics), meet the identifiability-criterion, under which it is the capacity to identify someone that makes data personal (FRA and CoE<\/a>, pp 88-93). Inferred emotions can, indeed, be relied upon to single out a person (on identifiability and the use of identifiers, see Opinion 4\/2007<\/a>, pp 12-15).<\/p>\n Furthermore, emotions as inferred <\/em>via AI can qualify as highly <\/em>sensitive personal data. As can be the case with the GDPR<\/a>and the LED<\/a>\u2019s special categories of personal data, the context of their processing may \u2018create significant risks to the fundamental rights and freedoms<\/em>\u2019 (recital 51 of the GDPR<\/a>). Compared to known sensitive data, inferred emotions may be equally intimate (Lench and Carpenter<\/a>); e.g., shame could be as private as sexual orientation. They may be equally exploitable (Bakir et al<\/a>), e.g., resulting in manipulation. They may be equally revealing, e.g., showing health-related data (Morie et al<\/a>). And, importantly, they may be (equally or) more<\/em> pervasive; e.g., emotional information could be gathered continuously in any context without a concrete collection operation (contrary to other data that demand, e.g., sample-taking) \u2013 our mere existence in a watched place can suffice.<\/p>\n In view of the unreliability concerns of AI-inferred emotions, the fact that such inferences may be untrue (for instance, frustration erroneously attributed to a facial expression) does not<\/em> mean that such falsely inferred data do not<\/em> constitute sensitive data (see, by analogy, C-492\/23 Russmedia Digital<\/em><\/a>, paras 52-53).\u00a0<\/p>\n In view of these considerations, EU regulators could introduce a new category of highly <\/em>sensitive personal data to expressly safeguard \u2018inferred emotions\u2019, including our humanly invisible and pre- or unconscious aspects. This would be a realistic and immediate response to law enforcement\u2019s emotional surveillance, in the sense that it could come as an amendment to the GDPR<\/a> (Article 9(1)) and the LED<\/a> (Article 10) \u2013 an amendment that could be deemed necessary by the Commission, in view of latest AI-developments (Article 97(5) GDPR<\/a>; Article 62(5) LED<\/a>).<\/p>\n The proposed prohibition-by-default could be accompanied by sector-specific bans in other areas of law. Of particular relevance is criminal law and procedure. If the police inferred our emotions via AI, we would risk being unduly mistreated on the basis of our looks (as read by AI), our involuntary (pre- or unconscious) mental states, our future actions that we may (but also may not<\/em>) take, as well as erroneously inferred emotions.<\/p>\n In such unfair mistreatment-scenarios, we could bring our case before courts. And judges would make questions on reliability (e.g., can the way in which evidence was gathered create doubts on reliability? did the defence have the opportunity to contest reliability?) or questions on the role (e.g., determinative\/non-decisive) that evidence played in the outcome of the case (before the ECtHR, see e.g.: Dragos<\/em><\/a>, para 49; El Haski<\/em><\/a>, para 83; Jalloh<\/em><\/a>, paras 95-96; Schenk<\/em><\/a>, para 48; before the CJEU, see La Quadrature du Net<\/em><\/a>, paras 222-226).<\/p>\n On the question of the weight attached to evidence, inferred emotions could problematically be accepted, if they played an (allegedly) non-decisive role in the outcome of the case. For instance, judges might say they used emotion-inferring for corroboration, in conjunction with other evidential materials, which (other evidential materials) can independently support their decision (see, by analogy, Schenk<\/em><\/a>, para 48). In practice, this would mean that judges might be permitted to consider inferred emotions, insofar as they were informed on AI\u2019s limitations (see, by analogy, experience from the US Loomis-<\/em>case<\/a>).\u00a0<\/p>\n In light of the foregoing considerations, a comprehensive prohibitive framework would, in the context of fair trials, require a ban on all uses of emotional surveillance via AI in the area of law enforcement; so that no inferred emotion gets into the courtroom (or any phase of the criminal proceedings).\u00a0<\/p>\n An important remark must be made on the above-proposed prohibitive scheme via data protection and other laws. The suggested ban could only work, if it were unconditional and<\/em> categorical.\u00a0\u00a0<\/p>\n More precisely, the recommended sensitive-data-categorisation should not treat emotion-inferring as a processing issue to be handled carefully. The problem is not how<\/em> law enforcement can process humanly invisible and pre\/unconscious emotional data. Rather, the problem is that law enforcement can<\/em> process such data in coercive contexts. If inferred emotions were merely seen as a sensitive personal data category, they might be subject to balancing exercises and, e.g., be overcome by an (allegedly) sufficiently compelling public interest (see exceptions to the ban on sensitive personal data processing: Article 9(2) GDPR<\/a>; Article 10(a)-(c) LED<\/a>). That would be a loophole that the \u2018police-must-be-fit-for-the-AI-age\u2019-advocates might use. Hence, inferred emotions should be put \u2018higher<\/em>\u2019 than other sensitive data, because, as this post argued, emotional surveillance can make what dystopian fiction had not imagined reality.<\/p>\n In the same vein, the fair-trials-ban should be unconditional. Any exception to the recommended rule of inadmissibility-by-default might result in unfairness. For instance, consent of the individual concerned would not work as a valid exception; it would be coerced, not freely given, in light of power asymmetry.<\/p>\n In other words, there should be no<\/em> procedural workaround (e.g., consent or compelling public interest), legitimising the state\u2019s access to the humanly invisible and the pre\/unconscious self of a person. The guarantee of the unconditionality of this categorical ban could come with a human-rights-reform.<\/p>\n Concluding with the right to opacity: Opacity belongs to our inner life, not<\/em> the government\u2019s decision-making processes\u00a0<\/strong><\/p>\n In most, if not all, democracies, states must give reasons for their actions. Contrary, we, citizens, owe no justification to no entity for our inner, unmanifested self. Emotional surveillance via AI turns this upside down: the state becomes unintelligible; and citizens get legible. The recognition of a human right to opacity could fix this.\u00a0<\/p>\n It could be introduced as a new human right, linked to privacy (and other rights\/freedoms, like the right to a fair trial); e.g., same as the right to the protection of personal data came as a distinct, enumerated right in the Charter of Fundamental Rights (\u2018CFR<\/a>\u2019, Article 8). As such, it would have its own weight and irreducible core that no<\/em> (allegedly compelling) public interest could surpass. Drawing inspiration from absolute bans (e.g., on torture), as well as inviolable human rights (e.g., human dignity; Article 1 CFR<\/a>), regulators could define it as the right not<\/em> to have one\u2019s humanly invisible and pre\/unconscious emotions and other mental states inferred or otherwise accessed by the state.\u00a0<\/p>\n