Human-centered explainable artificial intelligence: An Annual Review of Information Science and Technology (ARIST) paper
Abstract
Explainability is central to trust and accountability in artificial intelligence (AI) applications. The field of human-centered explainable AI (HCXAI) arose as a response to mainstream explainable AI (XAI) which was focused on algorithmic perspectives and technical challenges, and less on the needs and contexts of the non-expert, lay user. HCXAI is characterized by putting humans at the center of AI explainability. Taking a sociotechnical perspective, HCXAI prioritizes user and situational contexts, preferences reflection over acquiescence, and promotes the actionability of explanations. This review identifies the foundational ideas of HCXAI, how those concepts are operationalized in system design, how legislation and regulations might normalize its objectives, and the challenges that HCXAI must address as it matures as a field.
1 INTRODUCTION
Explanation has always been a central concern of artificial intelligence. From the seminal 1950 conference to rule-based expert systems of the late 20th century to the deep learning neural networks of contemporary AI and the emerging notion of neurosymbolic machine learning, explainability advances accuracy, trust, and accountability. However, “explanation is a wicked problem, perhaps the wicked problem of artificial intelligence research” (Sarkar, 2022). While explanations are tools for veracity, transparency, and justification, they can also be tools for deception, manipulations, and malfeasance.
The complexity, opacity, and ubiquity of machine learning in conjunction with increasing regulatory requirements for explainability have made explainable AI (XAI) a technical and a public priority. Human-centered explainable AI (HCXAI) arose from the shortcomings of explainable AI techniques that focused more on technical considerations and less on user contexts. HCXAI seeks to place the often-neglected user at the center of the explanatory process by promoting, designing, and implementing techniques and principles that align with the explanatory needs of these users. HCXAI resides within the larger field of human centered AI (HCAI), including notions of responsible AI (RAI), and shares objectives with XAI and the field of human-computer interactions (HCI) (Aragon et al., 2022; Hewett et al., 1992; Shneiderman, 2022). As machine learning systems become increasingly consequential in the lives of everyday people, effective explanations of decisions, predictions, and recommendations will be essential if trust is to be maintained and accountability realized.
Upol Ehsan, a co-founder of the annual HCXAI Workshop hosted by ACM's SIGCHI conference, explains HCXAI this way: XAI believes “if you can just open the black box of AI, everything will be fine” whereas HCXAI broadens that view because “not everything that matters lies inside the box. Critical answers can lie outside it. Why? Because that's where the humans are” (Ehsan & Riedl, 2023).
This review identifies and explores the discourse of this emergent field. The review is structured as follows. A discussion of the importance of explanations is followed by the methodology used in this review and a description of key terminological issues. A basic introduction to XAI is followed by an extensive analysis of HCXAI. The latter includes the identification of foundational papers and other related papers, and the exploration of key concepts and issues central to the field. A year-by-year analysis illustrates the evolution of HCXAI and explores a set of specific concerns and challenges. As HCXAI evolves, some researchers have reinterpreted the field in ways that expand core ideas or present different perspectives. These papers are examined. With the discourse of HCXAI considered, the next section explores HCXAI in a regulatory context. This includes examination of key regulations as well as regulatory challenges for HCXAI. The review concludes with an assessment of future research needs and outstanding challenges for HCXAI.
2 THE EXPLANATORY IMPERATIVE
Explanations “are more than a human preoccupation—they are central to our sense of understanding, and the currency in which we exchange beliefs” (Lombrozo, 2006, p. 464). There are five reasons for an explanation: (1) to predict similar events in the future, (2) to diagnose, (3) to assess blame or guilt, (4) to justify or rationalize an action, and (5) for aesthetic pleasure (Keil, 2006). These different objectives highlight that good explanations are recipient dependent and sensitive: “Every theorist of explanation can admit that the idea of a good explanation is audience-variant” (Ruben, 2012, p. 19) and that “a good explanation is one that meets the interests, and assumes what it should assume about the beliefs, of the audience” (Ruben, 2012, p. 26).
People select explanations that are easy to grasp, apply beyond the specific instance, and align with their understanding of other things (Schwarz et al., 1991). As Garfinkel cautions, “explanations that are too concrete are not merely ‘too good to be true’ (i.e., impractical) but rather ‘too true to be good’” (Garfinkel, 1981, p. 58). Hence an explanation can “satisfice” when it “meets a variety of criteria but maximizes none” (Simon, 1992, p. 157).
What constitutes a good explanation and how it is communicated are key questions for XAI. This includes why an explanation is wanted, what should be included in the explanation, how should it be presented, and when should it be presented.
The need for XAI is diverse: verification and accuracy of the models (Arrieta et al., 2020), model improvement (Gunning et al., 2021), compliance (Samek et al., 2017), learning (by the user) (Miller, 2019), user trust and acceptance (Lim et al., 2009), fairness (Doshi-Velez & Kim, 2017), decision-making and trouble-shooting (Rudin, 2019), and support for human agency (Abdul et al., 2018). The audience for an XAI system can be similarly diverse. This can include system developers (who are primarily interested in performance), clients (primarily interested in effectiveness or efficacy), professionals (primarily interested in work related outcomes), regulators (primarily interested in policy implications), and everyday users of the models (primarily interested in trust or accountability). The emphasis on specific users in specific contexts acknowledges that “different explanations are needed for different people” (Kühl et al., 2020, p. 4).
3 METHODOLOGY
The objective of this review is to identify the key concepts, issues, and desiderata of HCXAI. A thematic analysis of the historical and contemporary HCXAI and XAI literature was used to identify important themes. A narrative approach uncovered the discourse of this field as it emerged and continues to evolve. It does not claim to be either a comprehensive bibliography or a systematic review of HCXAI. It is a selective reading of the field that identifies foundational contributions and those that extend or advance the evolution of HCXAI.
Key papers were selected from prior literature reviews, including a systematic review (Williams, 2021), a publication trend analysis (Jacovi, 2023), an annotated bibliography (Mueller et al., 2019), and numerous reviews of the field (Ali et al., 2023; Arrieta et al., 2020; Das & Rad, 2020; Haque et al., 2023; Saeed & Omlin, 2023; Saranya & Subhashini, 2023).
Searches were conducted in Scopus, Web of Science, Google Scholar, arXiv, Library and Information Science Abstracts, and Library and Information Science Full Text. In addition to “HCXAI,” other key search terms included: human centered, human centric, user centered or user centric used in conjunction with XAI, explainability, explainable AI or explainable artificial intelligence. Alerts were set on Altmetric, Google Scholar, and arXiv using the same search terminology. As a rapidly evolving field, these alerts were useful in identifying the most current publications. Conferences are primary publication vehicles for XAI research. Key conferences, held in the past 5 years (such as the ACM Conference on Human Factors in Computing Systems [CHI] and the International Conference on Machine Learning [ICML]), were reviewed as a further exploration for relevant publications.
The literature of XAI is vast. A search of “explainable AI” on Google Scholar retrieved over 50,000 results. Adding “human-centered” as a key word reduced the number to a still daunting 5000. However, what is understood as HCXAI is more exclusive than this large corpus. XAI research often references human-centeredness but typically this means it either focuses on a narrow target population (e.g., developers, technicians, or domain experts) or it lacks the wider perspective emblematic of HCXAI. The essential distinction between XAI and HCXAI is the latter's emphasis on a broader sociotechnical, personalized, and contextualized approach with a focus on the lay, non-expert user. It puts the user at the center of considerations, not the XAI technique or strategy.
The search strategy outlined above yielded many candidate papers. While some papers could be excluded solely by their titles (it was evident that the emphasis was on narrow technical concerns), others required a review of their abstracts to determine exclusion or inclusion. Papers downloaded for inclusion were scanned for themes specific to HCXAI. This tertiary examination sifted out papers that included a human or user centric focus but failed to locate this within a sociotechnical framework. In all cases, citation chaining was used to identify further relevant literature. Only English language papers published prior to January 2024 were included.
3.1 Terminology
Explainable AI has a terminology problem (Schneeberger et al., 2023). Terms such as explainable, transparent, interpretable, intelligible, and traceable are used widely and often undifferentiated (Arrieta et al., 2020). The use of “a wide range of synonyms indicates a breadth of interpretations of what Explainable AI means” (Brennen, 2020, p. 3). This imprecision is further compounded by the field of artificial intelligence adopting descriptions that include notions of explainable AI: Trustworthy AI, Responsible AI, and Human-Centered AI. In 2006 Marvin Minsky coined the term “suitcase words” to describe the regrettable practice in the AI community of assigning a variety of meanings to a particular word (Minsky, 2006). This problem continues.
Lacking “a theory of explainable AI, with a formal and universally agreed definition of what explanations are” (Samek & Muller, 2019, p. 17), the fundamentals of XAI are still being explored, often from different disciplinary perspectives with diverse terminology (Mueller et al., 2019). The result is what Lipton calls “a surfeit of hammers, and no agreed-upon nails” (Lipton, 2017).
Traceability and interpretability are generally used to describe the explanatory process for AI systems that are inherently understandable such as decision trees or logic statements. Explainability and intelligibility are used as synonyms to refer to the explanatory process for “opaque-box” or “black-box” models that are not inherently understandable (i.e., they require post-hoc processes). These models or systems dominate the current AI landscape and are the preferred terms for this review. Transparency is the most problematic term because while some use it as a technical description (e.g., inherently interpretable systems are transparent) others use it as more of an aspirational term (e.g., transparency in AI system promotes accountability). This review will use the latter meaning.
For the discussion that follows, another terminological issue requires clarification. It is important to differentiate “XAI,” “algorithmic XAI,” and “human-centered XAI (HCXAI).” XAI refers to the broad field of explanation in AI and will be used to describe the field in total. Algorithmic XAI references work in XAI that focuses only on the algorithmic issues and implications. This is view of XAI that is mostly technically oriented and is the most dominant focus of the field. Human-centered XAI (HCXAI) is the topic of this review. It can be viewed a subfield or adjacent field of XAI that takes a sociotechnical perspective where the user is a primary focus.
4 EXPLAINABLE AI
Explainability in artificial intelligence has always been important. In the initial journal article describing MYCIN, an early and influential expert system, the authors laud as one of its specific contributions “its ability to explain the reasons for its decisions” through user accessible “decision rules” (Shortliffe et al., 1973, p. 545). A critique of MYCIN and similar systems noted that the “explanation must be in terms that the physician can understand” and that the strengths and weaknesses of the system be apparent to the user (Gorry, 1973, p. 50).
With “second generation” expert systems in the early 1990s, explainability had improved by incorporating more knowledge about “the domain, the user, the expert system and its design, and about the generation of explanations themselves” (Swartout & Moore, 1993, p. 582). Better explanations required a broader context than just the decision rules. By the end of the century, a cognitive science approach to explanation was recommended including recognition of cognitive effort, the role of learning theory, and the expectations that “social, ethical, or organizational theories” could contribute to better explanations (Gregor & Benbasat, 1999, p. 516).
Into the 21st century, with artificial neural networks and advanced computation, AI became increasingly complex and opaque. Explainability became more important, and techniques and motivations for explanations became more diverse. The term “explainable AI” and the abbreviation “XAI” was coined in a paper about explanation in an AI-based military training video game (Van Lent et al., 2004). As a result, the field of explainable AI (XAI) was named and launched.
According to the widely referenced US Defence Advanced Research Projects Agency (DARPA) description, the purpose of explainable AI (XAI) is for AI systems to have “the ability to explain their rationale, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future” (DARPA, 2016) and to “enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners” (Turek, 2016). More concisely, XAI is “concerned with developing approaches to explain and make artificial systems understandable to human stakeholders” (Langer et al., 2021).
XAI is a set of strategies, techniques, and processes that include testable and unambiguous proofs, various verification and validation methods that assess influence and veracity, and authorizations that define requirements or mandate auditing. Recent survey articles review the main approaches to XAI including desiderata and recommended research directions (Arrieta et al., 2020; Haque et al., 2023; Mueller et al., 2019; Saeed & Omlin, 2023; Yang et al., 2023).
A comprehensive review of concepts and techniques includes a useful taxonomy for XAI (Ali et al., 2023). Three explainability methods are defined: data explainability (e.g., using feature perturbation, knowledge graphs, saliency maps), model explainability (for specific models or model agnostic strategies such as LIME), and post hoc explainability (e.g., using visualization, narratives, examples, or rule extraction). This technical perspective is complemented by another overview of XAI concepts and techniques from the user or stakeholder perspective (Langer et al., 2021). The conceptual model presented in this paper links explainability approaches (i.e., the techniques described in Ali et al., 2023) to the resulting explanatory information and connects this to the context of the user or stakeholder.
“Opening the black box,” making complex, opaque AI systems understandable, is XAI's central motivation (Castelvecchi, 2016). However, “the solution to explainable AI is not just ‘more AI.’ Ultimately it is a human-agent interaction problem” (Miller, 2019, p. 2). For XAI to respond to Miller's challenge, it needed move in a different direction.
5 FROM XAI TO HCXAI
A blunt assessment of the state of XAI in 2017 captured a key deficiency: “AI researchers are building explanatory agents for ourselves, rather than for the intended users … the inmates are running the asylum” (Miller et al., 2017). A 2018 research agenda for HCI and explainability observed that XAI has neglected “the human side of the explanations and whether they are usable and practical in real-world situations” (Abdul et al., 2018, p. 9). A 2019 review noted the growing importance of “human factors” to XAI although it fell short of proposing a more holistic view (Mueller et al., 2019). It was clear that the XAI community was realizing the limitations of algorithmically centered XAI.
While HCXAI is most closely derived from the work of mainstream and algorithmic XAI, there are other foundational sources that have shaped how researchers work in this field. A review of the literature of explanation in human-AI systems identified a wide array of disciplinary influences: philosophy and psychology of science, psychology of explanation, social psychology, psycholinguistics, team science, and “other human factors, cognitive systems engineering, and decision aids” (Mueller et al., 2019). This aggregation of disciplines, including law, has been called “the explanation sciences” (Mittelstadt et al., 2019).
However, beyond XAI and the explanation sciences, the most prominent influence on HCXAI is the human-computer interaction (HCI) field. HCI is “concerned with the design, evaluation, and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them” (Hewett et al., 1992, p. 5). An extensive review of the HCI literature related to XAI developed a research agenda for the field identifying three areas for future consideration (Abdul et al., 2018): how people interpret explanations, deploying and evaluating interactive explanations, and widening the explanatory context to include to social dimensions. Broadening XAI research from a sociotechnical perspective is something that both HCI and HCXAI researchers address, cultivating influences from Science and Technology Studies (Felt et al., 2016) and the social construction of technology (Bijker et al., 1987). Noting that “theory-driven XAI is still a nascent area,” Liao and Varshney suggest that new influences could come from “users of XAI as information seekers” as theorized in sense-making and information seeking behaviors (Liao & Varshney, 2022, p. 13).
These four points all converge around a single point: explanations are not just the presentation of associations and causes (causal attribution), they are contextual. While an event may have many causes, often the explainee cares only about a small subset (relevant to the context), the explainer selects a subset of this subset (based on several different criteria), and explainer and explainee may interact and argue about this explanation. (Miller, 2019, p. 3)
In addition to highlighting the importance of context and actionable explanations, Miller's paper demonstrated the value of broadening the disciplinary perspectives of XAI. A 2023 review of the XAI literature demonstrated the impact of this expanded view, concluding that “XAI is becoming increasingly multidisciplinary, with relative growth in papers belonging to increasingly diverse (non-Computer Science) scientific fields, increasing cross-field collaborative authorship, increasing cross-field citation activity” (Jacovi, 2023, p. 1).
The limitations of algorithmic XAI, the imperative of recognizing user and social contexts, and the broadening of disciplinary perspectives motivated the emergence of a new subfield of XAI, human-centered XAI (HCXAI).
6 FOUNDATIONAL PAPERS
In 2020, Upol Ehsan and Mark Riedl, both from the Georgia Institute of Technology, working together and with other colleagues, initiated discussions about HCXAI in a series of papers that established the central objectives of the field and explored ramifications from a multidisciplinary perspective. Taken together, these papers provide an introduction to HCXAI and create the platform upon which other HCXAI papers reside. While these papers mark the beginning of HCXAI as a defined area of research and development, two earlier papers suggested a human-centered approach to XAI albeit in a less fully developed manner (Ribera & Lapedriza, 2019; Wang et al., 2019).
Noting that “current research is not paying enough attention to whom the explanations are targeted,” Ribera and Lapedriza propose a “user-centered explainable AI” that responds to the needs and interests of specific user groups (Ribera & Lapedriza, 2019). The authors identify “lay users” as a neglected constituency and recommend grounding explanation for this group in principles of human conversation (Grice, 1975) hence emphasizing interactivity as a core requirement. However, Ribera and Lapedriza tend to view users as aggregated groups rather than individuals.
Wang et al. describe a framework for “human-centered, decision-theory-driven XAI” that takes into consideration “how people reason, make decisions and seek explanations, and cognitive factors that bias or compromise decision-making” (Wang et al., 2019, p. 1). This cognitive science approach to XAI underscores that “understanding people informs explainable AI” (Wang et al., 2019, p. 1). Drawing from the literature of psychology, philosophy, and decision-making theory, the authors align human reasoning and reasoning heuristics with specific XAI strategies and techniques. While criticizing the “unvalidated guidelines” and taxonomies which have dominated XAI, the authors acknowledge that taxonomies provide the “high-level objectives” for their “lower-level building blocks.” However, as Wang et al. recognize, their cognitive science approach does not include a larger sociotechnical perspective.
The next sections follow the evolution of HCXAI through a year-by-year analysis of foundational papers and other related contributions (Table 1).
Year | Foundational papers |
---|---|
2020 | Foundational Concepts of HCXAI Ehsan, U., & Riedl, M. O. (2020). Human-centered explainable AI: Towards a reflective sociotechnical approach. |
2021 | Amplifying and Extending HCXAI Ehsan, U., Wintersberger, P., Liao, Q. V., Mara, M., Streit, M., Wachter, S., Riener, A., & Riedl, M. O. (2021). Operationalizing human-centered perspectives in explainable AI. Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2021). The who in explainable AI: How AI background shapes perceptions of AI explanations. Singh, R., Cheong, M., Miller, T., Ehsan, U., & Riedl, M. O. (2021). LEx: A framework for operationalising layers of AI explanations. Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021). Expanding explainability: Towards social transparency in AI systems. Ehsan, U., & Riedl, M. O. (2021). Explainability pitfalls: Beyond dark patterns in explainable AI. |
2022 | Critical Perspectives and Disciplinary Viewpoints Ehsan, U., Wintersberger, P., Liao, Q. V., Watkins, E. A., Manger, C., Daumé III, H., Riener, A., & Riedl, M. O. (2022). Human-centered explainable AI (HCXAI): Beyond opening the black-box of AI. Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daume III, H. (2022). Seamful XAI: Operationalizing seamful design in explainable AI. Ehsan, U., & Riedl, M. O. (2022). Social construction of XAI: Do we need one definition to rule them all? |
2023 | Elaborating and Responding Ehsan, U., Wintersberger, P., Watkins, E. A., Manger, C., Ramos, G., Weisz, J., Daumé III, H., Riener, A., & Riedl, M. O. (2023). Human-centered explainable AI (HCXAI): Coming of age. Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023). Charting the sociotechnical gap in explainable AI: A framework to address the gap in XAI. |
6.1 Foundational concepts of HCXAI (2020)
an approach that puts the human at the center of technology design and develops a holistic understanding of “who” the human is. It considers the interplay of values, interpersonal dynamics, and socially situated nature of AI systems. In particular, we advocate for a reflective sociotechnical approach that incorporates both social and technical elements in our design space. (Ehsan & Riedl, 2020, p. 464)
The Ehsan and Riedl conference paper is important not only because it is the first to formalize HCXAI and provide an early definition but also because it identifies some foundational concepts that will guide exploration in this area for the next 3 years and likely beyond. Arguing for a sociotechnical approach, HCXAI acknowledges the “rich tapestry of social relationships” in which technologies are embedded, and allows developers to “critically reflect or contemplate on implicit or unconscious values embedded in computing practices so that we can understand our epistemological blind spots” (Ehsan & Riedl, 2020, p. 450).
However, the authors caution that it is “premature to attempt a full treatise of human-centered XAI” (Ehsan & Riedl, 2020, p. 461) identifying challenges that require engaging “partner communities of practice.” A full understanding of the “who” at the center of HCXAI will require “active translational work” with the research and practices of other disciplines. In a later paper, the authors acknowledge that the different interpretations of these disciplines and stakeholders will shape the evolution of HCXAI and that “forcing a standardization (closure) on the pluralistic interpretations too early can stifle innovation and lead to premature conclusions” (Ehsan & Riedl, 2022). As a new and still emerging subfield deferring closure allows researchers to have “substantial agency in steering the discourse.”
How to “operationalize” HCXAI is noted as a challenge and becomes a theme in the three HCXAI workshops held between 2021 and 2023 (with a fourth workshop planned for 2024). Drawing from the work on critical technical practice (Agre, 1997), participatory design (Bødker, 2004) and value-sensitive design (Friedman et al., 2006), the authors propose two key directives: “(1) a domain that is critically reflective of (implicit) assumptions and practices of the field, and (2) one that is value-sensitive to both users and designers” (Ehsan & Riedl, 2020, p. 461). Both highlight the objective that HCXAI will “foster reflection instead of acceptance” (Ehsan & Riedl, 2020, p. 463).
Putting the user, the “who,” at the center of XAI results in a concise definition of the HCXAI process: “the who governs how the why is most effectively conveyed” (Ehsan & Riedl, 2020, p. 458). Other researchers have amplified this perspective noting the importance of “paying attention to user's prior backgrounds, experiences, and expectations” (Nourani et al., 2022, p. 28:24). Focusing on the “who” involves a “shift from improving the artifact (e.g., an explanation or explanation system) to understanding how humans make sense of complex, and sometimes conflicting, information” (Kaur et al., 2022, p. 703). The Sensible AI (i.e., sensemaking) proposed by Kaur et al. (2022) bridges the sociotechnical perspectives of Ehsan and Riedl (2020) and the cognitive decision-making of Wang et al. (2019).
In describing the core objectives of HCXAI, Ehsan and Riedl present an explicit critique of XAI and an implicit critique of HCI. XAI is criticized for its technical focus, lack of attention to the non-expert user, and dismissal of the holistic context in which explanation occurs. The implicit criticism of HCI centers on the limited efforts of this field to position explanation in the broader and deeper context HCXAI proposes. HCXAI is a response to the deficiencies of both XAI and HCI.
6.2 Amplifying and extending HXCAI (2021)
Held annually since 2021, the Workshop on Human-Centered Explainable AI, hosted by the ACM CHI Conference, charts the evolution of the field. The workshop proposals, required for CHI approval, highlight how the organizers position HCXAI and what key concerns were anticipated. As the workshops proceeded, HCXAI as a field matured establishing foundational areas that sought a unique role within AI and with the larger sociotechnical dialogue.
The questions posed to contributors to the first workshop focused on operationalizing HCXAI with an emphasis on the core who, why, and where questions (Ehsan, Wintersberger, et al., 2021). Other areas of concern were gaps in XAI that HCXAI could address, transferable evaluation methods, and broadening the HCXAI research agenda.
A series of papers published in 2021 amplified and extended HCXAI concepts. While algorithmic XAI continued its recently found interest in user studies, those exploring HCXAI deepened their interest in the user and the technology in the broader sociotechnical context. Four core and associated papers explored the “who” in HCXAI, conceptual frameworks, the idea of social transparency, and potential for explanatory pitfalls.
The central question HCXAI poses for algorithmic XAI is “explainable to whom?” (Ehsan, Passi, et al., 2021). Arguing that “who opens the [black] box” determines how explanations are perceived, this study looked at the differences between people with and without expertise in AI (an AI group and a non-AI group). User perceptions of explanations were measured by five characteristics: confidence, intelligence, understandability, second chance, and friendliness (where “second chance” measures user perception of past system failures on future successes). Different types of explanations were provided: natural language with justification (“why” explanations), natural language without justification (“what” explanations), and numbers that explained the systems actions (in this case Q-values) (“transparent” explanations).
Both groups had unwarranted faith in numbers, but exhibit it for different reasons and to differing degrees, with AI group showing higher propensity to over-trust numerical representations and potentially be misled by the presence of it. The two groups found different explanatory values beyond the usage that the explanations were designed for. Even in their aligned appreciation for humanlike-ness, each group had different requirements concerning what counts as humanlike explanations. (Ehsan, Passi, et al., 2021, p. 2)
Revealingly, the AI group viewed the numerical explanations as useful even though their meaning was unclear to some. The non-AI group, who acknowledged that they didn't understand the meaning of the numbers, perceived the numerical explanations as evidence of an intelligent system.
The “misalignment” between designers and users of a system is well known (Norman, 1983) leading HCXAI to recognize that “the design and use of AI explanations is as much in the eye of the beholder as it is in the minds of the designer” (Ehsan, Passi, et al., 2021, p. 20). The findings underscore not only that who the user is matters but that user perceptions can be manipulated by the “dark patterns” of intentionally deceptive practices (Gray et al., 2018).
Once we recognize that users will make their own interpretations (i.e., appropriations) of XAI statements, then we center our design around user agency by shifting our intention from control of users' understanding, toward providing resources through which users can construct their own understanding. (Ehsan, Passi, et al., 2021, p. 21)
While others concur about the importance and relevance of user backgrounds (Dhanorkar et al., 2021; Suresh et al., 2021), there is debate about the “assumption of improved agency” arising from explanations (Adenuga & Dodge, 2023). These authors argue that users need specific “control functions” to facilitate actionability and explicitly enable agency.
Conceptual frameworks have been used extensively in XAI to capture key concepts, relationships, and processes. They can be early attempts to chart an emerging area or late contributions summarizing a mature or maturing field. The “layers of explanation” (LEx) framework is an early attempt to describe and interrelate key HCXAI concepts (Singh et al., 2021).
Three key questions (who, why, and how) are framed through contextual characteristics of stakes and sensitivity. Stakes define the degree of impact or consequence (low to high risk), while sensitivity captures the “emotional response or susceptibility” of the user to an explanation (low or high). With “who” and “why” identified, the “how” is informed and different types of appropriate explanations (i.e., the layers of explanation) are defined. While no explanation is a valid option, explanations could be feature-based, contrastive, or directive reflecting increasing degrees of complexity and actionability.
The LEx framework is sparse with simplifications that obscure important elements and processes that operationalize HCXAI. Importantly, however, it creates a conceptual baseline for early work in the field and provides a starting point for more comprehensive frameworks.
Drawing from the social transparency literature which makes visible the socially situated context of technology (Stuart et al., 2012), Ehsan, Liao, et al. add that “we are not merely interested in making others' activities visible, but more importantly, how others' interactions with AI impact the explainability of the system” (Ehsan, Liao, et al., 2021). Emphasizing that neither technical nor algorithmic transparency is sufficient to instill trust or support system evaluation, social transparency applied to AI would make available the past outputs of the system (an historical record of performance) and how users interacted with those outputs (including explanations and resulting actions).
As an operationalizing concept within the sociotechnical perspective, social transparency provides additional contextual information that can inform and support HCXAI system design criteria by “infusing human elements in AI.” In addition, the organizational context of this study, where groups of people work together and interact together with AI, goes “beyond the 1-1 human-AI interaction” that dominates much XAI research.
Social transparency is operationalized in the iSee platform by “capturing, sharing and re-using explanation strategies based on past positive experiences … the goal … is to improve every user's experience of AI, by harnessing experiences and best practices in Explainable AI” (Wijekoon et al., 2023, p. 79). The concept of social transparency is expanded by Kroeger et al. (2022) to include “social explainability” where the user knowledge of the “governing ecosystem” of an industry or service is used to inform explanations. Information regarding external certification of the system, data security procedures, and privacy policies are examples of ecosystem knowledge that would influence user acceptance of explanations.
The malicious or deceptive use of explanations is key theme for XAI typically viewed in terms of algorithmic manipulation and often referred to as “dark patterns” (Ehsan & Riedl, 2020; Gray et al., 2018; Lipton, 2016; Stumpf et al., 2016). A more nuanced and related issue is that of “explainability pitfalls” described as “unanticipated and unintended negative downstream effects from AI explanations that can cause users to act against their own self-interests, align their decisions with a third party, or exploit their cognitive heuristics” (Ehsan & Riedl, 2021). The major differences between dark patterns and explainability pitfalls are intentionality, the effect being manifest downstream (i.e., long after initial deployment and evaluation), and the challenge of recognition and resolution.
The authors suggest four strategies to address these pitfalls by enhancing resiliency in the systems and users. HCXAI research should include more diverse stakeholders and contexts to detect pitfalls as early as possible. In addition, design strategies should support reflection in users rather than acquiesce. Furthermore, “seamful” design, methodology that deliberately reveals often hidden information and system limitations, is promoted. See (Ehsan, Liao, et al., 2022) below for a more complete discussion of this. A final strategy, for both designers and users, is promoting “pitfall literacy” presumably as part of larger algorithmic or AI literacy initiatives.
Related to explanatory pitfalls are “imperfect XAI” (ranging from incorrect explanations to oversimplifications). Such explanations can lead to inappropriate reliance on an AI system (Morrison et al., 2023).
The launch of the initial HCXAI workshop at the ACM CHI conference created a generative space to assemble those interested in research to “amplify and extend” this emerging perspective. Two aspects of this were to position the impact and value of HCXAI at both the design and use stages, and the desire to locate the field in the larger sociotechnical dialogue.
The concept of explainability pitfalls contrasts with the manipulative intent of “dark patterns” by highlighting the unanticipated consequences caused at the design stage and revealed in downstream use. While not anticipated in the Ehsan and Riedl paper, explainability pitfalls are more relevant in the context of large language models (LLMs). As Liao and Vaughan point out, which version of the LLM is being explained (e.g., the pre-trained model, the adapted model, or the application based on either of the models) determines the explanatory requirements (Liao & Vaughan, 2023). Failure to recognize and account for downstream changes to models exacerbates explainability pitfalls.
A continuing feature of HCXAI research as it developed was to select and interpret established sociotechnical frameworks in support of the new perspective. The concept of social transparency was understood not just to passively make actions visible but as a platform for actionability where user behavior influences the explainability of the system. As Bucher notes “while algorithms certainly do things to people, people also do things to algorithms” (Bucher, 2017, p. 42). By connecting HCXAI to recognized theories, researchers sought conceptual foundations. Conversely, attempts at this stage to demarcate the field and distinguish it from other concepts resulted in a framework proposal (e.g., the LEx proposal) that was limited and premature.
6.3 Critical perspectives and disciplinary viewpoints (2022)
The 2022 Workshop explored the new territory of HCXAI as the title of that year's event described: “Beyond the Opening of the Black-Box of AI” (Ehsan, Wintersberger, et al., 2022). The questions posed in the second workshop emphasized not just “who” but the “different whos” and their contextual explainability needs. There was a focus on building an HCXAI research agenda and on recognizing the issues of the Global South. Other concerns were the negative aspects of power dynamics and weaponization, and increasing urgency regarding governance issues.
As HCXAI research continues to explore its breadth and depth, grounding the field in other critical perspectives is the focus of two papers in 2022. Drawing from software design studies and the social construction of technology, HCXAI broadened its conceptual foundations.
HCXAI prioritizes reflection over acceptance as part of an emphasis on critical assessment. In another example that differentiates HCXAI from algorithmically centered XAI, Ehsan, Liao, et al. (2022) promote the adoption of seamful design. In contrast to the seamless or frictionless experience desired for much AI, the authors recommended that AI designers adopt seamful design methods (Chalmers & Galani, 2004).
Seams are “mismatches and cracks between assumptions made in designing and developing AI systems and the reality of their deployment contexts” (Ehsan, Liao, et al., 2022, p. 5). Seamful design allows users to see the broader context of system use, deliberately reveals weaknesses as well as strengths, and makes the affordances of AI visible allowing users to discern what an AI can and cannot do. Changing the AI design process to include a seamful strategy involves “envisioning breakdowns, anticipating the seams, and designing with seams” (Ehsan, Liao, et al., 2022, p. 5). This includes proactively considering the possibility or likelihood of seams (envisioning and anticipating) and using seams as an explanatory and empowerment tool.
Seamful design, in making seams visible, supports user agency by enhancing “actionability, contestability, and appropriation.” It is a design philosophy that prioritizes user concerns regarding trust and reliance. Seamful design has been viewed as a means to “promote reluctance to simplify” (Kaur et al., 2022, p. 710) maintaining user awareness and avoiding dysfunctional results. It is an example of where HCXAI principles can inform not only post-hoc explanations but initial AI system design.
Based on the social construction of technology (SCOT) (Bijker et al., 1987), Ehsan and Riedl draw on the field of science and technology studies (STS) (Felt et al., 2016) to address an issue that has vexed XAI: what is meant by “explainability” (Ehsan & Riedl, 2022). As with other technologies, the “social construction of XAI” involves relevant social groups, diverse interpretations, and closure around norms and standards. The authors argue that while there is a consensus that XAI is about making AI outputs easy for people to understand and that this requires more than algorithmic transparency, the lack of closure around issues like the sociotechnical view and the diversity of the user (the “who”) precludes a single definition of XAI at this stage of the evolution of the field. By identifying the breadth of social groups involved in XAI and their diverse interpretations, an important contribution of this short paper is to introduce “the sociology of XAI—who is saying what, when, and why” about explainability (Ehsan & Riedl, 2022, p. 2). Studying the beliefs and behaviors of those who create XAI tools and strategies addresses a missing perspective in the sociotechnical view of HCXAI. With respect to AI, “studying those who study us” has been a longstanding interest of sociology and anthropology (Forsythe, 2001).
Through the 2022 HCXAI workshop and the two highlighted papers, the field again looked to prior research to build a conceptual foundation. By drawing from the social construction of technology, Ehsan and Riedl opens the importance self-reflection with respect to those working in the field and the biases and beliefs that shape the research agenda. Similarly, looking backwards to seamful design, HCXAI challenged the dominance of frictionless design and placed an emphasis on disclosure, inherent limitations, and development criteria.
6.4 Elaborating and responding (2023)
The 2023 Workshop declared the “Coming of Age” of HCXAI indicating a level of maturity that prompted discussions about formalizing the HCXAI research community (Ehsan, Wintersberger, et al., 2023). The questions posed for the 2023 workshop expanded the areas of interest to impact assessments and the physical world, and linked HCXAI to the Responsible AI movement. Concerns about privacy, trust, and actionability were highlighted.
As HCXAI matured as field, attention turned to elaborating core themes and addressing new challenges. Defining the sociotechnical view led to considering the sociotechnical gap.
Understanding HCXAI as a sociotechnical approach highlights the ancillary challenge of the “sociotechnical gap,” defined as “the divide between what we know we must support socially and what we can support technically” (Ackerman, 2000, p. 179). These gaps are apparent between and among technical affordances (data, model, explanation) and social or organizational needs (trust, actionability, values). By focusing on “gap understanding” rather than “gap filling,” the authors illustrate that “the process of charting the gap can improve our problem understanding, which can reflexively provide actionable insights to improve explainability” (Ehsan, Saha, et al., 2023, p. 34:2).
In the proposed framework, affordances and needs are matched with operationalization strategies: data (fact sheets, datasheets, labeling), model (model cards), explanation (XAI Question Bank) (Liao et al., 2020), trust (user surveys, interviews, design), actionability (social transparency), and values (participatory and value-sensitive design). In this study, users expressed a desire for “peripheral vision.” They wanted to know how others interacted with the AI and the explanations. Making this social and experiential information more visible to users improved engagement, surfaced “blind spots” in organizational practices, and uncovered latent organizational information that could be made actionable.
The authors note the challenges of a multistakeholder context. Issues regarding responsibility, accountability, and collective buy-in require calibrating the framework to adjust for lesser or greater social transparency, levels of complexity both technically and socially (or organizationally), and the need to recognize personal consequences.
Through the lens of the sociotechnical gap, this paper brings together several HCXAI themes: actionability, social transparency, “blind spots,” and the notion of “peripheral vision.” It also broadens the explanatory context from the typical one to one engagement of a user with an explanation to an organizational setting where groups of co-workers engage with explanations both individually and collectively.
A HCXAI exploration of the sociotechnical gap in workplace environment explored the concept of sociodigital sovereignty, described as the confluence of transparency and explainability, confidence of action (efficiency), and freedom of action (divergence) (Schmuntzsch & Hartmann, 2023). Where any of these three characteristics was missing or diminished, users' sovereignty (actionable agency) was compromised. As a result, users “expressed fear of losing control because of not knowing what the machine is doing and why, but still being responsible for the result” (Schmuntzsch & Hartmann, 2023, p. 499). This research supports recommendations regarding “human-in-the-loop” approaches XAI where human knowledge and engagement is necessary for initial design and ongoing operation (Tocchetti & Brambilla, 2022).
The “coming of age” theme of the 2023 HCXAI workshop announced, perhaps precipitately, that the field had reached a significant level of maturity. By connecting its research to the larger Responsible AI agenda, HCXAI sought a place in this discussion that legitimized its role and importance. Bolstering this is the incorporation of the sociotechnical gap, once again drawing on prior research. In the context of HCXAI, the sociotechnical gap brings together the core HCXAI issues of actionability, seamfulness, and reflection rather than acquiescence and establishes a fundamental conceptual pillar of the field.
7 SPECIFIC ISSUES AND CHALLENGES
The papers discussed previously created a foundation for HCXAI research. The following section selects a few of the issues and challenges arising from that work and reflects on how the research community responded.
7.1 HCXAI design principles
Operationalizing HCXAI has been a focus from the outset (Ehsan & Riedl, 2020; Ehsan, Wintersberger, et al., 2021). Providing principles or guidelines for XAI system designers are a means to translate ideas into practice. Four papers (Hoffman, Miller, et al., 2023; Laato et al., 2022; Mohseni et al., 2021; Mueller et al., 2021) offer different perspectives on HCXAI system design.
The design principles of HCXAI presented in Mueller et al. (2021) and Hoffman, Miller, et al. (2023) are related, with four authors appearing on both papers. While the latter paper is more detailed, both take a cognitive approach to XAI. Among the principles are the importance of context (regarding user objectives, decision consequences, timing, modality, and intended audience), the value of using hybrid explanation methods that complement and extend each other, and the power of contrastive examples and counterfactual approaches. The principles emphasize that explanations cannot be “one-off.” HCXAI systems and users enter into relationships that can span limited encounters to ongoing interactions.
A key principle in both papers is the centrality of “self-explanation.” It is “the Golden Rule of XAI systems: Explain unto others in such a way as to help them explain to themselves” (Hoffman, Miller, et al., 2023). Users should be encouraged and empowered to formulate their own explanations and construct their own testable hypotheses.
A significant difference between these papers is that Mueller et al. (2021) encourage developers to “build explanatory systems, not explanations” since “rarely” does an initial explanatory response provide a “useful explanation.” Explanatory systems include additional information, tutorials, alternative interfaces, and other components. This supplementary information and these tools bolster the recommendation of “active self-explanation.” The idea of “explanatory systems” opens the possibility that these systems are independent of the platforms being explained (Ridley, 2023).
Another key difference is that Hoffman, Miller, et al. (2023) emphasize that explaining is an “exploratory activity” or a learning process for the user. As such this process should be guided by an explicit pedagogical model. Designers are encouraged to emulate the findings of intelligent tutoring systems (Clancey & Hoffman, 2021).
While Mohseni et al. (2021) address a broad audience for HCXAI, a specific area of discussion is for users defined as “AI novices.” These are “end-users who use AI products in daily life but have no (or very little) expertise on machine learning systems” (Mohseni et al., 2021, p. 24:12). These users are widely seen as the principal focus of HCXAI.
The main design goals for an HCXAI system for end-users includes algorithmic transparency (by exposing how the system works and facilitating the improvement of mental models), user trust and reliance (by assessing reliability and calibrating system accuracy), bias mitigation (through explanations that can expose or question bias), and privacy awareness (by assessing data privacy).
Identifying candidate explanation formats for the targeted system and user group is the first step to deliver machine learning explanations to end-users. The design process can account for different levels of complexity, length, presentation state (e.g., permanent or on-demand), and interactivity options depending on the application and user type. (Mohseni et al., 2021, p. 24:27)
a series of user-centered evaluations of explainable interface with multiple goals and granularity levels could be performed to measure the following: (1) User understanding of explanation, (2) User satisfaction of explanation, [and] (3) User mental model of the intelligent system. (Mohseni et al., 2021, p. 24:28)
Evaluation frameworks for HCXAI are underexamined. Citing the lack of “evaluation procedures [for HCXAI] that capture the complexity of the human-AI interaction,” Donoso-Guzmán et al. augment a prior framework (Knijnenburg & Willemsen, 2015) with components that include personal and situational characteristics (Donoso-Guzmán et al., 2023). However, “we did not analyse how specific situational and personal characteristics affect the properties” (Donoso-Guzmán et al., 2023). The very elements that distinguish HCXAI were identified but not examined making this framework embryotic at best.
In comparison to Mueller et al. (2021) and Hoffman, Miller, et al. (2023), who provide a checklist with a corresponding scorecard to determine effectiveness (Hoffman, Jalaeian, et al., 2023; Klein et al., 2021), Mohseni et al. take a more inclusive approach with a set of guidelines that define not only the content of an HCXAI system but the processes to create one.
Laato et al. provide a comprehensive set of XAI design recommendations drawn from a survey of the literature and grouped according to when, how, and what to explain. An overarching recommendation notes that “context is everything—There is no one-size-fits-all type of solution” (Laato et al., 2022, p. 14). Within the three main categories, specific recommendations provide design criteria.
The “when” category has only one recommendation: on demand. Not only does this mitigate the potential intrusiveness of explanations, but it also clearly places the user in control of their needs. The “how” category includes recommendations about personalization, preference for visualization, and the importance of aligning with user mental models. It also includes some less common recommendations such as the value of metaphors in explanations, using explanations to strengthen user curiosity, and being aware of “trade-offs,” acknowledging the potential negative aspects of explanations. The “what” category recommends explaining only the selective parts of the AI system of specific interest to a user, responding differently to unfavorable decisions, and using various methods to signal system uncertainties, weaknesses, and strengths.
Complementing HCXAI design principles has been the emergence of software tools to facilitate development. One such tool, Explainability in Design (EID), adopts the principles of value sensitive design and provides a standard workflow with specific deliverables for XAI software creation (Zhang & Yu, 2022).
7.2 Actionability
Actionability of explanations is a central concern of HCXAI (Ehsan & Riedl, 2020) underscoring the primary objective of “whether the explanation generated is of any use” (Wegener & Cassens, 2021). Explanations are viewed as a starting point because “user interactions do not end at receiving an XAI output, but continue until an actionable understanding is achieved” (Liao & Varshney, 2022, p. 12). However, “the underlying assumption that explanations are useful to users” (Mansi & Riedl, 2023) is challenged by research illustrating that a “positive attitude towards the explainable method does not mean they will accept or implement the explanations” (Zhao & Turkay, 2023).
By linking specific XAI strategies to potential user actions, three types of actions were identified: mental state actions (actions taken as a user learns and increases their understanding), XAI interactions (using interface features and engaging with the system), and external actions (actions outside system such as reaching out to another person or system, or simply ceasing to use the system) (Mansi & Riedl, 2023). The authors found that “while the user has a broad range of explanation types and resulting Mental State Actions, there are surprisingly limited potential XAI Interactions or External Actions.” Actionability was restricted in ways that a user could influence or act upon the system and in ways the user could engage other resources.
Related to actionability is contestability (Ehsan, Liao, et al., 2021), the function of explanations to empower user critiques of AI. Contestability in this context requires “explanations that go beyond outcomes and also capture the rationales that led to the development and deployment of the algorithmic system in the first place” (Yurrita et al., 2023). In a feminist analysis, the authors argue that HCXAI must support “response-ability” enabling users to challenge the system with particular emphasis on marginalized perspectives, systemic issues, and intersectionality (Klumbytė et al., 2023). The contestability of users must be informed by the “entirety of that socio-technical process” (Wieringa, 2020, p. 10).
Actionability has also been linked to causality suggesting that an “XAI system can accomplish everyday causal understanding of users” allowing users to make informed actions (Kim et al., 2022, p. 124). Causality is complex and illusive (Pearl & Mackenzie, 2018) and seldom discussed in the context of HCXAI. However, for some, “a non-causal explanation is not actionable; for this reason we believe causality in explanations to be core to achieving both user-centered and honest transparency” (Mei et al., 2023).
7.3 Datasets and performance metrics
Vaughan and Wallach (2021) argue for including explanations of datasets used as training data and system performance metrics, typically used only in algorithmic XAI, as key aspects of HCXAI. Performance metrics may include system accuracy, precision, and recall, which suitably presented, can inform user trust. System errors, supplied through a record of the system use, can be equally beneficial if difficult to convey.
The explanatory value of datasets has received considerable attention (Ali et al., 2023). While making explanatory datasheets and data cards available is recommended (Gebru et al., 2021; Pushkarna et al., 2022), doing such requires rules and norms for effective implementation (Mei et al., 2023) given the possible negative consequences of dataset explanations (Anik & Bunt, 2023). Interactive model cards are proposed as a more effective way to engage non-expert users in support decision-making and transparency. Viewed as “a bridge to further analysis” and a way to “bolster productive skepticism” (Crisan et al., 2022, p. 437), interactive data cards echo the HCXAI objective of “reflection instead of acceptance.” However, in an example where this proposal falls short of HCXAI, “unintended uses, ethics, and safety are too ambiguous to be actionable” and were not incorporated into the interactive data cards (Crisan et al., 2022, p. 437).
In a related proposal, consumer labels for machine learning systems, analogous to nutrition labels, could provide easy to understand information for non-expert end-users (Seifert et al., 2019). Label contents would include, among others, performance indicators, generalization expectations, robustness, privacy sensitivity, and accountability. However, once again, factors central to HCXAI such as transparency and social acceptability were deemed “inherently subjective and can therefore not be captured in a consumer label” (Seifert et al., 2019, p. 178).
7.4 Pedagogy and learning theories
Learning theories have been proposed to strengthen HCXAI by viewing explanations as “learning artifacts” (Cortiñas-Lorenzo & Doherty, 2023). As opposed to algorithmic XAI which proposes a technique and then evaluates its usefulness, the learner-centered XAI framework starts with learning objectives for a specific context and then defines learning measures, activities, and evaluation of outcomes before seeking specific XAI techniques (Kawakami et al., 2022). A learner-centered approach requires refocusing our efforts on the explainee rather than the explanation” with Cortiñas-Lorenzo and Doherty suggesting that from a “learning angle, explaining AI system's algorithms may not even be necessary.” Instead of the technical details of the system, the focus should be on “what can really augment human skills and have a lasting impact on human learning and growth.” The HCXAI principle of “self-explanation” (Hoffman, Miller, et al., 2023) or hypothesis development (Miller, 2023) are further reinforced noting that “past work has shown that generating explanations can be more effective in learning than actually receiving explanations” (Cortiñas-Lorenzo & Doherty, 2023).
8 HCXAI REINTERPRETATIONS
Three papers explore reinterpretations of HCXAI by proposing different ways to view explanation and different techniques to address human-centered outcomes: Broad-XAI (Dazeley et al., 2021), Explanatory AI–YAI (Sovrano & Vitali, 2022), and Evaluative AI (Miller, 2023).
These explanations do not attempt to understand the social context of the explainee and do not usually find a generalisation over the causes or counterfactuals. As AI systems increasingly become integrated into our everyday society, simply explaining a single decision point without the historical origins of that action or its social/cultural context will not carry sufficient meaning. (Dazeley et al., 2021, p. 6)
In response, Broad-XAI addresses social contexts and cognitive processes by defining levels of explanation. These levels, from the lowest “reactive” response to the highest “meta” level, include other levels that focus on social contexts and cultural expectations.
Aligning with proposals for a “process-centric XAI” (Yurrita et al., 2023) and an “AI lifecycle” approach (Dhanorkar et al., 2021), Dazeley et al. propose a “Meta-explanation” that “rather than explaining why a decision is the correct decision, it explains the problem solving process in solving it” (Dazeley et al., 2021, p. 16). Broad-XAI provides the factors and processes that result in an explanation in both social and cognitive contexts. However, the authors “believe it is unlikely that a Meta-explanation will be provided by certain systems … Forcing companies to provide a Meta-explanation is an issue to be taken up with governments—potentially the same ones that will be unwilling to provide such details themselves” (Dazeley et al., 2021, p. 17).
An important model to rearchitect the HCXAI explanatory framework proposes a set of explanatory tools that interface between XAI tools and the user. The tools of Explanatory AI (YAI) are “the missing connection between XAI and human understanding of complex behaviours of digital systems” (Sovrano & Vitali, 2022). In this HCXAI model, XAI techniques generate explanatory information and YAI is a toolset that selects from that information the most relevant and useful explanations for users. XAI creates an “explanatory space” and YAI creates “goal-driven paths” through it. The objective is to “disentangle making things explainable (i.e., XAI) from explaining (i.e., YAI) … separating the presentation logic from the application logic,” where presentation logic provides the personalized explanations from the explainable information (Sovrano & Vitali, 2022). In “explaining the explanation,” Sovrano and Vitali envision a conversational system that engages a user during “the evolution of the task for which the explanation is required” seeking answers that match the kind of questions documented in the XAI Question Bank (Liao et al., 2020).
Although not explored in detail, an important aspect of Explanatory AI is that the YAI tools are envisioned as “somehow independent and separate” from the XAI tools. This independence is seen to mitigate concerns about platforms providing manipulative or deceptive explanations. One option to create this independence is to create XAI and YAI protocols and standards forming a networked architecture for explainability (Ridley, 2023).
Motivated by the “unfounded” view that users act on explanations, Miller critiques the “recommend and defend” approaches to XAI which limit user agency and do not align with the cognitive processes people use when making decisions or judgments. In response, Evaluative AI uses hypothesis-driven methods which draw from the explanation criteria in his earlier work (Miller, 2019). Evaluative AI “tools do not provide recommendations …. [instead they support explanation] by either allowing the decision maker to determine which options are best or helping them to narrow down to a manageable set of options” (Miller, 2023, p. 334). The emphasis is on “evidence to support or refute human judgements, and explain trade-offs between any set of options” (Miller, 2023, p. 334). This approach resembles the “explanatory systems” from (Mueller et al., 2021; Sovrano & Vitali, 2022).
Despite the view that this HCXAI method is more aligned with human processes and expectations, Miller concedes that “it is difficult to imagine evaluative AI solutions will result in lower cognitive load” (Miller, 2023, p. 341). As a result, adoption by designers and users alike may be limited.
Three themes from these reinterpretations identify key strategies and challenges for HCXAI as it evolves. Sovrano and Vitali, and Miller both recommend explanatory systems like those proposed by others (Chromik & Butz, 2021; Mueller et al., 2021). The explanatory system approach recognizes that individual explanations are insufficient for addressing the complexities of the AI, the sociotechnical context, and the diversity of users. The “AI Assistant” (Dikmen & Burns, 2022) and the “Explanation Assistant” (Huynh et al., 2022) are two examples of such systems but further development is an important objective for HCXAI.
Dazeley et al. and Sovrano and Vitali encourage HCXAI designers to view explanations derived not from isolated parts or groups of characteristics but rather as the result of an end-to-end understanding of the AI system from initial creation through to downstream use in specific contexts. This contextual, lifecycle view further recommends the adoption of explanatory systems and recognizes the ongoing relationship users will have with AI systems.
Both Dazeley et al. and Miller have concerns that despite the efficacy of their proposals they will be resisted by the AI industry, governments, and even by lay end-users, the very people HCXAI is most designed to assist. The challenge of cognitive load and the competing interests of key stakeholders (e.g., industry and government) and users remain ongoing themes for HCXAI and potential weaknesses for its adoption.
9 LEGISLATION, REGULATION, AND THE RIGHT TO EXPLANATION: HCXAI GOVERNANCE
How to operationalize HCXAI has been a constant theme. Doing so would embed HCXAI principles and practices in areas such as research protocols, system design, evaluation, and oversight. Largely undiscussed are legislative or regulatory approaches to further HCXAI. Governmental or governance tools are mechanisms to establish norms and promulgate best practices.
Regulating technology has been fraught with concerns about responses that are either “too late” or “too soon” and obligations that are viewed as either “too little” or “too much” (Aspray & Doty, 2023). This is largely a trade-off between innovation and anticipated economic opportunities, and public interest in protection and fairness. More recently it has been suggested that “artificial intelligence will become a major human rights issue in the twenty-first century” (Noble, 2018, p. 1). Making AI, and by extension XAI, a rights issue both elevates and broadens the discussion.
Whether reflecting the general perspective of XAI or the specific view of HCXAI, the requirement for an explanation of an AI system has featured in various regulations or other governance tools. Given the role of explanation to foster trustworthiness and support accountability, particularly for the non-expert, lay public, exploring if these initiatives have been effective and sufficient is important and will provide guidance on how improvements can be made.
Despite the early promise of the European Union General Data Protection Regulation (GDPR) (European Union, 2016), subsequent legislation, regulation, codes of conduct, and other mechanisms have fallen short. However, recently the centrality of explanation in the AI regulation debate was recognized by politicians. Chuck Schumer, Senate Majority Leader in the US Congress, called explainability “one of the thorniest and most technically complicated issues we face, but perhaps the most important of all. Explainability is about transparency” (Schumer, 2023).
This urgency regarding explainability is, in part, a reaction to the emergence of large language models. Addressing the lack of transparency of proprietary LLM models “may not be possible without policy and regulatory efforts that enforce transparency requirements on LLM creators and providers” (Liao & Vaughan, 2023, p. 4).
This section will examine the “right to explanation” and the role of explanation in current and emerging AI legislation (notably in the EU, US, and UK). If HCXAI is to realize its full potential, it must be reflected in these initiatives.
9.1 The right to explanation
The idea of a right to an explanation from an AI system began with the 2018 approval of the EU's GDPR. While most of the regulation concerns topics such as data access, retention, privacy, and erasure, there are references to what has been called “the right to explanation” (although that specific phrase does not appear in the regulation).
The presence of this right in the GDPR has been both detailed (Goodman & Flaxman, 2017; Kaminski, 2019) and disputed (Edwards & Veale, 2017, 2018; Wachter et al., 2017). Articles 13(2)f and 14(2)g of the GDPR require that the user have access to “meaningful information about the logic involved.” In addition, Recital 71 (a recital is not formally part of the law and is non-binding) requires that users be able “to obtain an explanation of the decision reached … and to challenge the decision.” These sections form the basis for the notional right to explanation and align with HCXAI principles regarding disclosure, actionability, and contestability.
For all its shortcomings and lack of clarity, the notion of a right to explanation in the GDPR shifted explainability from a purely technical issue to one with an additional and urgent focus on public policy. However, it is significant to note that while HCXAI research makes note of the GDPR and the right to explanation (Ehsan, Passi, et al., 2021; Liao & Varshney, 2022; Mohseni et al., 2021; Mueller et al., 2021; Sokol & Flach, 2020), this right or policy is not pursed further in statements of HCXAI principles. Serious objections are raised suggesting that since explanations are “essentially inadequate” and place an “unreasonable burden” on users, Knowles concludes that “explanations are not fit for lay public consumption and therefore must never be used as a means of enabling informed consent for increasingly pervasive AI” (Knowles, 2022). Rather than empowering users, legislation requiring something like a right to explanation “will succeed in legitimizing new regimes of control, including the expansion of punitive and surveillance based AI” (Knowles, 2022).
9.2 Explainability in legislation and regulation
The debate about the necessity of explainability as a central regulatory component that began with the GDPR continues as numerous countries address the question of AI regulation and the role of explanation.
A comparison of explainability in regulatory documents from the European Union, the United States, and the United Kingdom reveal a disappointing situation (Nannini et al., 2023). In the EU “the GDPR and the proposed AI Act do not contain clear requirements for interpretability and explainability for end users.” In the US “the Algorithmic Accountability Act draft includes some provisions for end-user rights to explanation, but the policy focus remains on innovation.” In the UK “the regulatory landscape for AI is limited to data protection, and there is a lack of clear enforcement or standardization in the area of explainability.” Overall, the authors found that “policy trajectories prioritize AI innovation under a risk management lens rather than truly empowering users with explanations … [the documents] fall short in addressing the complexity and sociotechnical impact of explainable AI, leading to missed-opportunities over its development and implementation” (Nannini et al., 2023, p. 1207).
However, the lack of inclusion is, in part, a failure of XAI research to provide operational, and legally binding, definitions of a good explanation, evaluation frameworks, and practical implementation strategies for XAI (Nannini et al., 2023; Panigutti et al., 2023). Advancing explainability in regulations under these circumstances could cause legislators to “overestimate the capabilities of XAI and as a result overload the regulation with unachievable, highly detailed design prescriptions” (Gyevnar et al., 2023, p. 7).
Others question “whether, and to what extent, can end-user Explainability satisfy the right to explanation of AI systems' requirements by law” (Rozen et al., 2023, p. 1). It is not just the inadequate state of XAI but rather whether explainability, as presented by XAI, aligns with legal notions of explanations “which include promoting a better and more just decision, facilitating due-process and acknowledging human agency” (Rozen et al., 2023, p. 9).
9.3 Can HCXAI be regulated?
If AI legislation is emerging but with only marginal attention to explainability, where does this leave XAI and the specific mandate of HCXAI? It may be that explainability as envisioned in HCXAI is not conducive to regulation (Gyevnar et al., 2023; Nannini et al., 2023; Rozen et al., 2023). While AI legislation and regulation are poised to address risk and innovation, advancing the HCXAI agenda may be best addressed through consumer protection legislation.
Seamful design, explanatory pitfalls, and performance metrics point to disclosure requirements that make consumers aware of limitations, “side effects,” and product efficacy. Actionability and contestability principles align with consumer rights to complaint, recourse, mitigation, and remedial processes regarding services that inadequately perform. The principle of self-explanation through the provision of supplementary information is consistent with requirements for consumer education and product labeling.
With respect to explainability, with “appropriate modification … consumer protection seems able to retain its traditional values and in many instances its traditional form” (Howells, 2020, p. 171). However, efforts to link HCXAI to consumer protection are rare. In a study of consumer attitudes towards sustainable and transparent, explainable AI, the authors conclude that “it seems doubtful that simply placing the burden on ‘the informed consumer’ will lead to a demand for transparent and sustainable AI” (König et al., 2022, p. 10).
Even though the use of AI in various domains “is a market driven pre-condition of the digital everyday” (Kant, 2020, p. 214), the explanatory needs and protections of the lay, non-expert user of these systems seem to be forgotten. The prospects of HCXAI principles being embedded in consumer or other legislation seem remote.
10 FUTURE RESEARCH AND CHALLENGES FOR HCXAI
Numerous challenges and research requirements for HCXAI have been identified in the review literature. These include such fundamental issues as a common agreement on what “explainability” means (Arrieta et al., 2020; Haque et al., 2023; Saeed & Omlin, 2023) and better methods to measure the effectiveness of explanations (Gunning et al., 2021; Saeed & Omlin, 2023). Other desiderata include better explanation interfaces (Das & Rad, 2020; Haque et al., 2023; Saranya & Subhashini, 2023), understanding the trade-off between model performance and explainability (Arrieta et al., 2020; Saeed & Omlin, 2023), and the need for comprehensive governance and regulatory regimes (Ali et al., 2023; Haque et al., 2023; Liao & Vaughan, 2023).
More generally, some challenges are a result of material contexts such as robotics (Anjomshoae et al., 2019) and autonomous vehicles (Graefe et al., 2022). Others emerge from new or evolving computational models, for example neurosymbolic computing (Pisano et al., 2020) and reinforcement learning (Puiutta & Veith, 2020). Augmented reality (AR) and virtual reality (VR) systems pose unique explanatory challenges (Xu et al., 2023). Another area of concern is to recognize the Western bias inherent in much XAI research and the need for perspectives from the Global South (Bhallamudi, 2023; Okolo, 2023). Despite advances in the sociotechnical perspective of HCXAI, for some the need remains better computational models: “the future of human-centric XAI is neither in explaining black-boxes nor in reverting to traditional, interpretable models, but in neural networks that are intrinsically interpretable” (Swamy et al., 2023).
Three areas are reviewed as centrally important to the evolution of HCXAI: system and tool development, explanations for large language models, and the future of HCXAI as a field.
To date, HCXAI has been mostly about critical analysis, and unlike algorithmic XAI, less about system and tool building. One way to advance system development while honoring the principles and values of HCXAI is to adopt “critical making.” This perspective combines “critical thinking, typically understood as conceptually and linguistically based, and physical ‘making,’ goal-based material work” (Ratto, 2011, p. 253). Critical making in the context of HCXAI would create demonstration projects and various software deliverables that operationalize HCXAI and provide tangible, experimental platforms to formally evaluate their success.
Two areas offer potential for critical making. While HCXAI emphasizes the user and their context, personalization techniques are underexplored (Conati et al., 2022; Tambwekar et al., 2023) with only limited implementations (Lai et al., 2023). Prototyping the explanatory systems recommended by various researchers (Chromik & Butz, 2021; Miller, 2023; Mueller et al., 2021; Ridley, 2023; Sovrano & Vitali, 2021) offers another constructive opportunity for critical making. The implementations by Dikmen and Burns (2022) and Huynh et al. (2022) are encouraging but this area warrants further work.
The complexity and opacity of large language models (LLMs), also known as foundation or frontier models, has led some to suggest that these models “cannot be made explicable in any meaningful way to a human being or even to experts” (Pinhanez, 2023) resulting in a “post-explainability world” (Sarkar, 2022). With respect to these explanatory challenges and the role of HCXAI, Ehsan and Riedl (2023) ask “will human-centered XAI solve all our problems? No, but it will help us ask the right questions.” Those questions have elicited some HCXAI strategies for LLMs.
Viewing the challenge as “narrowing the socio-technical gap,” Liao and Xiao focus on “informational transparency” with four approaches: “model reporting, publishing evaluation results, providing explanations, and communicating uncertainty” (Liao & Vaughan, 2023, p. 1). Reinforcing the core perspective of HCXAI, the authors observe that “achieving an end goal may require information beyond details of the model, such as information about the domain and the social-organizational context the model or application is situated in” (Liao & Vaughan, 2023, p. 6).
Noting with concern that “many of the same paths tread by the XAI community over the past decade will be retread when discussing LLMs,” Datta and Dickerson argue that “humans' tendencies—again, complete with their cognitive biases and quirks—should rest front and center” (Datta & Dickerson, 2023). The focus should be on mental models, use case utility, and cognitive engagement.
A different perspective is provided by Mavrepis et al. in a preprint and through an LLM created by ChatGPT Builder. Rather than an LLM to be explained, this paper presents LLM as a tool to explain. The GPT model “x-[plAIn]” is an LLM designed to “generate clear, concise summaries of various XAI methods, tailored for different audiences” (Mavrepis et al., 2024). For “beginners,” those who self-identify as having limited AI expertise, the model interprets the output of widely used algorithmic XAI techniques (e.g., LIME, SHAP) with accessible descriptions that provide actionable insights for the user. While limited in scope, x-[plAIn] offers the opportunity of HCXAI enabled through an LLM.
With respect to LLMs, we are asked to “reload our expectations on XAI … centering the design and evaluation around the human. This positioning can reveal unmet needs that must be addressed while avoiding the costly mistake of building XAI systems that do not make a difference” (Ehsan & Riedl, 2023).
From a disciplinary perspective, perhaps one of the most important challenges for HCXAI is how it will evolve. HCXAI emerged from the limited perspectives of algorithmic XAI. It not only focused on who explainability was for but took a broader sociotechnical perspective where context in all its dimensions was central. While algorithmic XAI still dominates the vast and still growing XAI field, HCXAI has led and supported the wider adoption of human-centric XAI.
One direction would see HCXAI continuing as a vital but specialized community. The ACM CHI HCXAI workshop could grow and gain greater influence, and more formal publishing vehicles could emerge (i.e., an HCXAI journal for example). It could align more with HCI, its nearest disciplinary neighbor, and create affiliations with related areas such as Science and Technology Studies or Critical Algorithm Studies (Kitchin, 2017). The compelling perspectives of HCXAI would continue to evolve with increasing authority.
Another path would see the principles, values, and objectives of HCXAI incorporated into mainstream XAI with human-centric work continuing in and alongside existing XAI fora. In this direction HCXAI would simply be an understood component of XAI and cease to exist as a distinctive perspective.
Whatever the trajectory of HCXAI, it has successfully refocused XAI on human-centered perspectives and made XAI more relevant beyond its technical base. Its future likely depends on whether its research community still perceives unaddressed issues and applications, and whether society at large continues to value the principles HCXAI espouses.
11 CONCLUSION
Latanya Sweeney, Director of the Public Interest Tech Lab at Harvard, notes that “technology designers are the new policymakers; we didn't elect them but their decisions determine the rules we live by” (Sweeney, 2018). Those who theorize, design, create, and critique XAI systems are among these new policymakers. The ideas and values that guide their work inform our “algorithmically mediated” (Anderson, 2020) lives.
Human-centered explainable AI (HCXAI) was motivated by the lack of attention by algorithmic XAI to the user and their context. Along with the larger efforts regarding human-centered data science (Aragon et al., 2022), and human-centered AI (Shneiderman, 2022), HCXAI puts humans, not technology, at the center of its considerations.
The emergence of HCXAI from XAI and HCI has resulted in a growing literature that attempts to influence designers, policymakers, and perhaps most importantly, the lay public who are the main users of these systems. It is grounded in views that prioritize these users and understands them within the sociotechnical context of their lives. As such, HCXAI is both a technical and a public policy priority. This review identified the foundational ideas of HCXAI, how those concepts could be operationalized in system design, how legislation and regulations might normalize its objectives, and the challenges that HCXAI must address as it matures as a field.
The work of Upol Ehsan and Mark Riedl is featured widely in this review. This could lead to several conclusions. Is HCXAI simply the research agenda of two people? Is HCXAI defined exclusively by their work? Does the focus on their work as foundational ignore other similar contributions equally relevant to a human-centered XAI approach?
This review of HCXAI takes the position that the 2020 Ehsan and Riedl paper broke new ground in XAI not by just identifying limitations but by constructing an integrated view of how a human-centered approach could be different and valuable. This paper, their subsequent work, and the workshops that followed, drew in others who amplified and extended this work. Several papers (see section 8) reinterpreted HCXAI in different ways, in essence challenging the orthodoxy of the foundational work.
HCXAI is not just a personal agenda, but it was animated by the work of Ehsan and Riedl. Positioning their papers as foundational provides a narrative that contextualizes the work of others and tracks the maturation of this subfield of XAI.
The HCXAI research community, in comparison to XAI generally, is small and striving for influence. The limited and uneven adoption of explainability in current and emerging legislation and regulations suggests that the objectives at the core of HCXAI have not yet been operationalized and grounded in policy. However, with the release of ChatGPT and other generative AI systems, concerns about trust and accountability have become public debates. Explainability was identified as a core element in the six “grand challenges” of human-centered AI (Garibay et al., 2023). This is the essential groundswell that might propel HCXAI forward and make “where the humans are” a central feature of XAI and a public policy priority.