How to design responsible generative AI?

1. Introduction

The increasing adoption of generative AI, particularly large language models (LLMs), has reignited the debate on AI regulations to ensure that AI/ML systems are trained and deployed responsibly. Unfortunately, this effort is complicated by the fact that different government organizations and regulatory bodies publish their own guidelines and policies, with little to no agreement on the definitions of terms. The specification language is also (sometimes intentionally) maintained at such a high level that it is difficult to align it with an implementation and enforcement mechanism. For example, European AI law imposes a different set of dos and don'ts depending on the "risk level" of an AI application. However, quantifying the level of risk of an AI application is easier said than done because it depends on multiple parameters, and essentially requires classifying how the capabilities of a non-deterministic system will impact users and systems likely to interact with it in the future.

Thus, rather than trying to understand and regulate all types of AI systems, we take a different (and practical) approach in this article, namely to transparently describe the capabilities of the AI ​​system and set realistic expectations about what it can (or cannot) do.

For example, this would mean that the AI ​​system provider transparently states that a model has been trained on data containing certain demographic characteristics and that it can only provide predictions for users with those demographic characteristics—users with other demographic characteristics will require manual processing. In other words, the AI ​​system provider also specifies the scenarios in which (in their opinion) the system might make mistakes and recommends a "safe" approach with safeguards for those scenarios.

Intuitively, we can think of the regulatory approach as an open-world problem, where we try to regulate all the different ways a system can behave (even in the future). In contrast, the transparency proposal takes a silo approach, in which it attempts to frame the known capabilities of the system, so that end users and downstream systems can make a more informed decision.

The "transparency" approach also shifts responsibility to companies/vendors, who need to be more transparent about the internal details of their AI systems.

The two approaches can of course work in a complementary way, providing a more robust framework – allowing the user to have confidence in the system.

2. Responsible AI

"Ethical AI, also known as responsible (or trustworthy) AI, is the practice of using AI with good intentions to empower employees and businesses, and have a fair impact on customers and society. Ethical AI enables businesses to build trust and develop AI with confidence." [1]

In the following sections, we briefly present some of the key aspects of responsible AI: explainability, bias/fairness, and accountability – before introducing the design patterns of the Gen AI solution. The fourth aspect is that of "data privacy," which has already been sufficiently covered in our opinion [2], and there are mature practices in place within companies to address these concerns.

2.1 Explainability

Explainable AI is an umbrella term for a range of tools, algorithms, and methods that accompany AI model predictions with explanations. Explainability and transparency in AI models are clearly at the top of the list of "non-functional" AI characteristics that businesses should prioritize. For example, this means having to explain why a machine learning model profiled a user as belonging to a specific segment—leading them to receive advertising. This aspect is also covered by the "right to explainability" in most regulations; for example, the paragraph below is quoted from Singapore's AI governance framework:

"It's worth noting that technical explainability isn't always enlightening, especially for the average person. Implicit explanations of how AI model algorithms work can be more helpful than explicit descriptions of the models' logic. For example, providing someone with counterfactuals (such as 'you would have been approved if your average debt had been 15% lower' or 'these are users with profiles similar to yours who received a different decision') can be a powerful type of explanation that organizations might consider."

The EU's GDPR also covers the "right to explainability" – see the articles below:

Limits to decision-making based solely on automated processing and profiling (article 22) Right to receive meaningful information about the logic involved in the decision (articles 13 and 15).

It is important to note that the GDPR does not impose a "right to explainability," but rather a "right to information." The GDPR permits fully automated decision-making as long as personal data is not involved and the objective is not to assess a user's personality—human intervention is necessary in such scenarios.

AI/ML practitioners know that data and source code visibility is not synonymous with explainability. Machine learning (deep) algorithms vary in the level of accuracy and explainability they can provide—unsurprisingly, the two are often inversely proportional.

For example, fundamental ML algorithms, such as logistic regression and decision trees, are easier to explain because it is possible to trace the weights and coefficients of the independent variables, as well as the various paths from the nodes to the leaves of a tree. It can be noted that "explainability" becomes more difficult as one moves towards random forests, which are essentially a collection of decision trees. At the extreme end of the spectrum are neural networks, which have demonstrated human-level accuracy. It is very difficult to establish a correlation between the impact of a (hyper)parameter assigned to a layer of the neural network and the final decision—in a deep (multilayer) neural network. This is also why optimizing a neural network currently remains a very ad-hoc and manual process—often based on the intuition of the data scientist [3].

It is also important to understand that an "explanation" can have different meanings depending on the user.

"The important thing is to explain the right thing to the right person, in the right way and at the right time" [4]

The right level of abstraction of explanations depends on the objective, domain knowledge and abilities to understand the complexity of the topics. It's fair to say that most of today's explainability frameworks are aimed at AI/ML developers.

Improving model explainability is an active area of ​​research within the AI/ML community, and significant progress has been made in model-agnostic explainability frameworks. As the name suggests, these frameworks separate the explainability of the model, attempting to correlate the model's predictions to the training data, without needing to know the internals of the model.

For example, one of the best-known model-agnostic explainability frameworks is Local Interpretable Model-Agnostic Explanations (LIME). LIME is "a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful way by learning a locally interpretable model around the prediction." LIME provides easy-to-understand (approximate) explanations of a prediction by training a sample-based explainability model around that prediction. It then weights these samples according to their closeness to the original prediction. The approximate nature of the explainability model may limit its use for compliance purposes.

For example, the image below shows the LIME output of an artificial neural network (ANN) trained on a subset of California housing price data. It shows important features that have a positive or negative impact on the model prediction.

As noted, this is an active research area, which continues to progress with the publication of new frameworks and explainability tools (sometimes specific to a model).

2.2 Bias and fairness

[5] defines AI/ML bias "as a phenomenon that occurs when an algorithm produces results that are consistently detrimental due to flawed assumptions in the machine learning process."

Biases in AI/ML models are often unintentional, but they have been observed far too frequently in deployed use cases to be ignored. Google Photos labeling photos of a Black Haitian-American programmer as "gorilla," or more recently, images of "White Barack Obama," are examples of AI models discriminating based on gender, age, sexual orientation, and so on. The unintentional nature of these biases won't prevent your business from being fined by regulatory bodies or facing negative public backlash on social media, resulting in lost business. Even without the aforementioned repercussions, it is simply ethical for AI/ML models to behave fairly toward everyone, without any prejudice. However, defining "fairness" is easier said than done. Does fairness mean, for example, that the same proportion of male and female candidates achieve high scores in risk assessment? Or that the same level of risk results in the same score, regardless of gender? It is impossible to answer both of these definitions at the same time [6].

Biases creep into AI models, primarily due to inherent biases already present in the training data. The "data" aspect of AI model development is therefore crucial for combating biases.

The focus is on the types of bias related to "training data",

Historical bias: It is due to the historical inequality of human decisions captured in the training data.

Representation bias: it is due to training data that is not representative of the real population.

Measurement and aggregation bias: It is caused by inadequate selection and combination of characteristics.

A detailed analysis of the training data is necessary to ensure that it is representative and evenly distributed across the target population, with respect to the selected features. Explainability also plays an important role in detecting bias in AI/ML models.

In terms of tools, TensorFlow Fairness Indicators (link) is a good example of a library that makes it easy to calculate commonly identified fairness metrics.

It is important to mention here that bias detection is not a one-time activity. As with model drift monitoring, we need to plan for bias detection on an ongoing (real-time) basis. With the arrival of new data (including feedback loops), a model that is not biased today may become biased tomorrow. This is, for example, what happened with Microsoft's "teen girl" AI chatbot, which, just hours after its deployment, learned to become a "Hitler-loving sex robot."

2.3 Responsibility

As with the debate surrounding self-driving cars, the question is "who is responsible" in the event of an accident. Is it the driver, the car manufacturer, the insurer, or even the municipality (in the case of a problem with the road or signage)? The same debate also applies to AI models – who is responsible if something goes wrong, for example, as explained above in the case of deploying a biased AI model?

Below we present some questions (in the form of a checklist) that you should review/clarify before signing the contract with the Gen AI vendor of your choice:

• Liability: Given that we are engaging with a third party, to what extent is the third party liable? This question is tricky to negotiate and depends on the extent to which the AI ​​system can operate independently. For example, in the case of a Chatbot, if the bot is only allowed to provide a limited output (e.g. respond to the user with a limited number of pre-approved responses), the risk will likely be much lower than in the case of an open bot like ChatGPT which can generate new responses.
• Data ownership: Data is essential for AI systems, and negotiating ownership issues regarding not only training data, but also input data, output data, and other generated data is therefore crucial. For example, knowledge of the chatbot's prompts (user queries) and responses is very important for improving the bot's performance over time.
• Confidentiality clauses: In addition to the confidentiality of the (training) data, do we want to prevent the provider from giving its competitors access to the trained/refined model, or at least to any improvements made to this model – especially if it confers a competitive advantage?

3. Solution design

So far, we have highlighted key aspects of AI/ML model development and deployment that we need to start addressing today – as part of a holistic Responsible AI framework for businesses.

As with everything in life, and especially in AI, there is no black and white and a general AI policy mandating the use of explainable AI/ML models is not not optimal – which implies missing out on what unexplainable algorithms can provide.

Depending on the use case and geographical regulations, there is always room for negotiation. Regulations relating to different use cases (e.g., profiling, automated decision-making) vary from country to country. Regarding bias and explainability, we have a full spectrum ranging from "fully explainable" to "partially explainable but verifiable," and even "completely opaque but with very high accuracy." Under these conditions, it is necessary to assemble a competent and interdisciplinary team (comprising at least representatives from IT, legal, procurement, and business departments), often referred to as an AI ethics committee, that can make such decisions consistently, in accordance with the company's values ​​and strategy.

3.1 Solution design principles

In this section, we examine the design principles that allow a "transparent" approach to specifying the capabilities (and limitations) of the generic AI application, by setting realistic expectations for the user.

Rather than trying to invent a new framework, we draw inspiration from Microsoft's "business-friendly" approach, Google's "developer-friendly" approach, and Apple's "user-friendly" approach to enabling this "transparent" approach to generic AI system design. Let's take a look at the AI ​​design frameworks recommended by these three leaders:

Guidelines for Human-AI Interaction from Microsoft

People + AI Guidebook from Google

Machine learning: Apple's Human Interface Guidelines

source: Responsible Generative AI Design Patterns (LinkedIn)

To go further: these principles of responsible design converge our article on algorithmic power and human responsibility et our analysis on a sober and responsible AI.