How generative AI actually works
OpenAI's launch of ChatGPT in late 2022 was phenomenal—even my grandmother talks about it. Its ability to generate human-like speech inspired people to experiment with its potential in various products. Its highly successful launch even put pressure on tech giants like Google, who quickly launched their own versions of ChatGPT.
But let's be honest, for non-technical product managers, designers, and entrepreneurs, the inner workings of ChatGPT can seem like a magical black box. Don't worry! In this blog, I'll try to explain the technology and model behind ChatGPT as simply as possible. By the end of this post, you'll have a good understanding of what ChatGPT can do and how it works its magic.
The GPT Transformer and Timeline
Before we dive into the mechanics of ChatGPT, let's take a quick look at the timeline of development of the language model transformer architecture and the different versions of GPT, so you can get a better idea of how things evolved into the ChatGPT we have today.
Timeline of transformers, GPT and ChatGPT. Image by the author.
- OpenAI was founded by Sam Altman, Elon Musk, Greg Brockman, Peter Thiel and others. OpenAI develops many different AI models other than GPT.
- Google publishes the article Attention is All You Need, which presents the Transformer architecture [2]. Transformer is a neural network architecture that lays the foundation for many state-of-the-art SOTA language models, such as GPT.
- GPT is presented in Improving Language Understanding by Generative Pre-training [3]. It is based on a modified transformer architecture and pre-trained on a large corpus.
- GPT-2 is introduced in Language Models are Unsupervised Multitask Learners [4], which can perform a series of tasks without explicit supervision during training.
- GPT-3 is introduced in Language Models are Few-Shot Learners [5], which can achieve good results with few examples in the prompt without fine-tuning.
- InstructGPT is introduced in Training language models to follow instructions with human feedback [6], which can follow user instructions better through fine-tuning with human feedback.
- ChatGPT, a sibling of InstructGPT, is featured in ChatGPT: Optimizing Language Models for Dialogue. It can interact with humans in conversations, thanks to fine-tuning with human examples and reinforcement learning from human feedback (RLHF).
The timeline shows that the GPT evolved from the original Transformer architecture and acquired its capabilities through many iterations. If you don't understand terms like transformers, pretraining, fine tuning, or reinforcement learning from human feedback, don't worry! I will explain them all in the following sections.
Dive into the depths of models
Now that you know ChatGPT is based on transformers and previous GPT models, let's take a closer look at the components of these models and how they work. It doesn't matter if you're not familiar with deep learning, neural networks, or AI—I'll leave out the equations and explain the concepts using analogies and examples.
In the following sections, I will begin with an overview of language models and NLP, before moving on to the original transformer architecture, then to how GPT adapted the transformer architecture, and finally to how ChatGPT is refined on the basis of GPT.
Language models and NLP
There are many types of AI or deep learning models. For natural language processing (NLP) tasks like conversations, speech recognition, translation, and summarization, we'll turn to language models to help us.
Language models can learn from a library of texts (called a corpus) and predict words or word sequences with probabilistic distributions—that is, the probability that a word or sequence will occur. For example, when you say "Tom likes to eat…", the probability that the next word will be "pizza" is higher than "table". If the goal is to predict the next word in the sequence, it's called next-word prediction; if it's to predict a missing word in the sequence, it's called masked language modeling.
As it is a probability distribution, there can be many probable words with different probabilities. Even though one may think that it is ideal to always choose the best candidate with the highest probability, this can lead to repetitive sequences. In practice, researchers therefore add a degree of randomness (temperature) when choosing the word from among the best candidates.
Typical NLP process with a language model. Author's picture.
In a typical NLP process, the input text goes through the following steps [8]:
Preprocessing: cleaning the text using techniques such as sentence segmentation, tokenization (breaking the text into small pieces called tokens), stemming (removing suffixes or prefixes), stop word removal, spell checking, etc. For example, "Tom likes to eat pizza" will be segmented into ["Tom", "likes", "eat", "pizza", "."] and transformed into ["Tom", "likes", "eat", "pizza", "."].
Encoding or embedding: transforming the cleaned text into a vector of numbers, so that the model can process it.
Model feed: pass the coded input to the model for processing.
Get Result: Get from the model the result of a probability distribution of potential words represented in vectors of numbers.
Decoding: transcribing the vector into human-readable words.
Post-processing: refine the result with spelling, grammar, punctuation, capitalization, etc.
AI researchers have come up with many different model architectures. Transformers, a type of neural network, have been at the forefront of technology for a few years and form the basis of GPT. In the next section, we will examine the components and mechanisms of transformers.
Transformer Architecture
The transformer architecture is the basis of GPTs. It is a type of neural network, similar to the neurons in our human brain. The transformer can better understand contexts in sequential data such as text, speech, or music through mechanisms called attention and self-attention.
Attention allows the model to focus on the most relevant parts of the input and output by learning relevance or similarity between items, which are usually represented by vectors. If it focuses on the same sequence, it is called self-attention [2][9].
The attention mechanism measures the relevance/similarity between each item. Author's picture.
Consider the following sentence: "Tom likes to eat apples. He eats them every day." In this sentence, "he" refers to "Tom" and "they" refers to "apples." The attention mechanism uses a mathematical algorithm to indicate to the model that these words are related by calculating a similarity score between word vectors. Thanks to this mechanism, transformers can better "make sense" of the meanings in text sequences in a more coherent way.
Transformers have the following components [2]:
Embedding & Positional Encoding: transforming words into vectors of numbers.
Encoder: It extracts features from the input sequence and analyzes its meaning and context. It produces a matrix of hidden states for each input token to pass to the decoder.
Decoder: Generates the output sequence based on the encoder output and previous output tokens.
Linear layer and Softmax layer: transforms the vector into a probability distribution of the output words.
The encoder and decoder are the main components of the transformer architecture. The encoder is responsible for analyzing and "understanding" the input text, and the decoder is responsible for generating the output.
Illustration by the author. Adapted from Attention is All You Need [2].
If this interests you, you can continue reading for details on encoders and decoders. Otherwise, feel free to skip to the next section where we will discuss GPTs, which are a type of transformer.
The encoder is a stack of multiple identical layers (6 in the original transformers article). Each layer has two sub-layers: a multi-head self-attention layer and a feed-forward layer, with some connections, called residual connection and layer normalization [2]. The multi-head self-attention sublayer applies the attention mechanism to find the connection/similarity between the input tokens in order to understand the input. The lookahead sub-layer does some processing before passing the result to the next layer to avoid over-adjustment. You can compare coders to reading a book: you pay attention to each new word you read and think about how it relates to previous words.
The decoder is similar to the encoder in that it is also a stack of identical layers. However, each layer of the decoder has an additional attention layer between the self-attention and feedback layers, allowing the decoder to focus on the input sequence. For example, if you translate "I love you" (input) to "Je t'aime" (output), you need to know that "Je" and "I" are aligned, and that "love" and "aime" are aligned.
The multi-headed attention layers of the decoder are also different. They are hidden to not care what is to the right of the current token, which has not yet been generated [2]. We can consider decoders as free writing: we write according to what we have written and what we have read, without worrying about what we are going to write.
From transformers to GPT, GPT2 and GPT3
The full name of GPT is Generative Pre-trained Transformer. From this name you can see that it is a generative model, good for generating results; it is pre-trained, which means it has learned from a large body of textual data; it is a type of transformer.
In fact, GPT only uses the decoder part of the transformer architecture [3]. In the previous section on transformers, we learned that decoders are responsible for predicting the next token in the sequence. GPT repeats this process over and over, using previously generated results as input to generate longer texts; this is called autoregression. For example, if it translates "I love you" into French, it first generates "Je," then uses the generated "Je" to obtain "Je t'aime." (See the dotted line in the previous illustration.)
For training the first version of GPT, the researchers used unsupervised pre-training with the BookCorpus database, consisting of more than 7000 unique unpublished books [3]. Unsupervised learning is like asking the AI to read these books itself and try to learn the general rules of language and words. In addition to pre-training, they also used supervised fine-tuning for specific tasks like summarizing or Q&A. Supervised means that they show the AI examples of correct requests and responses and ask it to learn from those examples.
In GPT-2, researchers increased the size of the model (1,5 billion parameters) and the corpus they feed with WebText, which is a collection of millions of web pages, during unsupervised pre-training [4 ]. With such a large corpus to learn from, the model has proven to perform very well on a wide range of language-related tasks, even without supervised fine-tuning.
In GPT-3, the researchers took it a step further by extending the model to 175 billion parameters and using a huge corpus comprising hundreds of billions of words from the web, books and Wikipedia. With such a model and such a corpus in pre-training, the researchers found that GPT-3 can learn to perform tasks better with one (one-shot) or a few examples (few-shot) in the prompt without supervised fine-tuning. explicit.
(If you'd like to learn more about how to incentivize models to produce better results, you can read my other article: How to Use ChatGPT in Product Management)
At this stage, the GPT-3 model is already impressive. But it's more of a general-purpose language model. Researchers wanted to explore how it could follow human instructions and have conversations with humans. So they created InstructGPT and ChatGPT based on the general-purpose GPT model. We'll see how they did this in the next section.
Teaching GPTs to interact with humans: InstructGPT and ChatGPT
After iterating from GPT to GPT-3 with increasing model and corpus size, researchers realized that larger models do not necessarily mean they can accurately track human intent and can produce harmful results. They therefore attempted to refine GPT-3 using supervised learning and reinforcement learning from human feedback (RLHF) [6][12]. These training steps resulted in two refined models: InstructGPT and ChatGPT.
Illustration taken from Training language models to follow instructions with human feedback [6].
The first step is supervised learning from human examples. The researchers first provided the pre-trained GPT with a labeled data set of pairs of prompts and responses written by human labelers. This dataset is used to allow the model to learn the desired behavior from these examples. This step provides a supervised fine-tuning (SFT) model.
The second step is to train a reward model (RM) to assess the responses of the generative model. The researchers used the SFT model to generate multiple responses from each prompt and had human labelers rank the responses from best to worst based on quality, engagement, informativeness, safety, consistency. and relevance. Prompts, responses and rankings are passed to a reward model to learn human response preferences through supervised learning. The reward model can predict a scalar reward value based on how closely the response matches human preferences.
In the third step, researchers use the reward model to optimize the policy of the SFT model through reinforcement learning. The SFT model generates a response from a new prompt; the reward model evaluates the response and assigns it a reward value that approximates human preferences; the reward is then used to optimize the generative model by updating its parameters. For example, if the generative model generates a response that the reward model thinks humans might like, it will receive a positive reward for continuing to generate similar responses in the future, and vice versa.
Through this process of supervised learning and reinforcement learning from human feedback, the InstructGPT model (with only 1,3 billion parameters) is able to perform better in tasks that follow human instructions. than the much larger GPT-3 model (with 175 billion parameters).
(Note: This does not mean that InstructGPT is better than GPT-3 in all aspects or areas. For example, GPT-3 may still have an advantage in generating longer or more creative texts, such as stories or articles).
ChatGPT is a sister model to InstructGPT. The training process is similar for both, including the same supervised learning and RLHF methods discussed earlier. The key difference is that ChatGPT is trained using examples of conversational tasks, such as answering questions, chatting, trivia, etc. [7] Through this training, ChatGPT can hold natural conversations with humans in dialogues. During these conversations, ChatGPT can answer follow-up questions and admit its mistakes, making interaction with it more engaging.
To remember
Thanks to the previous explanations, I hope you have a better understanding of how the ChatGPT model works and how it has evolved into what it is today.
As a quick recap, here are the most important points to remember and the limitations of GPT models:
ChatGPT is based on a decoder-only auto-regressive transformer model to take a text sequence and output a probability distribution of the tokens in the sequence, generating one token at a time iteratively.
Since it does not have the ability to look up references in real time, it makes probabilistic predictions in the generation process based on the corpus it was trained on, which can lead to false assertions about facts.
It is pre-trained on a huge body of web and book data and refined with examples of human conversation through supervised learning and reinforcement learning from human feedback (RLHF).
Its capabilities are primarily based on the size of its model and the quality and size of the corpus and examples from which it learned. With additional supervised learning or reinforcement learning from human feedback, it can perform better in specific contexts or tasks.
Since the corpus comes from web content and books, it may have biases that the model can take advantage of, including social, cultural, political, or gender biases, leading to biased responses to some queries.
Stay curious and open to new technologies like ChatGPT. I believe an open mindset and curiosity can help us—non-technical product managers, designers, and entrepreneurs—navigate this new wave of technological revolution.
To learn more: discover our Introduction to Generative AI and our article on The functioning of RAG in generative AI.