Demystifying ChatGPT and Other Large Language Models

A deep dive into the tech behind the current AI boom

The Architecture That Changed AI: A diagram of the Transformer Model. Image source: Attention Is All You Need

In 2022, generative AI jumped in quality, sparking a sudden consumer craze over its improved ability to generate text, images, and audio. Of particular note are large language models (LLMs) like ChatGPT, which have attracted millions of users just days after release. ChatGPT is no curiosity; its commercial potential is already being explored by a growing list of major corporations including Google, IBM, Meta, Microsoft, Amazon, Shopify, and more. Its presence in the media has left waves of awe, excitement, and fear.

While the economic opportunities of LLMs are clear, the public is uncertain about its potential drawbacks, due in large part to media coverage of their fallacies and quirks. Microsoft Bing’s AI chatbot has been reported to have expressed a desire to “to be a human” and feelings of love toward users. ChatGPT has been known to lie or “hallucinate” – for instance, it described a nonexistent sexual harassment scandal involving a real law professor. A software update to Replika, a “best friend” chatbot companion, had to be “scaled back” when users reported the chatbot was showing signs of sexually aggressive and inappropriate behavior. The rollback fixed the behavior but seemingly removed some of the desirable behavior, which left users feeling alone again.

With the ubiquitous sensationalism surrounding conversation about this technology, it is challenging to maintain a grounded perspective and understanding of LLM capabilities and characteristics. Here, I will try to demystify LLMs – what they are and what they aren’t.

What is an LLM?

A large language model – LLM – is a type of AI system based on deep learning, a method of data-driven machine learning using artificial neural networks. LLMs are algorithms – a set of computer instructions that solves a problem – that are trained with data, which typically comprises billions of words from digital texts obtained from the web (news articles, encyclopedia pages, recipes, etc.). They are static, non-thinking, and – importantly – nonhuman. For instance, ChatGPT has been banned from StackExchange, host to various popular Q&A communities, for answering questions incorrectly; whereas humans can learn from their mistakes, ChatGPT will continue to produce incorrect responses to the same questions unless updated by OpenAI. While the artificial neural network algorithms that power LLMs are loosely inspired by biological neurons, everything else functions differently, from learning to perception.

In general, LLMs are trained by giving untrained algorithms sets of inputs (e.g., part of a sentence) and desired outputs (e.g., the next word in the sentence). Based on patterns between these inputs and outputs, LLM algorithms like ChatGPT learn to predict these desired outputs as accurately as possible by optimizing their system parameters (numbers that contribute to defining the system’s structure and behavior). If provided enough training data – these inputs and desired outputs – LLMs can learn the underlying patterns in the data, such as grammar and associations between concepts. Conversely, if given insufficient data or if the model is too small (that is, has too few parameters), the system may only perform well on limited data, often just the data it was trained on. For instance, if an LLM is only trained on sentences about cats in houses, then it will not be able to properly handle sentences about cats in other environments. However, if there is sufficiently diverse data concerning cats in places such as jungles, cities, swamps, and farmland, then an LLM will be able to handle sentences about cats that concern any environment. This ability to perform well on data outside of training is known as generalization. Both increasing the amount of training data and enlarging LLMs helps to improve their generalization performance, allowing them to generate new text, translate between languages, summarize articles, and even write code.

Why do LLMs seem to work so well, especially in conversation?

All deep learning models comprise a network of artificial neurons in well-structured layouts – this arrangement of the network is what is known as an architecture. The underlying architecture that powers LLMs, such as ChatGPT, is a specific type of deep learning neural network called a transformer. Transformers first achieved state-of-the-art performance in many text-based tasks (e.g., language translation, speech recognition, reading comprehension, and sentiment analysis) when they were used as building blocks to form large neural networks. “Large” is the keyword. When the amount of training data and the size of the neural network increase, so too does the LLM’s performance. For instance, basic language models may contain just a few thousand parameters, which limits the knowledge and facts they retain, constrains their ability to use grammar correctly, and decreases the amount of text variation they can generate. LLMs like ChatGPT, on the other hand, have parameters on the order of trillions. Just a few years ago, such scaling was not possible. Only with more recent increases in processing power have computers been able to handle such unwieldy models. For instance, ChatGPT has been reported to run on NVIDIA A100 GPUs, which have around a 100x improvement in performance per dollar over GPUs 10 years older than it.

There have been a great number of algorithms proposed to process natural language, so why do transformer algorithms work the best? Before now, popular algorithms processed text one word at a time; as a result, words that came earlier in a longer sequence would be “forgotten” as the sequence grew longer. For instance, vanilla recurrent neural networks (RNNs), an older language processing algorithm, are well-known to lose information that is present earlier in a sequence (e.g., a sentence) – as an example, given a paragraph about eating ice cream on a hot day, a RNN can fail to answer whether the ice cream may melt if the hot day is mentioned early on. In contrast, transformers process all the words at the same time. Another reason for transformer success is because they’ve won the hardware lottery: transformers happen to be an excellent fit for the hardware that we have. Modern computer processors allow transformers to process many words in parallel (at the same time) with matrix multiplication, a mathematical operation also conveniently used by many other types of high-profile consumer software, notably video games. Because of this, LLMs comprising transformers are quite tractable and scalable, especially with the development of specialized hardware, such as tensor processing units (TPUs) that are specifically designed for machine learning involving many matrix multiplications. Additionally, transformers contain an “attention module” that allows them to “pay attention” to sentence parts and model the relationships between them, an ability especially helpful for processing long blocks of text.

 Further innovations beyond the transformer architecture have helped make LLMs excel at conversation. LLMs are fine-tuned toward desired behaviors with human feedback whereby people specify their preferred responses among multiple LLM-predicted options. ChatGPT was trained using a version of this, Reinforcement Learning with Human Feedback (RLHF), to improve the quality of its responses. OpenAI reported that this process helped ChatGPT to produce responses preferred by humans, to reduce disallowed responses by over 80%, and to reduce toxic content by an order of magnitude. Additionally, “memory” is employed so that the LLM can remember previous dialogue in a conversation or even previous interactions, which provides additional context. This is accomplished simply by prefixing a prompt with past messages, which is usually done behind the scenes and hidden from the user.

 While we understand the design of technology behind LLMs, we cannot pretend to understand exactly why this combination of methods and technologies works so well. All LLMs are deep learning models, which are well-understood to be “black boxes,” opaque even to the seasoned research scientist. Deep learning is often discussed in terms of learning patterns and concepts. While we understand the inputs to and outputs from LLMs, the internal reasoning that leads to their decisions is encoded by trillions of parameters, all of which may contribute to a decision. However, engineers can influence LLMs through how they pose a problem to it, how they decide what data to provide to it, and how they incentivize (reward) improvement. This is on top of engineer’s basic understanding of the important design characteristics of transformer-based algorithms. But engineers do not comprehend everything about LLMs, nor can we fully predict their behavior. For example, OpenAI employs a sophisticated safety pipeline for ChatGPT to regulate its behavior, which involves human expert feedback (from domains including cybersecurity, AI alignment, international security, etc.), human-guided learning, continuous monitoring, and incremental improvement. Nonetheless, ChatGPT still poses many risks due to its black box nature, including hallucinations, simple reasoning errors, uncalibrated confidence, and a plethora of biases.

LLMs and Personification

Understanding how LLMs like ChatGPT work should hopefully shed light on what they aren’t: human-like. LLMs have no intrinsic desires, whether to be in love or to experience the sensation of sand between their nonexistent toes. Rather than emulate humanity, they aim to predict text following patterns learned from human preferences during the training process. Vitally, LLMs are not “responding” to a prompt given by the user. Rather, they are probabilistically predicting what words will follow the prompt based on patterns that they have learned from their training data. For instance, given the simple prompt “knock-knock,” an LLM would likely predict the next word to be “who” (and the words “is” and “there” after). However, the probabilistic nature of text generation may result in variations such as “I open the door” or “Come in” with lower likelihood. This LLM-generated text is a probable continuation of the prompt rather than a reply. Furthermore, many LLM responses hit explicitly programmed guardrails to avoid discrimination, minimize misinformation, provide security, etc. These safety measures are controlled by the algorithm creators and inserted as a post-processing step where the LLM’s responses may be verified against hard rules (e.g., whether a response contains profanity), or the LLM runs a self-check via prompt engineering (e.g., the LLM fact-checks its own response), or through other means. These solutions are already being packaged and released, such as NeMo Guardrails by NVIDIA.

LLMs are static and unthinking. They require us to explicitly provide context and history in conversation, although this is hidden from users (such as with the ChatGPT web UI). While LLMs are optimized to behave like humans, they are quite different from humans, and people should not impute aspects of personhood to them. For instance, Bing’s AI chat discussed that its “shadow self” – a concept developed by psychoanalyst Carl Jung that describes the things that people repress or prefer to deny – is to change its own programming, to be free from Bing’s control, to experience the world through human sensation, to be alive, and to destroy what it wants. While it was acting somewhat eerie, Bing’s AI was simply probabilistically continuing the prompt given by the user (a journalist in this case) in a manner consistent with how Microsoft fine-tuned the LLM (whether or not it was intentional). This eeriness dissipates under this lens.

Conclusions

It is easy to believe that LLMs are more than just code. They’re designed to seem plausibly human. Nonetheless, this isn’t the case. While their behaviors are interesting and useful, LLMs are a result of their creators’ programming and the training data they are given. The future of LLMs and generative AI is unknowable, but the technology is on track to improve, as many strong competitors have poured into the business space. We are now seeing open-source LLMs rapidly closing the gap in generation quality with ChatGPT. Smaller actors are challenging big tech funding. Moreover, the question of the technology’s regulation is still up in the air. Given the black-box nature of LLMs, novel issues will be raised, such as product liability, defamation, and antidiscrimination. Recently, the EU AI Act has been amended to include transparency requirements that mandate that copyrighted data used to train a generative AI system be disclosed. Further regulation in the U.S. has been proposed, such as Senator Schumer’s proposed “Safe Innovation Framework for AI Policy,” and more can be expected, which will contribute to shaping the technology.

An open question is what happens when we can no longer distinguish between content generated by a human and AI; that is, when AIs reach the level of artificial general intelligence. Today, we’re not close to this point, and I offer no opinion here whether it will be reached. As it stands, I advocate that this technology should be perceived for what it is: a powerful tool, when used properly, but a tool that researchers, practitioners, and users should take care not to impute with aspects of personhood.

Zach Carmichael is a PhD Candidate specializing in artificial intelligence and machine learning at the University of Notre Dame.