What is ChatGPT and how does it work?

ChatGPT has captured much media attention for its ability to produce text in response to various writing prompts. Its simple and friendly interface and speed in returning responses have led to concerns over the future of students’ writing skills and instruction with writing.

View of upstream river to a waterfall in the distance

ChatGPT is a form of artificial intelligence called a large language model (LLM). While ChatGPT-3 has garnered most media attention for its ability to generate text in response to a prompt rapidly, several large tech companies have their own LLMs (Google’s Pathways Language Model and Microsoft NVIDIA’s Megatron Turing LNG are reportedly the largest). Many companies are developing specialized models for customer service, marketing communication, and law and medicine.

Emily Bender and colleagues described LLMs as “Stochastic Parrots” in their 2021 article on the dangers of LLMs. Like a parrot who can mimic human speech according to cues, LLMs outputs are based on mathematical probabilities and limited and imperfect training sets. The models are stochastic, meaning they rely on statistical prediction and thus are characterized by probability and randomness. Output from models becomes less accurate and less plausible as tasks become more complex. While its responses appear meaningful and appropriate, the system doesn’t “understand” its output like a human.

How it works

The GPT in ChatGPT stands for Generative Pretrained Transformer. Transformers “learn" to create generalizations based on a training dataset by encoding and decoding samples. Some transformers are trained simply to recognize, and sort items into classes (so-called discriminative models), but generative transformers use the conventions learned from examples to generate probable examples. ChatGPT is pre-trained on massive statistical parameters (175 billion) to examine a gigantic quantity of text.  Based on its analysis of previously written and available discourse, it generates a text that mimics the plausible output of an actual writer.

Because the technology is familiar with the conventions of written language, its interface can use patterns of language and even uses first-person pronouns. Here’s how ChatGPT explains its use of writing conventions:

ChatGPT uses first-person pronouns as a linguistic convention to create a more conversational and user-friendly experience for users like yourself. It helps to establish a sense of interaction and connection between the AI and the user. However, it's important to note that as an AI language model, I don't possess personal experiences or consciousness. I am a machine learning model developed by OpenAI, and my responses are generated based on patterns and information from the training data. (ChatGPT query response, 2023)

One consequence of this design is that it is easy for users to mistake ChatGPT for artificial general intelligence. While sequences of prompts can appear to generate a conversation, the responses generated are predictions of appropriate responses, not a dialogue.

Limitations of Large Language Models

While large language models are helpful for producing models of writing that resemble human output, they face some challenging limitations.

  • It works according to predictions about the statistically likely next word, despite its apparent “choices” about sentences,  paragraphs, and overall response length.
  • It doesn’t have general intelligence features, meaning it doesn’t understand, read, choose, know, or quote.
  • ChatGPT 3 was trained on the publicly available internet of 2020
  • It has some trouble with mathematical and visual input (for now)
  • It produces counterfactual claims—reports false things as true (e.g., the Hillary Clinton Presidency)
  • It makes things up surprisingly well, including citations (what experts in AI research call “hallucinations”)

Because ChatGPT generates text differently from how humans compose written language, its model will include output with no relationship to reality. In a recent famous instance, an attorney asked ChatGPT to identify relevant legal precedents for an upcoming case. Because the technology could recognize the form of legal precedents, it generated text that appeared plausible but didn’t actually refer to real legal cases. Human content experts can detect these hallucinations, but the error-free, grammatical, and declarative output of Generative AI can fool casual readers.

What does this mean for writing instruction?

Despite initial concerns that ChatGPT could render some familiar genres and technologies obsolete (including the college essay and Google), ChatGPT and Generative AI are not yet able to replace human writers. However, the ability of Transformers to respond to exam questions accurately and concerns over academic misconduct implications require instructors to consider the implications of ChatGPT use in classrooms.

Currently, the University of Minnesota allows instructors to determine their course policies regarding ChatGPT in their courses, whether allowing unlimited use, use in limited circumstances, or outright restriction. The Office of the Provost offers examples of syllabus statements for each policy.

The Teaching with Writing program offers advice for responsible strategies for including ChatGPT in writing instruction or for restricting its use by students. Members of the WAC Team are also available for customized consultations.