Tokens, Weights, and Temperature

Introduction

Since I was 9, I wanted to be the person who developed the first artificial interlligence and name it CorneliusThe best name I could come up with for an artificial intelligence at the time. I took down a sheet of printer paper and started to do what I understood at the time. A bunch of if else statements. Looking at what I know now, I was way off.

Since November of last year, everyone has been talking about ChatGPT, I was hesitant to use it because I thought we were getting closer to developing Roko's BasiliskA future artificial superintelligence incentivized to retroactively punish anyone who knew of its potential existence but didn't actively work to bring it into being. But this didn't stop everyone I know from using it. Coworkers, friends, lovers.

Eventually I caved; however, API docs are full of words that get thrown around like everyone already knows them: tokens, weights, temperature, parameters, context window. I didn't, so I went and figured out what they actually mean. This is the explanation I wish I'd found.

The one-sentence version of a large language model: it predicts the next token, over and over. Everything below is about what a token is, what does the predicting, and how the prediction turns into a word on your screen.

Tokens

A model never sees letters or words. It sees tokens: chunks of text from a fixed vocabulary, usually a few tens of thousands of them. Common words are a single token. Rare or long words get broken into pieces. Punctuation and spaces get their own tokens or attach to the next word.

The algorithm most models use is BPEByte Pair Encoding: start with single characters and repeatedly merge the most frequent adjacent pair into a new token until the vocabulary is the target size. It's trained on a big pile of text, so the vocabulary ends up reflecting what's common. "the" is one token. "Tokenization" might be three.

Type anything into the box. The toy tokenizer below is hand-written, not trained, but it splits things roughly the way the real ones do.

Why does this matter to you?

• You pay per token, not per word. English runs about 1.3 tokens per word. Code and other languages run higher.
• The context window is in tokens. A 4,000 token limit is roughly 3,000 English words, including the model's own reply.
• Models are bad at letters because they don't see them. Ask one to count the r's in a word or reverse a string and it has to reason about pieces it never directly observes.
• Numbers are weird. "2023" might be one token and "20231" might be two, which is part of why arithmetic is shaky.
• Funny Tangent - I prompted the AI to reply like Kevin in that episode of the office. "Why use big word, when small word do trick", and I noticed less token usage.

Weights

A model is a giant function that takes the tokens so far and outputs a score for every token in the vocabulary. The weights (also called parameters) are the numbers inside that function. When you read "7 billion parameters" or "175B", that's how many of these numbers there are.

Training is the process of adjusting the weights. The model is shown text, asked to predict the next token, and every weight is nudged a tiny bit in the direction that would have made the right answer more likely. Repeat a few trillion times.

The key thing about weights:

• They're fixed after training. When you chat with a model, nothing is being learned. The weights are the same for every user and every conversation. The "memory" within a chat is just the previous tokens being fed back in as input.
• They're not a database. There's no row for "capital of France." The fact is smeared across millions of weights as patterns. That's why models can be confidently wrong: the pattern is close but not exact.
• Bigger is usually better, but costs more. More weights means more patterns can be stored, and more computation per token.

Fine-tuning is training again, on a smaller dataset, to nudge an existing model's weights toward a specific style or task. A system prompt doesn't touch the weights at all; it's just more input tokens.

Temperature

Here's the part most people have a knob for and don't understand. The model's output for the next token is a list of raw scores called logitsthe unnormalized scores a model assigns to each possible next token before they are turned into probabilities, one per vocabulary word. To pick a word, those scores get turned into probabilities with a function called softmax, and then one token is sampled from that distribution.

Temperature is a number you divide the logits by before softmax.

probability(token) = softmax(logit / temperature)

• Low temperature (0.1 to 0.3) exaggerates the gaps. The top token gets almost all the probability, so the output is predictable and repetitive. Good for code and extraction.
• Temperature 1 uses the model's scores as-is.
• High temperature (1.5+) flattens the gaps. Unlikely tokens get a real chance, so the output gets creative, then weird, then incoherent.

Drag the slider and watch the bars. The logits on the left never change; only the probabilities do. Then sample a few times at 0.2 and a few at 1.8.

Two related knobs you'll see next to temperature:

• Top-k only considers the k most likely tokens and ignores the rest.
• Top-p (nucleus sampling) only considers the smallest set of tokens whose probabilities add up to p, like 0.9. It adapts: when the model is confident, few tokens are in play; when it's unsure, more are.

Both are ways of cutting off the long tail that temperature would otherwise let through.

Putting it together

One response to one prompt, step by step:

1. Your prompt is tokenized into a list of integers.
2. The tokens go through the network, where the weights do a lot of matrix math and produce a logit for every vocabulary token.
3. Logits are divided by temperature, run through softmax, trimmed by top-k or top-p, and one token is sampled.
4. That token is appended to the input and we go back to step 2.
5. Stop when the model emits an end-of-sequence token or hits the token limit.

That loop is the entire thing. Every chatbot, every autocomplete, every "write me a poem" is step 2 through 4 on repeat.

Conclusion

Tokens are what the model reads and writes, and what you pay for. Weights are the frozen result of training, and the reason it knows anything. Temperature is how much randomness you let into the sampling. Now I have no experience training AI models yet but do weights always need to be fixed?