Your AI feature gives a perfect answer in testing. Then a user asks the same question twice and gets two different answers, or your support bot writes a poem when it should return a refund status. The setting behind that behavior is small and easy to overlook: temperature.
This guide explains what LLM temperature is, how top-p and max tokens work beside it, and how to choose values in your backend. It also covers a change that matters this month: one major provider has removed these settings from its API.
What Is LLM Temperature?
A language model writes one token at a time. Before each token, it scores every possible next token, and those scores become probabilities. If you are new to the unit, our guide to LLM tokens explains how text is split into tokens.
Temperature changes how the model chooses from those probabilities. A low temperature makes the most likely token win almost every time. A high temperature gives the less likely tokens a bigger share, so the output varies more from run to run. Ollama's documentation puts it plainly: increasing the temperature will make the model answer more creatively.
Temperature is a dial on randomness. It does not change what the model knows. It changes how boldly the model picks from what it already scores as possible.
See Temperature Work in Node.js
You do not need an API key to understand this. The function below turns made-up scores into probabilities and divides each score by the temperature first. Create the file sampling.js:
// sampling.js
const logits = { Abuja: 5.0, Lagos: 3.5, Kano: 1.5, Ibadan: 1.0 };
export function softmax(scores, temperature) {
const entries = Object.entries(scores).map(([token, score]) => [token, score / temperature]);
const max = Math.max(...entries.map(([, s]) => s));
const exps = entries.map(([token, s]) => [token, Math.exp(s - max)]);
const total = exps.reduce((sum, [, e]) => sum + e, 0);
return exps.map(([token, e]) => [token, e / total]);
}
for (const t of [0.2, 1.0, 2.0]) {
const line = softmax(logits, t).map(([k, v]) => `${k} ${(v * 100).toFixed(1)}%`).join(", ");
console.log(`temperature ${t}: ${line}`);
}
The scores stand for how strongly a model might favor each city after the words "The capital of Nigeria is". The numbers are invented for teaching, not taken from a real model. Run it with node sampling.js after adding "type": "module" to your package.json. On October 29, 2026, with Node.js 22, we got this output:
temperature 0.2: Abuja 99.9%, Lagos 0.1%, Kano 0.0%, Ibadan 0.0%
temperature 1: Abuja 78.6%, Lagos 17.5%, Kano 2.4%, Ibadan 1.4%
temperature 2: Abuja 56.1%, Lagos 26.5%, Kano 9.8%, Ibadan 7.6%
At 0.2 the top choice takes almost everything. At 1 the model still prefers Abuja but will sometimes say Lagos. At 2 the wrong answers get real weight. The scores never changed. Only the dial did.
What Top-P Does
Top-p, also called nucleus sampling, is a second way to limit randomness. Instead of reshaping the probabilities, it cuts the list. The model sorts tokens from most to least likely, keeps the smallest group whose probabilities add up to p, and discards the rest. A Microsoft answer on OpenAI models describes it this way: a top-p of 0.1 means only the tokens making up the top 10% of probability mass are considered.
Add this function to sampling.js to see the cut:
// sampling.js
export function topP(probs, p) {
const sorted = [...probs].sort((a, b) => b[1] - a[1]);
const kept = [];
let cumulative = 0;
for (const [token, prob] of sorted) {
kept.push([token, prob]);
cumulative += prob;
if (cumulative >= p) break;
}
const total = kept.reduce((sum, [, prob]) => sum + prob, 0);
return kept.map(([token, prob]) => [token, prob / total]);
}
With the temperature 1 probabilities above and p set to 0.9, the function keeps Abuja at 81.8% and Lagos at 18.2%. Kano and Ibadan are dropped because Abuja and Lagos already cover 96.1% of the probability. The long tail of unlikely tokens can no longer be picked.
Ollama documents a related setting, top-k, which keeps only a fixed number of the most likely tokens. Its docs note that a higher value gives more diverse answers and a lower value is more conservative.
Change One Dial, Not Both
Temperature and top-p both control randomness, so changing both at once makes behavior hard to predict and hard to debug. The Microsoft answer quotes OpenAI's guidance: we generally recommend altering one or the other but not both. Pick one, leave the other at its default, and change a single setting per experiment.
Max Tokens Is a Cost Cap, Not a Style Setting
Max tokens is the third dial, and it has a different job. It sets the most tokens the model may generate in one response, which is called max_tokens on the Claude API and num_predict in Ollama. When the model reaches the limit it stops, even in the middle of a sentence.
Set it on every call. Output tokens are billed, and a runaway response costs money and keeps your user waiting. Ollama's default of -1 means generation can continue without a limit, which is fine on your own machine and risky behind a public endpoint.
Do not use it to get shorter answers. A low cap cuts the answer off. If you want a short answer, ask for one in the prompt and keep the cap as a safety net. The output also has to fit in the model's working memory, which we cover in our guide to the context window.
What to Set for Common Backend Tasks
Start from the nature of the task, then test on your own prompts.
| Task | Starting direction | Why |
|---|---|---|
| Extracting fields, classifying tickets, returning JSON | Low | There is one correct answer and you want the same result every run. |
| Answering from your own documents | Low | You want consistent, grounded answers, not creative ones. |
| Drafting replies, product copy, or names | Higher | Variety is useful and you can review the output. |
| Generating test data or brainstorming | Higher | You want options, not one repeated answer. |
The Microsoft answer gives the same shape for OpenAI models: try 0.9 for more creative applications and 0 for ones with a well-defined answer. Treat those as starting points, then record which value you shipped and why.
Providers Handle These Settings Differently
The biggest mistake is to assume every provider accepts the same dials. Three examples from the vendors' own documentation, read on October 29, 2026:
- Anthropic's migration guide for API version 2026-10-21 says a request that sends
temperature,top_p, ortop_kreturns a 400 error, even at the default value. The versioning page says the parameters are removed and the error applies on every model. Requests on version 2023-06-01 keep working. Anthropic's guide says no other setting replaces them. - Microsoft's documentation for Azure OpenAI reasoning models lists
temperatureandtop_pamong the parameters that many reasoning models do not support. Check the table for the exact model you call. - Ollama, which runs open-weight models on your own hardware, documents defaults of 0.8 for temperature, 0.9 for top-p, and 40 for top-k, and all are adjustable.
Anthropic's versioning page says the client SDKs set the anthropic-version header for you. If you call the API directly, you set it yourself. In both cases, the lesson is the same: old code that always sends temperature: 0.2 can start failing after an upgrade.
Build a Provider-Safe Settings Helper
Keep sampling settings in one place and let the code decide whether to send them. Create the file sampling-config.js:
// sampling-config.js
const SUPPORTS_SAMPLING = {
claude: false,
ollama: true,
openai: true,
};
export function samplingFor(provider, wanted) {
if (!SUPPORTS_SAMPLING[provider]) {
return {};
}
const options = {};
if (wanted.temperature !== undefined) options.temperature = wanted.temperature;
if (wanted.topP !== undefined) options.top_p = wanted.topP;
return options;
}
The function returns an empty object for a provider that rejects the settings, and an options object for one that accepts them. Calling samplingFor("claude", { temperature: 0.2 }) returns {}, and samplingFor("ollama", { temperature: 0.2, topP: 0.9 }) returns { temperature: 0.2, top_p: 0.9 }. Fill the table from each provider's docs, and set the flag to false for any model that rejects the parameters, including reasoning models.
When a provider takes the dial away, your prompt becomes the control. Ask for exact formats, give examples, and validate the output in code, because you can no longer lower randomness at the API.
Temperature Does Not Fix Wrong Answers
A low temperature makes a model repeat its best guess, and the best guess can still be wrong. If your feature must be correct, ground it in your own data and check the result before you show it. Our guide to LLM hallucination covers the patterns that work.
For repeatable runs when you host a model yourself, Ollama also documents a seed setting that makes the model generate the same text for the same prompt. Whichever provider you use, log the model name, the settings, and the prompt with every result so you can reproduce a bad answer later.
Free Ways to Practice
Experimenting with a paid API in dollars adds up quickly, especially when you are testing 20 variations of one prompt. A local Ollama model costs nothing per request, so you can compare temperature 0.2 against 1.0 on the same prompt as many times as you like, as long as your machine has power. The behavior you learn carries over to hosted APIs that still expose the dials.
When you are ready to learn this in a structured order, the AI Engineering course moves from AI foundations and prompt engineering to RAG pipelines, agents, and production concerns such as cost.
Sources
- Migrating to API version 2026-10-21, Claude Platform documentation, read October 29, 2026. The page shows no publication date.
- Versions, Claude Platform documentation, read October 29, 2026.
- Recommended openai temperature and top_p, Microsoft Q&A, read October 29, 2026. The page shows no publication date.
- Azure OpenAI reasoning models, Microsoft Learn, read October 29, 2026.
- Modelfile reference, Ollama documentation, read October 29, 2026.
- Probabilities in the example: our own run of the code above, with invented scores, using Node.js 22 on October 29, 2026.
Summary
Temperature controls how random a model's next-token choice is, top-p cuts off the unlikely tokens, and max tokens caps how long the answer can run. Use low randomness for tasks with one right answer, more for drafting and brainstorming, change only one of temperature and top-p at a time, and set a max tokens value on every call.
Do not assume every provider accepts these settings. Anthropic's API version 2026-10-21 rejects them, many reasoning models do not support them, and local models through Ollama accept all three. Put your sampling settings behind one helper, rely on prompts and validation where the dials are gone, and log the settings with every result.
Frequently Asked Questions
What Does Temperature Do in an LLM?
Temperature controls how random the next-token choice is. A low temperature makes the model pick the most likely token almost every time, so answers are consistent. A higher temperature gives less likely tokens a bigger chance, so answers vary more. Ollama's documentation says increasing the temperature makes the model answer more creatively.
What Is a Good Temperature for an LLM?
It depends on the task. For work with one correct answer, such as extraction, classification, or structured output, start low. For drafting and brainstorming, start higher. A Microsoft answer on OpenAI models suggests 0.9 for creative uses and 0 for tasks with a well-defined answer. Treat any number as a starting point and test it on your own prompts.
What Is the Difference Between Temperature and Top-P?
Temperature reshapes the whole probability distribution before a token is picked. Top-P, also called nucleus sampling, keeps only the smallest group of tokens whose probabilities add up to P and drops the rest. Both control randomness. Microsoft's guidance for OpenAI models is to change one and leave the other alone.
Does Temperature 0 Make an LLM Deterministic?
It makes output much more consistent, but we do not treat it as a guarantee. Ollama documents a separate seed setting that makes the model generate the same text for the same prompt. Whatever the setting, store the model name, version, and prompt with each result so you can reproduce a problem later.
Can You Set Temperature on the Claude API?
Not on API version 2026-10-21. Anthropic's documentation says the temperature, top_p, and top_k parameters are removed in that version, and a request that sends any of them returns a 400 error on every model. Requests on the earlier version 2023-06-01 keep working, and Claude 4.7 and later models already accept only the defaults.
What Does Max Tokens Do?
Max tokens sets the largest number of tokens the model may generate in one response. When the model reaches it, generation stops, which can cut an answer off mid-sentence. It is a safety cap for your bill and your response time, and it is not a way to ask for shorter answers.
Does Temperature Reduce Hallucinations?
A lower temperature makes output more consistent, but it does not make wrong facts true. A model can repeat the same wrong answer every time at temperature 0. Grounding answers in your own data and verifying them in code is what contains hallucination.
