Tuning LLM outputs is essentially a decoding drawback: you form the mannequin’s next-token distribution with a handful of sampling controls—max tokens (caps response size beneath the mannequin’s context restrict), temperature (logit scaling for extra/much less randomness), top-p/nucleus and top-k (truncate the candidate set by chance mass or rank), frequency and presence penalties (discourage repetition or encourage novelty), and cease sequences (exhausting termination on delimiters). These seven parameters work together: temperature widens the tail that top-p/top-k then crop; penalties mitigate degeneration throughout lengthy generations; cease plus max tokens gives deterministic bounds. The sections under outline every parameter exactly and summarize vendor-documented ranges and behaviors grounded within the decoding literature.
1) Max tokens (a.okay.a. max_tokens, max_output_tokens, max_new_tokens)
What it’s: A tough higher sure on what number of tokens the mannequin could generate on this response. It doesn’t develop the context window; the sum of enter tokens and output tokens should nonetheless match inside the mannequin’s context size. If the restrict hits first, the API marks the response “incomplete/size.”
When to tune:
- Constrain latency and value (tokens ≈ time and $$).
- Stop overruns previous a delimiter whenever you can’t rely solely on
cease.
2) Temperature (temperature)
What it’s: A scalar utilized to logits earlier than softmax:
softmax(z/T)i=∑jezj/Tezi/T
Decrease T sharpens the distribution (extra deterministic); larger T flattens it (extra random). Typical public APIs expose a variety close to [0,2][0, 2][0,2]. Use low T for analytical duties and larger T for artistic enlargement.
3) Nucleus sampling (top_p)
What it’s: Pattern solely from the smallest set of tokens whose cumulative chance mass ≥ p. This truncates the lengthy low-probability tail that drives basic “degeneration” (rambling, repetition). Launched as nucleus sampling by Holtzman et al. (2019).
Sensible notes:
- Widespread operational band for open-ended textual content is
top_p ≈ 0.9–0.95(Hugging Face steering). - Anthropic advises tuning both
temperatureortop_p, not each, to keep away from coupled randomness.
4) Prime-k sampling (top_k)
What it’s: At every step, prohibit candidates to the okay highest-probability tokens, renormalize, then pattern. Earlier work (Fan, Lewis, Dauphin, 2018) used this to enhance novelty vs. beam search. In trendy toolchains it’s usually mixed with temperature or nucleus sampling.
Sensible notes:
- Typical
top_kranges are small (≈5–50) for balanced range; HF docs present this as “pro-tip” steering. - With each
top_kandtop_pset, many libraries apply k-filtering then p-filtering (implementation element, however helpful to know).
5) Frequency penalty (frequency_penalty)
What it’s: Decreases the chance of tokens proportionally to how usually they already appeared within the generated context, decreasing verbatim repetition. Azure/OpenAI reference specifies the vary −2.0 to +2.0 and defines the impact exactly. Constructive values cut back repetition; destructive values encourage it.
When to make use of: Lengthy generations the place the mannequin loops or echoes phrasing (e.g., bullet lists, poetry, code feedback).
6) Presence penalty (presence_penalty)
What it’s: Penalizes tokens which have appeared at the least as soon as thus far, encouraging the mannequin to introduce new tokens/matters. Identical documented vary −2.0 to +2.0 in Azure/OpenAI reference. Constructive values push towards novelty; destructive values condense round seen matters.
Tuning heuristic: Begin at 0; nudge presence_penalty upward if the mannequin stays too “on-rails” and gained’t discover alternate options.
7) Cease sequences (cease, stop_sequences)
What it’s: Strings that power the decoder to halt precisely after they seem, with out emitting the cease textual content. Helpful for bounding structured outputs (e.g., finish of JSON object or part). Many APIs enable a number of cease strings.
Design suggestions: Choose unambiguous delimiters unlikely to happen in regular textual content (e.g., "", "nn###"), and pair with max_tokens as a belt-and-suspenders management.
Interactions that matter
- Temperature vs. Nucleus/Prime-k: Elevating temperature expands chance mass into the tail;
top_p/top_kthen crop that tail. Many suppliers suggest adjusting one randomness management at a time to maintain the search area interpretable. - Degeneration management: Empirically, nucleus sampling alleviates repetition and blandness by truncating unreliable tails; mix with gentle frequency penalty for lengthy outputs.
- Latency/price:
max_tokensis essentially the most direct lever; streaming the response doesn’t change price however improves perceived latency. ( - Mannequin variations: Some “reasoning” endpoints prohibit or ignore these knobs (temperature, penalties, and so on.). Verify model-specific docs earlier than porting configs.
References:
- https://arxiv.org/abs/1904.09751
- https://openreview.web/discussion board?id=rygGQyrFvH
- https://huggingface.co/docs/transformers/en/generation_strategies
- https://huggingface.co/docs/transformers/en/main_classes/text_generation
- https://arxiv.org/abs/1805.04833
- https://aclanthology.org/P18-1082.pdf
- https://assist.openai.com/en/articles/5072263-how-do-i-use-stop-sequences
- https://platform.openai.com/docs/api-reference/introduction
- https://docs.aws.amazon.com/bedrock/newest/userguide/model-parameters-anthropic-claude-messages-request-response.html
- https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/content-generation-parameters
- https://cloud.google.com/vertex-ai/generative-ai/docs/be taught/prompts/adjust-parameter-values
- https://be taught.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning
Michal Sutter is an information science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and information engineering, Michal excels at remodeling complicated datasets into actionable insights.
