A large language model (LLM) is a neural network trained to predict the next piece of text (a token) from everything that came before it; question answering, translation, classification and code generation all grow out of that skill. The model only sees what fits in its context window, remembers nothing between requests and can be confidently wrong, so in a product it is backed by data, format validation and human review.
After this you will be able to
- explain how a model generates an answer one token at a time and why hallucinations follow from that
- estimate what fills a request's context window and why a long conversation costs more at every turn
- explain why
temperaturedoes not deliver consistency on new models and what replaces it - use the task → fit → backing table to decide whether to hand a task to a model
Know beforehand
Contents
What a language model is: predicting the next token
Every LLM (large language model) rests on a task that is simple to state: given the text so far, predict which piece should come next. The model does not output a single word but a probability distribution over all possible continuations; it then picks one, appends it to the text and repeats. The answer is assembled piece by piece, which is why a long answer takes longer than a short one.
Anthropic's glossary describes the path to a finished assistant in two stages. First comes pretraining: the model learns to predict the next word on a huge text corpus. Such a raw model is not good at answering questions or following instructions. Then fine-tuning and RLHF (reinforcement learning from human feedback) turn it into an assistant that understands requests and keeps to a format.
- 01Your text
- 02Tokens
- 03Model: continuation probabilities
- 04Chosen token
- 05Answer
Tokens and the context window
A model reads neither letters nor words but tokens — pieces of text: a whole word, part of one, a single character or a byte. How many characters make a token depends on the model and the language. Anthropic's glossary notes that the tokenizer used by Claude models from 4.7 onward produces roughly 30% more tokens for the same text than the previous one, where a token averaged about 3.5 English characters. Exact counts come from the token counting API, not from a formula.
The context window is the model's working memory: all the text it takes into account while generating one answer. According to Anthropic's documentation, everything in the request counts: the system prompt, every message in the conversation, tool results, images, tool definitions and the generated answer itself, including the model's reasoning. As of 2026-10-12, current Claude models have a 1M-token window, roughly 555k English words; OpenAI's current flagship models list 1,050,000 tokens, of which up to 922,000 can be input.
- The API keeps no conversation history: Anthropic's documentation calls the Messages API stateless, so every request carries the whole history it needs
- A long conversation gets more expensive at every turn: older messages are billed again as input tokens
- More context is not always better: as volume grows, accuracy and recall degrade (the documentation calls it context rot), so put curated material in the window, not everything you have
- A model's knowledge stops at its training data cutoff: about later events it knows only what you pass in the context
Training, fine-tuning and instructions — no myths
The most common misconception among clients and junior developers is that the model learns from API requests. It does not: model weights do not change while you send requests. Everything the model knows about your company is the text you put into the context of that particular request. Leave it out of the next request and the model has forgotten it.
Prompt and system instruction
- What changes
- Only the current request
- When to use it
- Almost always: role, rules, answer format
Context with data (RAG)
- What changes
- Only the current request
- When to use it
- You need fresh or internal data: a catalog, policies, a knowledge base
Fine-tuning
- What changes
- Model weights
- When to use it
- Rarely: a persistent style or a narrow task a prompt cannot solve; per Anthropic's glossary the Claude API does not currently offer it
Prompt and data first, everything else later is not an opinion but a practical consequence: a prompt changes in minutes and is covered by tests, while fine-tuning needs a dataset, separate evaluation and has to be redone whenever you switch models. In our experience, most business tasks are solved with a good instruction, examples and the right data placed in the context.
Generation parameters: temperature and limits
Temperature controls how random the choice of the next token is. A low temperature makes the model pick the most probable continuations, so answers are more conservative and uniform; a high one allows rare options, so text is more varied and creative. Even at temperature 0 the output is not fully deterministic: identical requests can produce different answers, as Anthropic's documentation explicitly notes.
Instead of temperature, the main levers on current models are reasoning effort and output limits. Effort (effort at Anthropic, reasoning.effort at OpenAI) sets how much the model thinks before answering: more effort usually means better quality on hard tasks, but slower and more expensive responses. The output length limit (max_tokens at Anthropic, max_output_tokens at OpenAI) cuts generation off; set it too low and the answer stops mid-sentence, and a JSON answer loses its structure.
Consistent results come not from sampling parameters but from the engineering around the model: a clear instruction, examples, schema-constrained output and validating the answer in code. That is the subject of the prompt engineering lesson.
Hallucinations and why context matters
A hallucination is a confident, fluent answer that does not match the facts or the supplied context: a law clause that does not exist, an invented library method, a wrong price. It follows directly from the mechanism: the model picks a plausible continuation and has no reason to say I don't know unless asked to and unless it has nothing to rely on.
Allow I don't know
Explicit permission to admit missing information, according to Anthropic, noticeably reduces fabricated answers.
Provide a source
Put the relevant documents in the context and ask the model to answer only from them, not from its general knowledge.
Require quotes
For long documents, ask the model to extract verbatim quotes first and only then draw conclusions from them; a claim without a quote gets removed.
Verify with code and people
Check numbers and identifiers against your database, run the same request several times and compare the answers, and leave costly decisions to a human.
What LLMs are good at and what they are not
A good task for an LLM needs language and common sense, and its errors are cheap to spot and fix. A bad one needs exact arithmetic or guaranteed truth, or produces answers nobody can check. Below is a working table for sorting product ideas.
Triage an incoming request: topic, urgency, contacts
- Is an LLM a fit
- Yes
- How to back it up
- Schema-constrained output, schema validation in code, manual review of borderline cases
Answer a customer from the company knowledge base
- Is an LLM a fit
- Yes, with data
- How to back it up
- Put the relevant articles in the context (RAG), forbid answers outside the documents, cite the source
Summarize a long document or thread
- Is an LLM a fit
- Yes
- How to back it up
- Require quotes, cap the length, spot-check by a person
Draft code or a test
- Is an LLM a fit
- Yes, as an assistant
- How to back it up
- Compile, test, code review — never accept it without these
Compute an invoice total or a tax
- Is an LLM a fit
- No
- How to back it up
- Compute in code; the model can only extract the numbers from text
State the current price, stock level or exchange rate
- Is an LLM a fit
- Not on its own
- How to back it up
- Fetch data from your system via an API or a tool call; the model only phrases the answer
Make a legally binding decision
- Is an LLM a fit
- No
- How to back it up
- The model drafts and flags risks; the decision and signature stay with a person
- Can the answer be checked automatically: by a schema, a test, a database lookup?
- What does a mistake cost and who will notice it before the customer does?
- What data does the model need, and are you allowed to send it to an external service?
- What happens if the API is down: is there a fallback path?
Cost and latency
You pay for tokens: separately for input (everything you send, including history and instructions) and for output (the generated answer). The model's reasoning is billed as output tokens too, at both Anthropic and OpenAI. Exact prices change, so take them only from the pricing pages. For orientation: on the Anthropic and OpenAI pricing pages as of 2026-10-12, an output token on current flagship models costs 5 times more than an input token, and the cheapest and most capable models of one provider differ in price by roughly a hundred times.
- Shorter context is cheaper and faster: do not send history and documents you do not need
- A repeated prefix (a long instruction, a reference document) can be cached: cache reads cost less than regular input at both providers
- Work that is not urgent goes through the Batch API — a 50% discount at both Anthropic and OpenAI
- For high-volume simple tasks (classification, field extraction), pick a smaller model and check quality on your own examples
Latency is the time to first token (TTFT) plus generation speed. It depends on model size, request and answer length, reasoning effort and the network. In user interfaces the answer is streamed so people see text immediately instead of waiting for the whole response. How to do that in code is covered in the lesson on your first LLM API request.