LLMs, RAG and orchestration

Agent orchestration

Agent orchestration is the layer that coordinates the parts of a voice agent: which model runs, which tools are called, in what order, and what happens when something fails.

Chunking

Chunking is splitting documents into smaller passages before they are indexed for retrieval, and where those splits fall largely decides whether the right answer can be found at all.

Context window

The context window is how much text a model can consider at once, counting both what it has been given and what it produces, and everything outside the window does not exist as far as the model is concerned.

Conversation memory

Session state is what a voice agent remembers within a single conversation and conversation memory is what it remembers across conversations, and the two are different problems carrying different risks.

Distillation

Distillation trains a smaller model to imitate a larger one, producing a compact model that performs close to the original on the tasks it was trained to imitate, at a fraction of the cost and latency.

Embeddings

An embedding is a list of numbers representing the meaning of a piece of text, arranged so that two passages about the same thing sit close together even when they share no words, which is what lets a system search by meaning instead of by keyword.

Fallback

A fallback is what happens when the primary path fails, covering everything from a voice agent not understanding a question to a model provider going offline, and its quality decides whether the system degrades or breaks.

Fine tuning

Fine tuning continues training a model on your own examples so it behaves the way you want, changing the behavior of the model rather than its knowledge of your current facts.

Function calling

Function calling is how a language model acts rather than only speaks: it emits a structured request for an action, and the application around it checks that request, runs it against a real system, and feeds the result back into the conversation.

Grounding

Grounding is the discipline of tying what an AI says to a verifiable source, so the answer can be traced to real material rather than to whatever the model absorbed during training.

Guardrails

Guardrails are the limits that keep a voice agent from saying or doing what it should not, whatever the visitor asked and whatever the system retrieved, and the thing that makes them guardrails is that they hold outside the model instead of being asked of it.

Hallucination

A hallucination is when an AI states something false with the same confidence it states something true, which is not a lie, because there is no intent, and not a bug, because the system is working as designed.

Human handoff

Human handoff is the act of moving a live conversation from an automated agent to a person, and escalation is the judgement that the move is needed, and together they are what a customer runs into whenever the automation could not finish the job.

Inference

Inference is running a trained model to produce an output, as distinct from training which builds the model, and in production it is where nearly all the cost and latency live.

Knowledge base

A knowledge base is the collection of material a voice agent answers from; in a RAG system it is the uploaded documents, and its quality sets the ceiling on answer quality.

Knowledge graph

A knowledge graph stores information as connected entities and relationships rather than as passages of text, so it can answer structured questions by traversing those connections rather than by retrieving a passage.

Large language model

A large language model is a neural network trained on vast amounts of text to predict the next token, and that single capability at scale produces systems that answer questions, follow instructions and hold conversations.

Prompt injection

Prompt injection is when text supplied to a model contains instructions that override or subvert its intended behavior, and it is the defining security problem of language model applications.

Reranking

Reranking is a second pass that reorders retrieved results by relevance using a slower and more accurate model than the initial search, and it is what fixes the near-misses fast retrieval gets wrong.

Retrieval-augmented generation

Retrieval-augmented generation, or RAG, is a technique that makes an AI look the answer up in specific documents you have given it before it replies, instead of answering from whatever it absorbed during training.

Semantic search

Semantic search finds results by meaning rather than by matching words, so a spoken question and the written answer can share no vocabulary at all and still be matched correctly.

Small language model

A small language model is a compact model with far fewer parameters than a frontier one, faster and cheaper and less capable, which for many narrow voice tasks is the right trade.

State machine against LLM-driven flows

A state machine follows an authored path with defined states and transitions, an LLM-driven flow lets the model decide what happens next, and the choice between them sets how predictable and how flexible a voice agent is.

System prompt

A system prompt is the standing instruction given to a model before any conversation begins, setting the role, the scope, the tone and the rules.

Temperature

Temperature is a sampling parameter that controls how varied a model output is, low producing consistent and predictable responses and high producing more varied ones, by reshaping the probability distribution over next tokens before selection.

Token

A token is the unit a language model reads and writes, usually a fragment of a word rather than a whole word, and cost, context window capacity and latency are all measured in it.

Vector database

A vector database holds embeddings and returns the nearest ones to a query fast enough that nobody waits, which makes it the search index that meaning-based retrieval runs on.