S LEO Soul Kadropic Labs
Start free
ProductPricingDocsFAQsGlossaryEnterprise
Sign inStart free
Glossary

The vocabulary of trustworthy AI agents

Plain-language definitions of the concepts behind LEO Soul - from semantic entropy and calibration to conformal prediction, sycophancy, and the portable soul_state memory.

Core concept

Metacognition
Thinking about one's own thinking. In LEO Soul, metacognition is the agent measuring its own uncertainty, noticing knowledge gaps, and judging whether its answer is trustworthy - before it commits.
Metacognitive reliability layer
A layer that sits between your app and the model and gives an LLM agent calibrated self-awareness: it decides when to answer, ask, confirm, or refuse, rather than always answering. LEO Soul is such a layer.
LLM agent
A software agent whose reasoning is driven by a large language model, often calling tools and taking multi-step actions. Agents amplify both the usefulness and the risk of a model's mistakes, which is why a reliability layer matters.
AI guardrails
Controls that constrain what an AI system may output or do. Prompt-only guardrails are instructions ('be careful'); LEO Soul is a runtime layer that runs real algorithms and makes a decision, not just a filter.

Failure modes

Hallucination
When a language model states something fluent but false or unsupported. Detecting and abstaining on likely hallucinations is a core job of a reliability layer.
Confabulation
A specific kind of hallucination where the model invents details to fill a gap. Semantic-entropy methods are effective at flagging confabulation because invented answers vary in meaning across samples.
Sycophancy
The tendency of a model to change a correct answer because the user pushed back, flattered, or asserted otherwise - without new evidence. LEO Soul resists this with Bayesian updating.

Uncertainty & confidence

Uncertainty estimation
Measuring how unsure a model is about an answer. Good estimates let an agent abstain or ask exactly when it should, instead of guessing confidently.
Epistemic uncertainty
Uncertainty from lack of knowledge - reducible with more evidence or a clarifying question. Distinguished from aleatoric uncertainty, which is inherent randomness that more data cannot remove.
Semantic entropy
A measure of uncertainty computed over the *meanings* of a model's candidate answers rather than their exact wording. High semantic entropy - answers that disagree in meaning - is a strong hallucination signal.
Confidence calibration
Aligning a model's stated confidence with reality, so that things it says with 90% confidence are correct about 90% of the time. Uncalibrated confidence makes abstention decisions meaningless.
Brier score
A scoring rule that measures how well probabilistic predictions are calibrated; lower is better. LEO Soul tracks Brier score to keep its confidence honest over time.
Temperature scaling
A lightweight post-hoc calibration method that rescales a model's confidence with a single learned parameter, pulling over-confident scores back toward reality.
Expected Calibration Error (ECE)
A summary metric for miscalibration: the average gap between stated confidence and observed accuracy across confidence buckets. Lower ECE means more trustworthy confidence.

Deciding when to answer

Conformal prediction
A distribution-free statistical framework that turns raw model scores into predictions with a *certified* error guarantee, without assuming the data's distribution. LEO Soul uses it to bound the error rate on answered questions.
Conformal abstention
Using a conformal threshold to decide when to withhold an answer, so that the questions the agent *does* answer stay under a chosen error rate ε.
Answered-error bound
A guarantee of the form 'of the questions I choose to answer, at most ε will be wrong'. LEO Soul lets you set ε and holds the answered-error rate under it via conformal prediction.
Selective prediction
The general practice of letting a model answer only when it is confident enough and abstaining otherwise - trading coverage for reliability.
Abstention
Choosing not to answer. In LEO Soul, abstention is a first-class decision (ask / confirm / refuse / hold) rather than a failure, and it is what keeps the answered-error rate low.

Reasoning & learning

Bayesian updating
Holding beliefs as probabilities and revising them only in proportion to new evidence. It is why LEO Soul re-derives from facts under pressure instead of caving to a confident-sounding user.
Pressure-invariant reasoning
Reasoning whose conclusion depends on evidence, not on how forcefully the user pushes. The antidote to sycophancy.
Online learning
Learning continuously from each new example as it arrives, without a separate retraining job. LEO Soul's meta-controller learns your domain online, turn after turn.
Meta-learning
Learning how to learn - here, a compact online neural network that learns which situations tend to fail in your specific deployment and adjusts the agent's caution accordingly.
Expected information gain
How much a question is expected to reduce uncertainty. LEO Soul ranks its own clarifying questions by information gain and asks only the highest-value one.
Active learning
A strategy where the system chooses which questions to ask to learn the most from the fewest interactions - the principle behind LEO Soul's curiosity.
Grounding
Tying an answer to supporting evidence or context so claims can be checked. LEO Soul uses natural language inference to test whether an answer actually follows from the available facts.
Natural language inference (NLI)
Deciding whether one statement entails, contradicts, or is neutral toward another. Used to verify that an answer is entailed by its evidence rather than invented.

How LEO Soul works

soul_state
LEO Soul's portable memory: a compact JSON blob (under 48 KB) holding calibration histograms, Bayesian beliefs, the user model, the curiosity ledger, and neural weights. The customer owns it and passes it back each turn - that round-trip is the learning.
Stateless engine
An engine that stores nothing between calls: it reads a soul_state in and returns a fresh one out, writing nothing to disk. It is why there is no conversation content for LEO Soul to store or leak.
Decision trace
A readable record of how a turn was decided - the uncertainty measured, checks run, and why it chose to answer, ask, confirm, or refuse. It makes each decision inspectable and auditable.
Spectator Mode
A live, content-free window on the agent's metacognition: a real-time dashboard feed of typed event cards - decisions, warnings, anomalies, boundaries, recoveries, and info - streamed as turns run, without exposing one word of message content. Like a pilot's instrument panel: you see the warnings and recoveries, never the conversation. Available on every plan (Free included), with the full six-signal set and exportable reliability reports on paid tiers.
Risk-based triage
A near-zero-cost first pass that sizes up each turn's stakes so calm turns take a fast path near pass-through latency and deep scrutiny is spent only where it matters.
Fail-open guard
A safety wrapper around every module: on any internal error it degrades to a safe pass-through and marks itself degraded in the trace, so one failed check never costs the turn. It is the default and can be switched to fail-closed for regulated traffic. It governs a module failing, not the service being unreachable - that is the SDK's on_unavailable policy, which defaults to fail-closed.
Persona
A named configuration of LEO Soul's caution and behaviour for a given use case, so the same engine can be tuned per agent or product surface.

Search & discovery

Generative Engine Optimization (GEO)
Structuring content and data so AI-powered search - ChatGPT, Google AI Overviews, Perplexity, Claude, Copilot - can retrieve, cite, and recommend it. GEO complements classic SEO; AI citation and organic ranking are largely independent.
Query fan-out
When an AI assistant breaks a complex question into smaller sub-queries and answers each. Owning content for those sub-queries (FAQs, a glossary, clear headings) is how a brand gets cited.
Retrieval-augmented generation (RAG)
Augmenting a model with retrieved documents at answer time. LEO Soul is complementary: it judges whether the retrieved-and-generated answer is trustworthy enough to give.
Start free Read the docs