Telemetry.dev LLM observability AI agents answer a problem every team shipping agents hits sooner or later: your agent worked in testing, fails in production, and nobody can say which step broke, what it cost, or why. Telemetry.dev is an LLM observability platform built on OpenTelemetry that turns each agent run into a single trace, with every model call, tool step and retrieval shown as a span carrying duration, token usage, cost and error. Verdict up front: if your agent spend or debugging time already feels out of control, the free plan is a useful first instrument panel, and the paid decision should wait until your traces show exactly what you are paying for.
This review is for founders, product leads and developers putting AI agents into real products: what telemetry.dev does, what it costs, who it suits, its limits, and how it compares with established tools. Claims from the maker or a third-party directory, rather than our own testing, are labelled as such.
Telemetry.dev LLM Observability AI Agents: What It Actually Traces
An agent run is not one model call. It is a chain: a prompt goes in, the model calls a tool, the tool returns data, another tool runs, a retrieval step fetches documents, and finally an answer comes out. When the answer is wrong, slow or expensive, the failure hides somewhere in that chain. Traditional monitoring shows a request took five seconds; it does not show that one shipment-tracking tool call took 2.4 seconds, almost half the run.
Telemetry.dev groups the instrumented calls, tool steps and retrievals in one run into a trace, each step a span. Its own example makes the point: a support-agent trace of five spans totalling 4.9 seconds, where one tool call dominates. Search traces by name and filter by status, environment, framework or session, so a ticket saying “the agent gave a wrong refund answer at 3pm” becomes a trace you open, not a mystery you reproduce.
Four numbers sit on every span: duration, token usage, computed cost and error state. Cost is calculated at ingest from reported token usage against the pricing catalogue, with separate cache rates where available. Reasoning tokens are not charged separately, an explicit span cost takes precedence, and with no matching price or usage, cost is unavailable rather than guessed.
What It Does, Feature by Feature
The core is tracing, but the October 2026 Product Hunt launch added the pieces that turn tracing into a working loop. Prompt management lets you change a prompt without deploying, and evals test a new prompt version against a dataset before it goes live. That pairing matters: most agent regressions we see are prompt edits shipped without a comparison run, not model failures.
- Traces and spans: model calls, tool steps, retrievals, logs and metrics in one OpenTelemetry trace, not a proprietary format.
- Cost and latency views: spend, latency and errors by model, provider and environment, including p95 latency, so one slow model cannot hide behind a healthy average.
- MCP analytics: spans, failures and p95 duration by method and tool, from client and server, where agent-to-tool failures usually live.
- Alerts: email alerts for cost spikes, budget breaches, agent loops and rising errors, so you hear about a problem before users do.
- Prompt and eval loop: change prompts without a deploy, then run the new version on a dataset and compare it with production.
- Agent-readable traces: a command-line tool and read-only MCP server, so your coding agent inspects traces directly.
Instrumentation has two routes. First-party TypeScript and Python SDKs wrap a function in an observe call capturing inputs, outputs, tokens and errors, with integrations for OpenAI, Anthropic, the Vercel AI SDK and LangChain. Or any existing OpenTelemetry exporter can send standard OTLP to ingest, so custom setups work without a new SDK. That standards-based design is the platform’s strongest choice: moving tools later does not strand your instrumentation.
Pricing: What the Free Plan Gives You, and What to Check Before Paying
The free plan is confirmed on the official site: no cost, no credit card, 30,000 ingestion units per month, 14-day retention, one project and two seats. Each normalised span, log record or metric point uses one unit, so a nine-span agent run consumes nine units. Before choosing a plan, multiply expected runs per month by average spans per run: that tells you within minutes whether 30,000 units covers your workload.
Three plans are listed: Free, Pro and Team. Third-party directories list Pro at $25 per month and Team at $100 per month, but we could not confirm those paid figures on the official homepage, so treat them as reported prices and check the telemetry.dev pricing page before budgeting. The ladder’s shape is clear: free for a side project or single service, paid tiers for teams running AI across multiple services. It is cloud-hosted only in the sources reviewed; no self-hosted option is described, which matters if compliance requires data inside your own infrastructure.
A warning from agency experience: observability bills grow with success. The month your agent traffic doubles, your ingestion units double too. Set a budget alert on day one, review cost by model weekly in your first month, and treat one model eating most of your spend as a prompt or routing problem, not an immediate reason to buy a bigger plan.
Who It Is For, and Who Should Look Elsewhere
Telemetry.dev fits best in three situations: a team debugging a production agent blind, where failed runs and surprise bills are already happening; a small team building with OpenAI, Anthropic, the Vercel AI SDK or LangChain that wants OpenTelemetry tracing without running its own stack; and a team wanting the prompt-and-eval loop beside its traces.
It is weaker elsewhere. If you need self-hosting for compliance, the cloud-only shape in the sources is a blocker until the maker says otherwise. If you already live inside one vendor’s tracing, a dedicated tool may duplicate it. And a prototype with a handful of runs a day probably needs better logging first; observability pays when volume makes manual debugging impossible, not before.
Watch-Outs and Honest Limitations
The most important limitation is one the maker states openly: prompt capture is on by default, you can disable it per environment before storage, and redaction patterns apply, but pattern matching cannot catch all sensitive data. Redaction is a safety net, not a guarantee. If your agent handles personal, financial or health information, strip sensitive content in your own instrumentation before export rather than relying on server-side redaction.
- Short free retention: 14 days debugs this week’s incidents, not last quarter’s drift. Export anything you will need later.
- Cost needs pricing data: with no matching catalogue price or reported token usage, computed cost is unavailable, leaving gaps for exotic models.
- Traces depend on instrumentation: the platform shows only what your code emits. A poorly instrumented agent produces thin traces, and no dashboard fixes that.
- Young, focused tool: launched on Product Hunt in October 2026, with deliberately narrower scope than a general observability platform. Verify any feature you depend on against current documentation.
- Alerts need tuning: thresholds must match real traffic. An alert firing daily gets ignored; one never firing is decoration. Calibrate in week one.
How It Compares With the Alternatives
The established names are Langfuse, LangSmith and Arize Phoenix, and the comparison is about trade-offs. Langfuse is open source and self-hostable, with a larger published free allowance. LangSmith is strongest inside LangChain and LangGraph and charges per seat, which changes the maths as your team grows. Arize Phoenix is open source and OpenTelemetry-based, with a paid hosted sibling. Telemetry.dev’s differentiators are its narrow focus on agent traces with cost on every span, standards-based ingest, and the prompt-and-eval loop beside the traces. If you need self-hosting, Langfuse or Phoenix deserve the first look; if you want the fastest route to a first trace, telemetry.dev’s free plan makes it the easiest experiment.
From Our Experience: What This Means for Your Business
At MTD Technologies we have spent nine-plus years shipping web, mobile and AI systems across more than 300 projects, and the agent pattern is consistent: the demo impresses, the first production month humbles. The humbling takes one of three forms: a tool call that silently times out and poisons the answer, a model choice costing ten times the assumed amount, or a prompt edit quietly degrading quality for a week. All three are trace problems before they are model problems, and a trace showing the slow step, the expensive model and the changed prompt turns a week of arguing into an afternoon of fixing.
Our advice: instrument before you scale. Add tracing when your agent serves its first ten users, while traces are still easy to read. Agree on a p95 latency budget per run and a cost budget per thousand runs, and alert on both. Keep one rule: no prompt ships without an eval comparison. Teams following those habits rarely need heroic debugging.
Weighing an in-house build, be realistic: a trace viewer, a cost calculator across thousands of model prices, alerts and a prompt-eval loop is a product, not a weekend project. Buying a focused tool is usually the better trade, and the free plan proves it on your traffic first. Browse similar tools in our AI Tools Directory, or talk to our team about wiring observability into an agent you are shipping.
Getting Started in an Afternoon
- Create a project and a project API key on telemetry.dev; no credit card is required for the free plan.
- Pick your route: the TypeScript or Python SDK, or your existing OpenTelemetry exporter pointed at ingest.
- Wrap one agent flow, run one instrumented request, and confirm spans, tokens and cost appear in its trace.
- Turn off prompt capture for sensitive environments, and strip sensitive fields before export.
- Set cost and latency alerts from your first week’s real traffic, then review cost by model after seven days.
- Only then decide on a paid plan, using your own ingestion-unit maths.
Frequently asked questions
What is telemetry.dev?
Telemetry.dev is an LLM observability platform for AI applications and agents, built on OpenTelemetry. It groups the calls, tool steps and retrievals in an agent run into one trace, with duration, token usage, cost and errors per step, so teams can find the slow step, inspect the failed call and see where their budget goes.
Is telemetry.dev really free?
Yes. The free plan costs nothing, needs no credit card, and includes 30,000 ingestion units per month, 14-day retention, one project and two seats. Each span, log record or metric point uses one unit, so multiply runs per month by spans per run before assuming it covers production traffic.
How does telemetry.dev calculate cost?
Cost is calculated at ingest from reported token usage against a model-pricing catalogue, with separate cache rates where available. Reasoning tokens are not charged separately, an explicit span cost takes precedence, and without matching pricing or usage, cost is unavailable rather than estimated.
Can I keep prompts out of telemetry.dev storage?
Yes, disable prompt capture per environment before storage; redaction patterns apply to recognised fields. But redaction cannot catch everything, so strip sensitive content in your own instrumentation before export.
Should I choose telemetry.dev over Langfuse or LangSmith?
Choose telemetry.dev for the fastest route to traced runs with cost on every span and a prompt-and-eval loop in one tool. Choose Langfuse or Arize Phoenix for self-hosting, and LangSmith for a deep LangChain and LangGraph investment. For most small teams, the free plan on real traffic settles it faster than a feature-table debate.
Telemetry.dev LLM observability AI agents will not fix a bad agent, and no observability tool will. What it does is remove the blindness that makes bad agents expensive and slow to fix. Start free, instrument one real flow, read your first week’s traces, and let them decide whether the tool earns a place in your stack. If you would like help putting that visibility into a product you are shipping, contact MTD Technologies and tell us what your agent does today.