Observability

AI Observability Platforms Compared: Langfuse, LangSmith and Built-In Tracing

LangSmith, Langfuse and built-in agent tracing compared: what each records, pricing, self-hosting, evals, and how to trace AI agent runs in production.

DNotifier Team••14 min read
Back to blog
AI Observability Platforms Compared: Langfuse, LangSmith and Built-In Tracing

Short answer: LangSmith is the most complete LLM observability platform if you build with LangChain: tracing, evaluations, prompt management and monitoring in one product. Langfuse is the leading open-source option: MIT-licensed, free to self-host, and framework-agnostic. If your agents run on a platform that already traces every run (DNotifier does), you may only need that built-in tracing at first, and add an evals tool later.

An AI agent that answers wrong doesn't throw an exception. It returns a confident, well-formatted, wrong answer, and the user screenshots it. The only way to fix it is to replay what happened: what the model was asked, which tools it called with which arguments, what came back, and how long each step took.

That's what AI observability platforms are for. (Where tracing fits in the overall design of an agent app is covered in AI Agent Architecture.) This guide compares the main options, including where our own built-in tracing stops.

What an AI agent trace shows

What an AI agent trace showsA trace of one agent run with its execution ID, user, model, status and total time of 3.9 seconds. It contains three model calls and two tool calls, orders.lookup and policy.search, each with its own duration and token count or result summary, laid out on a timeline. ONE AGENT RUN, AS A TRACEexecutionId: run_8f2c… user: cus_ayesha model: gpt-4o status: ok 3.9 sRunRun3.9 s totalModel call: pick a toolModel call: pick a tool0.8 s · 612 tokensTool: orders.lookupTool: orders.lookup0.3 s · { status, eta }Model call: next stepModel call: next step0.7 s · 840 tokensTool: policy.searchTool: policy.search0.4 s · 3 passagesModel call: final answerModel call: final answer1.6 s · 1,120 tokens0s1s2s3s4sInputs and outputs are stored per step. Click a row for what to look at when an answer is wrong.
Click a row to see what to check there when an answer is wrong.
What an AI agent trace shows

What an AI observability platform records

Traditional monitoring tells you a request took 4 seconds and returned 200. LLM observability needs to tell you why the answer was what it was. The useful records are:

  • Traces and spans. One trace per request, with a span for every model call, tool call, retrieval and sub-agent.
  • Inputs and outputs per step. The actual prompt, the tool arguments, the tool result, the completion.
  • Tokens, latency and cost. Per step and per trace, so you can find the expensive part.
  • Metadata. Which user, which session, which model and prompt version.
  • Evaluations. Scores on outputs, from automated checks, an LLM judge or human review, run on datasets or on live traffic.
  • Monitoring. Dashboards and alerts on error rates, latency, cost and evaluation scores over time.

The first four are tracing. The last two are what separate a full platform from a log viewer.

LangSmith

LangSmith is LangChain's commercial platform, and it's the market leader for teams on LangChain and LangGraph. It covers the whole loop: tracing, datasets and evaluations, prompt management, monitoring and alerts. It works with any stack through its SDK and OpenTelemetry, but the deepest integration is with LangChain, where tracing is a couple of environment variables.

Pricing (checked November 2026): a free Developer plan (1 seat, 5,000 base traces a month), Plus at $39 per seat per month with 10,000 base traces included, and custom Enterprise pricing. Usage beyond the included traces is pay-as-you-go. Base traces are kept for 14 days; extended traces, at extra cost, for 180 days. Self-hosting is available on Enterprise.

Pick it if you build on LangChain or LangGraph, you want evaluations and prompt management in the same tool as traces, and a hosted SaaS is fine for your data.

Langfuse

Langfuse is one of the most widely used open-source LLM observability platforms. The core is MIT-licensed and free to self-host with all core features, on Docker Compose, Kubernetes or the major clouds. It does tracing, evaluations (including LLM-as-a-judge and human annotation), prompt management, datasets and dashboards, and it's built on OpenTelemetry, so it isn't tied to one framework. Langfuse is now part of ClickHouse.

Pricing (checked November 2026): Langfuse Cloud has a free Hobby plan (50,000 units a month, 30 days of data access), Core at $29 a month and Pro at $199 a month (both with 100,000 units, then $8 per 100,000 units; 90 days and 3 years of data access respectively), and Enterprise from $2,499 a month. Self-hosting the open-source version costs only your infrastructure.

Pick it if you want to own your observability data, need to self-host for compliance, or use several frameworks and don't want to be tied to one vendor's tracing.

Langfuse vs LangChain (and LangSmith)

This comparison comes up because the names get mixed together. LangChain is a framework for building agents. LangSmith is LangChain's observability and evaluation platform. Langfuse is an independent observability platform that also works with LangChain, through a callback handler. So the real question is usually Langfuse vs LangSmith for a LangChain app:

LangSmithLangfuse
Made byLangChainLangfuse (part of ClickHouse)
LicenseCommercialMIT open source core
Self-hostingEnterprise planFree, all core features
LangChain integrationNative, env varsCallback handler
Other frameworksSDK + OpenTelemetryOpenTelemetry-based, many integrations
EvaluationsYes, deepYes
Prompt managementYesYes
Free tier5k base traces/month, 1 seat50k units/month (Cloud), or self-host

Other names you'll see on "Langfuse alternatives" lists include Arize Phoenix, Helicone and Datadog's LLM Observability. They're worth a look if you already use the parent company's tools.

Built-in tracing in your agent platform

The third option is the one most comparisons skip: tracing that comes with the platform your agents already run on. On DNotifier, turning it on is one flag:

const agent = DNotifier.defineAgent({
  name: "order-help",
  instructions: "Help customers with their orders. Use the tools for facts.",
  provider: "open_ai",
  model: "gpt-4o",
  observability: true, // every run is recorded
  tools: [lookupOrder, searchPolicy],
  permissions: { "orders.lookup": ["execute"], "policy.search": ["execute"] },
});

const result = await notifier.runAgent(agent, { input, userId, maxToolRounds: 5 });
console.log(result.executionId); // open this run in the dashboard

For multi-agent workflows it's the same flag on the workflow, and you can turn on session logs for every model call on the client. The multi-agent workflow tutorial walks through a traced router example end to end:

const notifier = new DNotifier({
  appId: process.env.DNOTIFIER_APP_ID,
  secret: process.env.DNOTIFIER_SECRET,
  transport: "http",
  userId: "support-api",
  logs: true, // session logs for each AI call, under Logs in the dashboard
});

const workflow = new DNotifier.Workflow({
  name: "support-router",
  observability: true, // each execution under Workflows in the dashboard
  async entry(ctx) {
    const { intent } = await ctx.runAgent("intent-agent", { input: ctx.input });
    ctx.recordStep({ label: "Route", output: { intent } }); // your own labelled step
    return ctx.runAgent(intent === "order" ? "order-help" : "policy-agent", { input: ctx.input });
  },
});

What gets recorded: the run's status and duration, each agent as a node, each AI step with its latency, the provider and model, and tokens when the provider returns them, plus knowledge-base searches and any custom steps you add with ctx.recordStep().

Where it stops, to be straight about it: DNotifier doesn't have an evaluations product, dataset management or alerting on quality scores today. It's tracing included with your agent platform, not a LangSmith replacement. If you need evals, run them in a dedicated tool alongside it.

Pick it if your agents run on DNotifier and you want every run traced from day one, with no second vendor to set up. The full list of what's included is on the features page.

How to trace AI agent runs in production

Include authorization decisions in the trace as well as successful actions. See logging calls and denials for the permissions side of the audit trail.

Whichever tool you pick, the same practices make traces useful.

Log the run ID next to the request

This is also on the checklist in How to Deploy AI Agents to Production.

Put the trace or execution ID in your own application logs, next to the HTTP request and the user ID. When support forwards a complaint, you search your logs for the user and land on the exact trace in one click.

logger.info({ userId: req.user.id, executionId: result.executionId, route: "/api/agent" }, "agent run");

Decide what never goes into a trace

Traces hold prompts and tool results, and those can contain personal data. Before you turn tracing on everywhere:

  • Have tools return only the fields the model needs (an order's status, not the customer's full address).
  • Don't put secrets, tokens or payment data in prompts or tool results at all.
  • Check each platform's retention and deletion options against your privacy policy.

Label steps for the person debugging later

"AI step 3" helps nobody. "Classify intent" and "Search refund policy" do. In DNotifier that's the label on ctx.sendAI and ctx.search, and ctx.recordStep() for code decisions.

Review a sample every week

The most useful observability habit costs nothing: read twenty real traces a week. You'll find the wrong tool choices, the slow steps and the prompts that need work long before a dashboard does.

AI agent monitoring: the signals that matter

Tracing tells you about one run. Monitoring tells you about all of them. The handful of signals worth watching from the start:

SignalWhy it matters
Runs that hit the step limitThe agent was confused or a tool kept failing
Tool error rate per toolA broken integration shows up here first
p95 latency per agentUsers notice the slow tail, not the average
Tokens per runCost creep from longer prompts or extra tool rounds
Hand-offs to a humanQuestions the agent can't answer yet

Which AI observability platform should you choose?

  • Building on LangChain or LangGraph, want everything in one place: LangSmith.
  • Want open source and to own the data, or need self-hosting: Langfuse.
  • Agents run on DNotifier and you need tracing now: start with the built-in tracing. Add Langfuse or LangSmith when you need evaluations and datasets.
  • Already deep in a monitoring vendor: check its LLM observability add-on before adding another tool.

Every agent run traced from the first one: steps, tools, latency and the model used. Turn on observability: true. Tracing on the features page · Create your app

FAQ

What is the best AI observability platform?

For LangChain and LangGraph users, LangSmith is the most complete: tracing, evaluations, prompt management and monitoring in one product. Langfuse is the leading open-source choice and is free to self-host. If your agents run on a platform with built-in tracing, start there and add an evals tool when you need it.

What are the best Langfuse alternatives for LangChain apps?

LangSmith is the most direct alternative, with native LangChain integration. Arize Phoenix and Helicone are other common options, and teams already using Datadog can look at its LLM Observability product.

What is the difference between Langfuse and LangSmith?

LangSmith is LangChain's commercial platform with native LangChain tracing and deep evaluation features; self-hosting is an Enterprise option. Langfuse is an independent, MIT-licensed open-source platform that is free to self-host and works with LangChain through a callback handler.

Is LangChain the same as LangSmith?

No. LangChain is an open-source framework for building LLM applications and agents. LangSmith is LangChain's separate commercial platform for tracing, evaluating and monitoring them.

How do I trace AI agent runs in production?

Turn on tracing in your agent framework or platform, record the trace or execution ID next to each request in your own logs, label each step clearly, keep personal data and secrets out of prompts and tool results, and review a sample of real traces every week.

Does DNotifier include AI observability?

Yes, for tracing. With observability: true on an agent or workflow, each run is recorded with its agents, AI steps, latency, model and searches, and session logs record each AI call. DNotifier doesn't currently offer evaluations or datasets, so pair it with a dedicated tool for those.

Tags

#observability#langfuse#langsmith#langchain#tracing#monitoring#ai-agents

Keep reading

Related articles

View all →