LangSmith

What is LangSmith?

LangSmith is an observability and evaluation platform for teams building LLM applications and AI agents. It records the steps inside an agent run so developers can inspect prompts, model calls, tool use, retrieval, latency, cost, outputs, and conversation threads instead of debugging from the final response alone.

The platform is framework agnostic. Teams can instrument applications built with LangChain, LangGraph, OpenAI or Anthropic SDKs, Vercel AI SDK, LlamaIndex, custom stacks, or OpenTelemetry. Production monitoring combines traces with online evaluators, dashboards, alerts, topic clustering, error analysis, and a trace database designed for structured agent queries.

How to Use LangSmith

1. Create a project and choose the cloud, bring-your-own-cloud, or self-hosted deployment that matches the data policy.
2. Add the relevant SDK or OpenTelemetry instrumentation to the application.
3. Group related runs into threads and attach useful metadata such as environment, release, user segment, or experiment.
4. Inspect traces for failed tool calls, poor retrieval, excessive latency, token cost, and unexpected trajectories.
5. Add code-based or LLM-as-judge evaluators, then monitor results on production traffic.
6. Configure dashboards and alerts for regressions, and turn representative failures into datasets and repeatable tests.
7. Redact or avoid sensitive inputs and confirm retention, access, and regional hosting controls before sending production traces.

Core Features

• End-to-end tracing: Shows nested model, retrieval, tool, and agent steps with inputs, outputs, latency, and cost.
• Framework-agnostic instrumentation: Supports popular agent frameworks, model SDKs, custom applications, and OpenTelemetry.
• Production monitoring: Tracks agent quality and operational metrics in real time.
• Online evaluation: Runs code rules or LLM judges against live traffic and conversation trajectories.
• Trace search: Filters full text, JSON paths, metadata, and agent paths through SmithDB.
• Automated insights: Clusters topics and surfaces recurring errors for investigation.
• Alerts and integrations: Can notify engineering workflows through webhooks and incident-management tools.
• Deployment control: Offers managed cloud, bring-your-own-cloud, and self-hosted options.

Use Cases

• Debug a multi-step agent that calls the wrong tool or follows an inefficient path.
• Compare prompt, model, retrieval, or agent versions before a release.
• Monitor response quality, latency, token consumption, and cost in production.
• Find recurring failure themes across large volumes of user conversations.
• Build evaluation datasets from real traces and prevent regressions in CI.
• Give engineering and product teams a shared record of agent behavior.

Pricing

LangSmith offers a free tier intended for development and smaller production workloads. Paid usage scales with trace volume and team requirements, while enterprise deployment and support options are quoted separately. Check the current plan limits before committing a production workload.

Frequently Asked Questions

Does LangSmith require LangChain or LangGraph?

No. It can trace other frameworks, model-provider SDKs, OpenTelemetry instrumentation, and custom applications.

Does observability prove that an agent is correct?

No. Traces expose behavior, but teams must define useful evaluators, review representative cases, and test real business outcomes.

What data should not be traced?

Avoid unnecessary personal data, credentials, secrets, regulated records, and proprietary content. Apply redaction, least-privilege access, suitable retention, and an approved deployment model.