LangSmith

Status

📋 Todo — send sources when ready.

Learning roadmap

• Setup — API key, env vars, automatic tracing for LangChain
• Tracing — runs, projects, tags, metadata
• Tracing non-LangChain code with @traceable
• Datasets — creating from production traces or manually
• Evaluators — built-in (LLM-as-judge, exact match), custom
• Running evaluations — evaluate() API
• Playground — testing prompts against datasets
• Production monitoring & alerting
• Annotation queues for human review
• Cost & latency analysis

What I want to be able to do after this topic

• Get full visibility into any LLM pipeline I build.
• Build a regression eval suite that runs on every PR.
• Debug a misbehaving agent by tracing through its runs.
• Track cost and latency in production.