LangSmith CLI Skill

CLI: scripts/langsmith.py. Requires LANGSMITH_API_KEY in env (or ~/.zshrc).

No second API key needed — the ask command fetches and formats traces as structured context for your agent to analyze. No trace data is sent to any third-party LLM.

Commands

Tier 0 — Ask (agent Q&A over traces)

python3 scripts/langsmith.py ask "<question>" --project <name> [--since 24h] [--limit 50]

Fetches recent runs and prints them as structured JSON context. Your agent reads the output and answers the question — no external LLM calls, no data leaving your machine beyond the LangSmith API.

Examples:

• - INLINECODE4
• INLINECODE5
• INLINECODE6

Tier 1 — Situational Awareness

CODEBLOCK1

Tier 2 — Before/After Comparisons

python3 scripts/langsmith.py diff <project> --before <ISO_date> --after <ISO_date>
python3 scripts/langsmith.py prompt-diff <run_id_a> <run_id_b>

diff compares avg latency, error rate, cost, output length across two time windows. prompt-diff shows side-by-side system prompts + outputs for two specific runs.

Tier 3 — Deep Analysis (stubs, expand as needed)

CODEBLOCK3

Auth Setup

CODEBLOCK4

Test with: INLINECODE9

Security & Data Flow

This skill makes outbound network requests only to api.smith.langchain.com (the LangSmith API). That's it.

• - LANGSMITH_API_KEY — sent as an HTTP header to api.smith.langchain.com only. Never logged or stored.
• Trace data — fetched from LangSmith and printed to stdout for your agent to read. No trace data is sent to any third-party LLM or external service.
• No second API key required — the ask command outputs structured trace context for your existing agent to analyze, rather than making its own LLM calls.
• No telemetry — the script collects no usage data.

The script is ~300 lines of pure Python with no obfuscation. Audit it at scripts/langsmith.py.

API Reference

See references/langsmith-api.md for endpoint details and run object schema.