Integration · Developer tools

Connect Claude & ChatGPT to LangSmith

Ask your AI how a tracing project is doing, and it pulls run counts, latency, tokens, cost and errors from LangSmith, LangChain's platform for tracing and evaluation. It can add examples to datasets, record feedback on runs, read and save prompt versions, and find the annotation queues assigned to you.

Connect LangChain (LangSmith) free

Free plan, no credit card. Takes about a minute.

Try asking

“How many runs errored in the support-bot project in the last 24 hours, and what did they cost?”
The AI gets run statistics for the project in that time range: counts, errors, latency, tokens and cost.
“Add this question and the expected answer to the regression dataset.”
It finds the dataset and adds an example with the inputs, the expected output and a split if you name one.
“Show me the latest version of our summarizer prompt.”
It reads the prompt's content at the latest commit. It can also list earlier versions so you can compare.

What your AI can do

Tracing projects and feedback

Get run statistics per project, preview the runs in a conversation thread, and create, list and update feedback.

Datasets and examples

Search, create and update datasets, add, edit and delete examples, and export a dataset as CSV.

Prompts and annotation queues

Search the prompt hub, read and save prompt versions, create prompts, and create or look up annotation queues.

“Mark run 4f2a as incorrect and add a comment on why.”
It records feedback on the run with the key correctness, a score and your comment.
“Export the eval dataset as CSV.”
It downloads the dataset's examples as CSV, or as the dataset looked on an earlier date.
“Which annotation queues are assigned to me?”
It searches annotation queues assigned to you and can open one to show its rubric and settings.

26 tools for LangChain (LangSmith)

These tools are switched on when you connect. You can switch any of them off, or require your approval before it runs.

create_annotation_queueChanges dataCreate an annotation queue with a `name`, optional description, rubric instructions and reviewer settings.
create_datasetChanges dataCreate a dataset with a `name`, optional `description` and `data_type` (kv, llm or chat).
create_exampleChanges dataAdd an example to a dataset: `dataset_id` plus `inputs`, optional `outputs`, `metadata` and `split`.
create_feedbackChanges dataRecord feedback on a run: a feedback `key` (e.g. correctness) with `score` or `value`, optional `comment` and `run_id`.
create_promptChanges dataCreate a new prompt by `repo_handle` and `is_public`, with optional description, readme and tags.
delete_exampleDeletesPermanently delete one dataset example by `example_id`. Cannot be undone.
download_dataset_csvDownload a dataset's examples as CSV by `dataset_id`, optionally as of a past version (`as_of`).
get_annotation_queueGet one annotation queue by `queue_id`, including its rubric and settings.
get_datasetGet one dataset by `dataset_id`, including its description and example count.
get_exampleGet one dataset example by `example_id`, including its inputs and outputs.
get_feedbackGet one feedback entry by `feedback_id`.
get_promptGet a prompt's details by `owner` (tenant handle, or '-' for your own) and `repo` (prompt handle).
get_prompt_versionGet the content (manifest) of one prompt version by `owner`, `repo` and `commit` (hash or 'latest').
get_run_statsChanges dataGet aggregate run statistics (counts, latency, tokens, cost, errors) for one or more tracing projects in `session`, with optional time range and filters.
get_threadPreview a conversation thread's runs by `thread_id`.
get_tracing_projectGet a tracing project by `session_id`, optionally with run statistics (`include_stats`).
list_feedbackList feedback, filtered by `run`, `source`, score or comment presence, date range and paging.
list_prompt_commitsList the saved versions (commits) of a prompt by `owner` and `repo`.
save_prompt_versionChanges dataSave a new version of a prompt: `owner`, `repo` and the prompt `manifest`, optionally the `parent_commit`.
search_annotation_queuesSearch annotation queues by name, dataset or assignment (`assigned_to_me`).
search_datasetsSearch LangSmith datasets by `name_contains`, `name`, `id`, `data_type` or metadata, with limit/offset paging.
search_examplesList examples in a dataset (pass `dataset`), with optional `splits`, `metadata`, `full_text_contains` and paging.
search_promptsSearch prompts in the LangSmith prompt hub by `query`, tags, owner or visibility.
update_datasetChanges dataUpdate a dataset's `name`, `description` or metadata by `dataset_id`.
update_exampleChanges dataUpdate an example's `inputs`, `outputs`, `metadata` or `split` by `example_id`.
update_feedbackChanges dataUpdate a feedback entry's `score`, `value`, `comment` or `correction` by `feedback_id`.

Set up in three steps

1
Pick the app
Create a free PipMCP account and choose LangChain (LangSmith) from the app list.
2
Paste your key
Paste a LangSmith API key. In LangSmith open Settings > API Keys and click Create API Key. You can use a Personal Access Token or a Service Key scoped to a workspace. Copy the key right away, it is shown only once.
3
Add the link to your AI
You get a personal MCP link. Add it to Claude, ChatGPT or Cursor:
1. Click your name, then Settings › Connectors › Add custom connector.
2. Paste your link as the Remote MCP server URL.
3. Switch it on from the + menu in a chat.

Questions

What can the AI do in LangSmith?

It can read run statistics for tracing projects, preview conversation threads, and read datasets, examples, feedback, prompts and annotation queues. It can create datasets, examples, feedback, prompts, prompt versions and annotation queues, update datasets, examples and feedback, and delete examples if you switch that tool on.

Does the AI see my LangSmith API key?

No. Your LangSmith API key is encrypted at rest and never shown to the AI. After you save it, it is not shown again, not even to you. The AI only sees the results of the tools it calls.

Can I control what the AI is allowed to do?

Yes. You choose which tools are switched on, so you can start read-only. Deleting a dataset example is permanent and can require your approval before it runs. Every tool call is logged.

Does it work with ChatGPT?

Yes. In ChatGPT go to Settings › Apps & Connectors › Advanced and turn on Developer mode, then add your PipMCP link. Developer mode needs a paid ChatGPT plan: Plus, Pro, Business or Enterprise. The same link also works in Claude (Settings › Connectors › Add custom connector), Cursor and other MCP clients.

What does it cost?

PipMCP has a free plan with no credit card required. Paid plans bill per completed task. You also need your own LangSmith account.

Should I use a Personal Access Token or a Service Key?

Both work. A Personal Access Token acts as you. A Service Key is scoped to a workspace, which suits a shared setup where the access should not depend on one person. You create either under Settings > API Keys in LangSmith.