> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-sineha-1783732909-9486b1c.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage evaluators

> View and manage evaluators at the workspace level in LangSmith.

[Evaluators](/langsmith/evaluation-concepts#evaluators) in LangSmith are [workspace-level](/langsmith/administration-overview#workspaces) resources. You can attach a single evaluator to multiple [tracing projects](/langsmith/observability-concepts#projects) and [datasets](/langsmith/evaluation-concepts#datasets), so you can apply consistent evaluation logic across your work without recreating it each time.

<Tip>
  The [LangSmith Engine](/langsmith/engine) suggests custom evaluators for detected issues and can deploy them with one click.
</Tip>

## View evaluators

In the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-evaluators), select **Evaluators** in the left sidebar to view all evaluators in your workspace.

The evaluators table shows the following columns:

| Column                            | Description                                                                                                                                                                     |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Name                              | The evaluator name                                                                                                                                                              |
| Type                              | **LLM as a judge** or **Code**. Composite score evaluators are scoped to individual tracing projects and datasets and do not appear here.                                       |
| Feedback Key                      | The feedback key the evaluator produces                                                                                                                                         |
| Projects & Datasets               | Tracing projects and datasets this evaluator is attached to                                                                                                                     |
| Evaluator Trace Count (this week) | Number of traces this evaluator ran on in the past week. Only shown when spend tracking is enabled; **–** for Code evaluators or evaluators with no attached rules.             |
| Spend (this week)                 | Estimated USD spend for this evaluator in the past week. Only shown when spend tracking is enabled; **–** for Code evaluators or evaluators with no attached rules.             |
| Spend Status                      | Whether the evaluator is **Under limits**, **Unlimited**, or has hit one or more configured spend limits. Only shown when spend tracking is enabled; **–** for Code evaluators. |
| Created By                        | The workspace member who created the evaluator                                                                                                                                  |
| Updated At                        | When the evaluator was last modified                                                                                                                                            |
| Created At                        | When the evaluator was created                                                                                                                                                  |

## Create an evaluator

You can create an evaluator in the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-evaluators) or programmatically with the [SDK](#create-an-evaluator-with-the-sdk). Evaluators created either way are workspace-level resources that appear in the **Evaluators** table.

### Create an evaluator in the UI

1. In the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-evaluators), select **Evaluators** in the left sidebar.
2. Click **+ Evaluator** to open the new evaluator panel.
3. The panel lets you:
   * **Create from scratch**: Build a new [LLM-as-a-Judge](/langsmith/llm-as-judge) or [Code](/langsmith/online-evaluations-code) evaluator.
   * **Create from a template**: Start from a ready-made evaluator (also known as a prebuilt evaluator) for common evaluation patterns. A **Recommended** section surfaces popular templates first, followed by templates organized by the following categories:

     | Category          | Description                                          |
     | ----------------- | ---------------------------------------------------- |
     | Security          | Detect leaks, injections, and adversarial inputs.    |
     | Safety            | Evaluate content safety and moderation.              |
     | Quality           | Measure output quality and accuracy.                 |
     | Conversation      | Evaluate conversational quality and user experience. |
     | Trajectory        | Evaluate agent tool use and decision paths.          |
     | Image Evaluations | Evaluate image content quality and safety.           |
     | Voice Evaluation  | Evaluate voice and audio interaction quality.        |

You can also add an evaluator directly from a [tracing project](/langsmith/observability-concepts#projects) or [dataset](/langsmith/evaluation-concepts#datasets). In that flow, you can additionally **attach an existing evaluator** from your workspace, or create a [Composite](/langsmith/composite-evaluators-ui) evaluator. Refer to [Set up LLM-as-a-judge online evaluators](/langsmith/online-evaluations-llm-as-judge) and [Automatically run evaluators on experiments](/langsmith/bind-evaluator-to-dataset).

### Create an evaluator with the SDK

Use the LangSmith SDK to create evaluators programmatically. The SDK is available for [Python](/langsmith/smith-python-sdk) and [TypeScript](/langsmith/smith-js-ts-sdk). Evaluators created through the SDK appear in the **Evaluators** table alongside those created in the UI.

<Note>
  Managing evaluators through the SDK requires `langsmith>=0.9.8` (Python, PyPI) or `langsmith>=0.7.16` (TypeScript, npm).
</Note>

<CodeGroup>
  ```python Python theme={null}
  from langsmith import Client

  client = Client()

  created = client.evaluators.create(
      name="Correctness evaluator",
      type="code",
      code_evaluator={
          "code": "def perform_eval(run, example):\n    return {'score': 1}",
          "language": "python",
      },
  )
  print("Created evaluator:", created.evaluator.id)
  ```

  ```typescript TypeScript theme={null}
  import { Client } from "langsmith";

  const client = new Client();

  const created = await client.evaluators.create({
    name: "Correctness evaluator",
    type: "code",
    code_evaluator: {
      code: "def perform_eval(run, example):\n    return {'score': 1}",
      language: "python",
    },
  });
  console.log("Created evaluator:", created.evaluator?.id);
  ```
</CodeGroup>

To create an LLM-as-a-judge evaluator and to retrieve, update, list, or delete evaluators, refer to [Manage evaluators with the SDK](/langsmith/manage-evaluators-sdk).

## View evaluator details

Click any evaluator in the table to open its detail view. The detail view has four tabs:

* **Overview**: The evaluator's feedback configuration and prompt or code definition.
* **Traces**: Traces processed by this evaluator across all attached resources.
* **Logs**: Execution logs for this evaluator across all attached resources.
* **Projects & Datasets**: The tracing projects and datasets this evaluator is attached to.

## Edit an evaluator

Open an evaluator. In the **Overview** tab, click the **Edit evaluator** <Icon icon="pencil" /> icon to open the **Configure Evaluator** panel. Update the evaluator's configuration. Click **Save**.

Because the evaluator is shared, changes apply across all tracing projects and datasets it is attached to.

## Manage evaluator trace retention

When an online evaluator scores a trace, it attaches feedback to that trace and can [auto-upgrade the trace to extended retention](/langsmith/usage-and-billing#data-retention-auto-upgrades). Extended retention keeps the trace longer but costs more. When you set up an online evaluator on a [tracing project](/langsmith/observability-concepts#projects), you can opt out of this upgrade so that scored traces stay at the project's base retention.

This control is available only when the project's [default retention](/langsmith/billing#change-project-level-default-retention) is the [base tier](/langsmith/usage-and-billing#how-it-works). If the project defaults to extended retention ([set at the project or workspace level](/langsmith/data-purging-compliance#data-retention)), traces scored by the evaluator follow that default and the option is locked.

To opt out of extending retention for scored traces:

1. When you [create](#create-an-evaluator) or [edit](#edit-an-evaluator) an online evaluator, set the source to a [tracing project](/langsmith/observability-concepts#projects), rather than a [dataset](/langsmith/evaluation-concepts#datasets).
2. Expand the **Advanced** section in the evaluator configuration panel.
3. Clear **Extend trace retention**.

The change applies to traces scored after you save the evaluator. Existing scored traces keep their current retention tier.

This retention control applies to both trace online evaluators and thread online evaluators.

## Delete an evaluator

You cannot delete an evaluator while it is attached to a tracing project or dataset. To delete an evaluator:

1. In the [LangSmith UI](https://smith.langchain.com?utm_source=docs\&utm_medium=cta\&utm_campaign=langsmith-signup\&utm_content=langsmith-evaluators), select **Evaluators** in the left sidebar.
2. Select the evaluator you want to delete.
3. Open the **Projects & Datasets** tab. For each attached tracing project and dataset, select **Detach** in the **Actions** menu at the right of the row.
4. Return to the **Evaluators** page and click **Delete** at the top of the page.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/evaluators.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>