> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wandb.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Set up annotation queues
> Create annotation queues, route traces to domain experts, and export structured feedback.
export const GitHubLink = ({url}) =>
GitHub source
;
export const ColabLink = ({url}) =>
Try in Colab
;
## Overview
This guide shows you how to set up annotation queues in Weave so you can collect structured feedback from domain experts on selected traces. Use annotation queues when you need expert-labeled data for evaluation, failure analysis, or dataset creation. Annotation queues also let reviewers focus on a curated set of traces rather than the full Weave UI.
Annotation queues let you route selected traces to domain experts for structured review without requiring them to navigate the full Weave UI. You define what feedback is collected, select which traces require review, and later can export completed annotations for analysis or dataset creation. Use cases can include:
* **Manual trace scoring**: Have SMEs rate model outputs on correctness, quality, or style.
* **Failure analysis**: Annotate failure modes (hallucinations, refusals, loops) to understand where your model breaks.
* **Domain expert review**: Enable medical, legal, or safety experts to review content with a task-focused interface.
* **Dataset creation**: Turn annotated traces into evaluation or training datasets.
## End-to-end workflow
The following workflow summarizes how to use annotation queues to obtain reviews. The following sections walk through each step in detail.
1. Define annotation fields.
2. Create an annotation queue.
3. Load traces into the queue for review.
4. Monitor progress while domain experts complete reviews.
5. Filter and export completed annotations.
## Define annotation fields
Before you create an annotation queue, define your annotation fields so they can be selected during queue setup. Annotation fields define the feedback that the annotator provides for each trace item. Fields are reusable across queues and projects.
Field types include:
* Boolean judgments such as correctness or acceptability.
* Numeric or integer values such as quality or confidence.
* Categorical labels such as failure mode or intent.
* Free-form text for qualitative feedback.
To create an annotation field:
1. Navigate to [wandb.ai](https://wandb.ai) and select your project.
2. In the Weave project sidebar, click **Annotate**. If you don't see **Annotate**, it might be nested in the menu under **More**.
3. In the tab bar, click the **Fields** tab.
4. In the Fields table toolbar, click **New Field**.
5. In the **Create annotation field** modal dialog, configure:
* **Type**: Boolean, Integer, Number, String, or categorical options.
* **Name**: Name of field to be displayed to the annotator.
* **Description** (Optional): Details for evaluating this field to be displayed to the annotator.
6. Click **Create annotation field** to save the field.
The new field appears in the **Fields** tab and is available to add to any annotation queue in the project.
Fields can't be edited after creation to ensure annotation consistency.
## Create an annotation queue
With your annotation fields defined, the next step is to create the queue that annotators work from. An annotation queue consists of:
* A set of annotation fields.
* Guidelines that provide task instructions for annotators.
* A collection of trace items awaiting review.
To create an annotation queue:
1. In the Weave project sidebar, click **Annotate**.
2. In the tab bar, click the **Queues** tab.
3. In the Queues table toolbar, click **Create Queue**.
4. In the **Create Annotation Queue** modal dialog, configure:
* **Queue name**: This is the queue name the annotator selects to complete their work.
* **Guidelines** (Optional): Any additional instructions for the annotator.
5. Click **Next**.
6. Click **Manage fields** and choose what annotation fields to include in this review work. All existing annotation fields for the project are available for selection.
7. After you have selected all the fields for the queue, click **Create Queue** to save the queue.
All annotation queues for the project are listed in the Annotation Queues page.
Creating an annotation queue defines the fields and guidelines for evaluation, but you still need to add traces to the queue to identify the data to evaluate.
### Add traces to a queue
Once a queue exists, you populate it with the specific traces you want reviewed. Add traces to an annotation queue directly from the **Traces** page.
To add traces to an annotation queue:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, filter traces as needed (such as by hallucination scores, failure modes, or specific ops).
3. In the table, select the traces you want annotated.
4. In the table's action bar, click **Add to queue** to add the selected rows to an annotation queue.
5. In the **Add to annotation queue** modal dialog, configure:
* **Queue Name**: In the list, select the name of the existing queue to add these traces to.
* **Select trace data to display**: Select the Weave trace data elements to display to the annotator as they evaluate the results.
* **Inputs**: Select which trace input fields to show during annotation.
* **Outputs**: Select which trace output fields to show during annotation.
6. Click **Add `[COUNT]` traces to annotation queue** to assign these traces as a part of the annotation queue review.
When you add traces, you control which trace inputs and which outputs or model responses are reviewed, so annotators see only the context needed to make accurate judgments.
## Monitor review progress
After traces are added, annotators can begin work and you can track how the review progresses. Share the queue name with your annotator so they can begin their review. See [Review items in an annotation queue](/weave/guides/tracking/annotation-review) for details on the review process.
To share a direct link to the annotation queue with an annotator:
1. In the Weave project sidebar, click **Annotate**.
2. In the tab bar, click the **Queues** tab.
3. In the Annotation Queues table, click the name of your queue to open the queue items.
4. In the Queue header bar, click the link button to copy a direct link to this queue. You can also copy the URL from the browser address bar.
In the Annotation Queues table, the **State** column indicates reviewing progress:
* **Not started**: Queue has items but no annotations have been submitted.
* **In progress**: At least one item has been reviewed.
* **Completed**: All items have been reviewed.
In the Annotation Queues table, the **Calls with responses** column indicates the percent of items (out of the total number of **Calls**) that have at least one submitted review.
## Filter and export annotations
After annotators have submitted reviews, you can use the resulting feedback for analysis, evaluation, or training. Weave stores completed annotations as structured metadata on traces.
You can:
* Filter traces by queue assignment and annotation completion.
* Save filtered views for reuse.
* Export annotated traces to datasets for evaluation or training workflows.
This connects expert human feedback directly to model evaluation and iteration.
### Filter annotated traces
You can use the filter controls of the **Traces** page to display only traces with annotations.
To view only traces with annotations:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, click **Filter**.
3. Add three values to a filter row:
* For **Column**, type "Queue", then press **Enter**.
* For the second list, choose **Text: "is"**.
* For **Select a queue**, choose your annotation queue name.
4. To also filter on 'completed' queue items only, click **+ Add Filter**:
* For **Column**, type "feedback". A dialog populates with annotations and includes your annotation field names. Choose a required field from your queue.
* For the second list, choose **Other: "is not empty"**.
5. Filter rows are automatically applied. Click elsewhere in the page for the filter entry to close.
6. Optional: Save as a view for quick access. In the Traces table header, click **Save View**.
### Export annotated traces to datasets
You can export annotated traces either through the UI or programmatically, depending on how you plan to use the data.
#### Add annotated traces to a dataset
Select annotated traces and click **Add to dataset** to include expert labels in your eval or training data.
To add annotated traces to a dataset:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table, select the traces that you want to export.
3. In the table toolbar, click **Add to dataset**. Follow the on-screen prompts to complete the addition.
To learn more about using datasets, see [Collect and track datasets](/weave/guides/core-types/datasets).
## Access annotations programmatically
In addition to the UI workflow, you can read annotation data programmatically. This is useful for automating analysis, building custom reports, or integrating annotations into downstream pipelines. To integrate annotations programmatically, you must know your project name and queue ID:
* **Project**: The W\&B project name (can be project or team/project). If you don't specify a W\&B team (such as "team/project"), your default team is used.
* **Queue ID**: The annotation queue's unique identifier.
To find the queue ID:
1. In the **Annotation Queues** table, select the name of the queue to open its items.
2. Copy the ID from the end of the page URL.
**Example**
```text theme={null}
https://wandb.ai/.../annotation-queues/019c0f63-7acb-7497-8f87-08873368fcd4
```
In this example, the queue ID is:
`019c0f63-7acb-7497-8f87-08873368fcd4`
You can iterate through the traces (calls) in your queue using the following code.
```python lines theme={null}
import weave
from weave.trace_server.trace_server_interface import AnnotationQueueItemsQueryReq
# Update project and queue identifiers to your own values.
PROJECT = "your-team-name/your-project-name"
QUEUE_ID = "019c0f63-7acb-7497-8f87-08873368fcd4"
# Initialize Weave.
client = weave.init(PROJECT)
# Get call IDs from calls in a queue.
calls = client.server.annotation_queue_items_query(
AnnotationQueueItemsQueryReq(
project_id=PROJECT,
queue_id=QUEUE_ID,
)
)
# Iterate through calls and get feedback.
for i, item in enumerate(calls.items):
call = client.get_call(call_id=item.call_id, include_feedback=True)
feedback = call.feedback or {}
# Count total feedback items for call.
total_feedback_items = len(feedback)
print(f"\nItem {i}, call_id: {item.call_id}, total feedback items: {total_feedback_items}")
if not feedback:
print("No feedback for item")
continue
# Get the first annotation value.
# Field annotations are not added in repeatable order, thus the first field will vary.
field_name = next(iter(feedback))
field_value = feedback[0]
print(f" {field_name}: {field_value}")
```
The following Google Colab notebook provides an in-depth example of programmatically accessing annotation data and collecting an analytics report:
The new field appears in the **Fields** tab and is available to add to any annotation queue in the project.
Creating an annotation queue defines the fields and guidelines for evaluation, but you still need to add traces to the queue to identify the data to evaluate.
### Add traces to a queue
Once a queue exists, you populate it with the specific traces you want reviewed. Add traces to an annotation queue directly from the **Traces** page.
To add traces to an annotation queue:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, filter traces as needed (such as by hallucination scores, failure modes, or specific ops).
3. In the table, select the traces you want annotated.
4. In the table's action bar, click **Add to queue** to add the selected rows to an annotation queue.
5. In the **Add to annotation queue** modal dialog, configure:
* **Queue Name**: In the list, select the name of the existing queue to add these traces to.
* **Select trace data to display**: Select the Weave trace data elements to display to the annotator as they evaluate the results.
* **Inputs**: Select which trace input fields to show during annotation.
* **Outputs**: Select which trace output fields to show during annotation.
6. Click **Add `[COUNT]` traces to annotation queue** to assign these traces as a part of the annotation queue review.
When you add traces, you control which trace inputs and which outputs or model responses are reviewed, so annotators see only the context needed to make accurate judgments.
## Monitor review progress
After traces are added, annotators can begin work and you can track how the review progresses. Share the queue name with your annotator so they can begin their review. See [Review items in an annotation queue](/weave/guides/tracking/annotation-review) for details on the review process.
To share a direct link to the annotation queue with an annotator:
1. In the Weave project sidebar, click **Annotate**.
2. In the tab bar, click the **Queues** tab.
3. In the Annotation Queues table, click the name of your queue to open the queue items.
4. In the Queue header bar, click the link button to copy a direct link to this queue. You can also copy the URL from the browser address bar.
In the Annotation Queues table, the **State** column indicates reviewing progress:
* **Not started**: Queue has items but no annotations have been submitted.
* **In progress**: At least one item has been reviewed.
* **Completed**: All items have been reviewed.
In the Annotation Queues table, the **Calls with responses** column indicates the percent of items (out of the total number of **Calls**) that have at least one submitted review.
## Filter and export annotations
After annotators have submitted reviews, you can use the resulting feedback for analysis, evaluation, or training. Weave stores completed annotations as structured metadata on traces.
You can:
* Filter traces by queue assignment and annotation completion.
* Save filtered views for reuse.
* Export annotated traces to datasets for evaluation or training workflows.
This connects expert human feedback directly to model evaluation and iteration.
### Filter annotated traces
You can use the filter controls of the **Traces** page to display only traces with annotations.
To view only traces with annotations:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table toolbar, click **Filter**.
3. Add three values to a filter row:
* For **Column**, type "Queue", then press **Enter**.
* For the second list, choose **Text: "is"**.
* For **Select a queue**, choose your annotation queue name.
4. To also filter on 'completed' queue items only, click **+ Add Filter**:
* For **Column**, type "feedback". A dialog populates with annotations and includes your annotation field names. Choose a required field from your queue.
* For the second list, choose **Other: "is not empty"**.
5. Filter rows are automatically applied. Click elsewhere in the page for the filter entry to close.
6. Optional: Save as a view for quick access. In the Traces table header, click **Save View**.
### Export annotated traces to datasets
You can export annotated traces either through the UI or programmatically, depending on how you plan to use the data.
#### Add annotated traces to a dataset
Select annotated traces and click **Add to dataset** to include expert labels in your eval or training data.
To add annotated traces to a dataset:
1. In the Weave project sidebar, click **Traces**.
2. In the Traces table, select the traces that you want to export.
3. In the table toolbar, click **Add to dataset**. Follow the on-screen prompts to complete the addition.
To learn more about using datasets, see [Collect and track datasets](/weave/guides/core-types/datasets).
## Access annotations programmatically
In addition to the UI workflow, you can read annotation data programmatically. This is useful for automating analysis, building custom reports, or integrating annotations into downstream pipelines. To integrate annotations programmatically, you must know your project name and queue ID:
* **Project**: The W\&B project name (can be project or team/project). If you don't specify a W\&B team (such as "team/project"), your default team is used.
* **Queue ID**: The annotation queue's unique identifier.
To find the queue ID:
1. In the **Annotation Queues** table, select the name of the queue to open its items.
2. Copy the ID from the end of the page URL.
**Example**
```text theme={null}
https://wandb.ai/.../annotation-queues/019c0f63-7acb-7497-8f87-08873368fcd4
```
In this example, the queue ID is:
`019c0f63-7acb-7497-8f87-08873368fcd4`
You can iterate through the traces (calls) in your queue using the following code.
```python lines theme={null}
import weave
from weave.trace_server.trace_server_interface import AnnotationQueueItemsQueryReq
# Update project and queue identifiers to your own values.
PROJECT = "your-team-name/your-project-name"
QUEUE_ID = "019c0f63-7acb-7497-8f87-08873368fcd4"
# Initialize Weave.
client = weave.init(PROJECT)
# Get call IDs from calls in a queue.
calls = client.server.annotation_queue_items_query(
AnnotationQueueItemsQueryReq(
project_id=PROJECT,
queue_id=QUEUE_ID,
)
)
# Iterate through calls and get feedback.
for i, item in enumerate(calls.items):
call = client.get_call(call_id=item.call_id, include_feedback=True)
feedback = call.feedback or {}
# Count total feedback items for call.
total_feedback_items = len(feedback)
print(f"\nItem {i}, call_id: {item.call_id}, total feedback items: {total_feedback_items}")
if not feedback:
print("No feedback for item")
continue
# Get the first annotation value.
# Field annotations are not added in repeatable order, thus the first field will vary.
field_name = next(iter(feedback))
field_value = feedback[0]
print(f" {field_name}: {field_value}")
```
The following Google Colab notebook provides an in-depth example of programmatically accessing annotation data and collecting an analytics report: