> ## Documentation Index
> Fetch the complete documentation index at: https://hellotars.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Measure retrieval quality

> Build a set of test questions, then run an experiment that scores your retrievers against them.

Use this before you change the default retriever. **Test retriever** lets you inspect retrieved chunks for one question at a time. An experiment compares retrievers across a set of test questions.

## Before you start

Before you start, sign in to [the dashboard](https://dashboard.hellotars.com) and select the organization you want to work in.

* The knowledge base needs at least one retriever with a live index. To compare retrievers, index at least two. See [Configure a retriever](https://hellotars.com/docs/guides/knowledge/configure-a-retriever.md).
* Collect some real questions your customers ask. They make the best test questions.

## Create test questions

**1. Open the knowledge base and select the Dataset tab**

Here you build test questions, each paired with the passage that answers it. With none yet, the tab reads **No test questions yet**.

**2. Select Create test questions**

The **Create dataset** dialog opens on **Basics**.

**3. Enter a Name and set Maximum questions**

**Maximum questions** accepts 10 to 100. Tars generates up to that many from all documents in the knowledge base.

**4. On Context, paste real questions under Real questions**

Enter one question per line, up to 50. Selected questions keep their original wording. Generation limits can leave some questions out.

**5. To vary the questions, add dimensions under Coverage**

A dimension is a category, such as plan, with values, such as Free and Pro. Enter a website URL and select **Auto-discover** to suggest them.

**6. Set the wording under Question style**

Choose **Question types**, a **Tone**, and an optional **Focus area**. A preview line shows how a question will read.

**7. Check the Review step and select Generate**

The new set of test questions opens while Tars generates the questions.

![The Create dataset dialog on Basics, with a Name field and Maximum questions set to 50](https://hellotars.com/docs/docs/guides/knowledge/images/create-test-questions.png)

You can define up to six dimensions. Their values can combine into at most 1,000 variations.

## Review the questions

The page sorts every test question into one of these groups. Select a group to filter the list.

| Group | Meaning |
| - | - |
| **Ready to score** | The question and its answer passage both still hold. |
| **Needs review** | The source document changed after the answer passage was set. |
| **No answer found** | Tars found no answer in your documents. This can point to a gap in the knowledge base. |
| **Source deleted** | The document that held the answer is gone. |

A question under **No answer found** stays in the set so you can see the gap, but an experiment cannot score it. Select **Edit** on the question and point it at the right passage. Select **Add question** to write a question by hand.

Compare the generated set with the real questions you supplied. Use **Add question** for any missing questions you want to test.

## Run an experiment

**1. Select the Experiments tab, then Run experiment**

The **New experiment** dialog opens.

**2. Choose the test questions and the retrievers**

Pick the set under **Dataset**, and the retrievers under **Retrievers to compare**. Only retrievers with a live index can run.

**3. Tick the Metrics to report, then select Run experiment**

With more than one retriever, the winner is the retriever with the best weighted score across the metrics you ticked.

| Metric | What it measures |
| - | - |
| **Recall** | How much of the marked answer text the retriever found. |
| **Precision** | The share of returned text that overlaps the marked answer text. Higher means less noise. |
| **MRR** | How high the first correct answer appears in the results. |
| **Hit Rate** | The share of questions where at least one marked answer was retrieved. |

An experiment card shows **Queued**, **Running**, **Completed**, **Completed with errors**, **Failed**, or **Canceled**. Open a finished experiment to compare each retriever's **Score**, and select a retriever to see its results question by question.

Image scoring requires **Media search** to be on, completed image labels, and at least one relevant image marked for a question. The results show **Media retrieval** when image scores are available. Images come only from documents that the text retriever already found for a question.

## Verify

Open the finished experiment. Each retriever shows a **Score**, and the retriever with the best score ranks first. Make that retriever the default with **Make default** on the **Retriever** tab.

## Related pages

* [Retrievers explained](https://hellotars.com/docs/guides/knowledge/retrievers-explained.md)
* [Configure a retriever](https://hellotars.com/docs/guides/knowledge/configure-a-retriever.md)
* [Manage documents](https://hellotars.com/docs/guides/knowledge/manage-documents.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.