Before you start
Before you start, sign in to the dashboard and select the organization you want to work in.- The knowledge base needs at least one retriever with a live index. To compare retrievers, index at least two. See Configure a retriever.
- Collect some real questions your customers ask. They make the best test questions.
Create test questions
1
Open the knowledge base and select the Dataset tab
Here you build test questions, each paired with the passage that answers it. With none yet, the tab reads No test questions yet.
2
Select Create test questions
The Create dataset dialog opens on Basics.
3
Enter a Name and set Maximum questions
Maximum questions accepts 10 to 100. Tars generates up to that many from all documents in the knowledge base.
4
On Context, paste real questions under Real questions
Enter one question per line, up to 50. Selected questions keep their original wording. Generation limits can leave some questions out.
5
To vary the questions, add dimensions under Coverage
A dimension is a category, such as plan, with values, such as Free and Pro. Enter a website URL and select Auto-discover to suggest them.
6
Set the wording under Question style
Choose Question types, a Tone, and an optional Focus area. A preview line shows how a question will read.
7
Check the Review step and select Generate
The new set of test questions opens while Tars generates the questions.
Review the questions
The page sorts every test question into one of these groups. Select a group to filter the list.
A question under No answer found stays in the set so you can see the gap, but an experiment cannot score it. Select Edit on the question and point it at the right passage. Select Add question to write a question by hand.
Compare the generated set with the real questions you supplied. Use Add question for any missing questions you want to test.
Run an experiment
1
Select the Experiments tab, then Run experiment
The New experiment dialog opens.
2
Choose the test questions and the retrievers
Pick the set under Dataset, and the retrievers under Retrievers to compare. Only retrievers with a live index can run.
3
Tick the Metrics to report, then select Run experiment
With more than one retriever, the winner is the retriever with the best weighted score across the metrics you ticked.
An experiment card shows Queued, Running, Completed, Completed with errors, Failed, or Canceled. Open a finished experiment to compare each retriever’s Score, and select a retriever to see its results question by question.
Image scoring requires Media search to be on, completed image labels, and at least one relevant image marked for a question. The results show Media retrieval when image scores are available. Images come only from documents that the text retriever already found for a question.