[Evals] CLI for scoring custom datasets

[yanxi0830]

TL;DR

• Update dataset register CLI to have flexibility to take input files
• Run app eval on custom dataset
  • (1) via --dataset-path to read rows locally
  • (2) via --dataset-id to read

Test

Run on custom dataset

llama-stack-client eval run_scoring braintrust::answer-correctness braintrust::factuality \
--dataset-path <path-to-local-dataset> \
--output-dir ./ \
--num-examples 5

1. Register a custom dataset

llama-stack-client datasets register --dataset-id <custom-dataset-id> \
--schema '{"generated_answer": {"type": "string"}, "expected_answer": {"type": "string"}, "input_query": {"type": "string"}}' \
--dataset-path <local-file-path> \
--provider-id huggingface-0

2. Run scoring on custom dataset

llama-stack-client eval run_scoring \
braintrust::answer-correctness braintrust::factuality \
--dataset-id <custom-dataset-id> --output-dir ./ 

[ashwinb]

lg!