Ragit Benchmark Tests

1st chunk of `tests/benchmark/README.md`

f361e29530a41c91b85f232a1f75f6534b8e30d6f466eb2b000000010000024f

# Ragit Benchmarks

TODO

## 1. Rerank Summary

A testset consists of (summaries, query, relevant summaries).

It runs `rerank_summary.pdl` with the set, and counts how many tuples it get correct. The problem is that it's tough to tell whether a summary is relenvant or not. It might seem relevant to someone and not relevant to another.

## 2. End-to-End

A testset consists of a knowledge-base and a set of questions and answers (multi-choice).

It first runs the testset without RAG, then with RAG.

It counts the number of questions that it got correct with RAG, but failed without RAG.

Title: Ragit Benchmark Tests

Summary

The Ragit benchmarks consist of two tests: Rerank Summary, which evaluates the relevance of summaries, and End-to-End, which compares the performance of a system with and without RAG on a knowledge-base and question set.