The 500-problem test split released with PRM800K; not the complete MATH test set or the PRM800K training labels.
Problems and expected answers
Each task asks for a mathematical solution with a final answer. The set samples MATH competition problems across subjects and difficulty levels, rather than testing only arithmetic or one competition year.
Why exactly 500 questions
The authors moved 4,500 original MATH test problems into training and held out 500 selected uniformly at random. The released test.jsonl identifies this specific split; an arbitrary 500-question sample is a different evaluation.
How answers are checked
The released grader normalizes mathematical text and uses symbolic equivalence checks. Its authors explicitly acknowledge false rejections and possible false acceptances. A different answer extractor or equality checker can change which responses receive credit.
One answer versus selected answers
The introducing study also evaluates reward models selecting among many generated solutions. Such best-of-N results combine generation and selection. Compare them with other best-of-N runs at the same budget, rather than treating them as single-attempt accuracy.
What the number leaves out
Final-answer agreement does not verify every reasoning step. Read solution traces when reasoning reliability matters, and keep the test split separate from process-supervision labels. The original paper discusses contamination and evaluates its own exposure checks.
Let’s Verify Step by Step · Evaluation and contamination · OpenAI PRM800K · MATH splits and answer grading