Original Mostly Basic Python Problems. Its original and sanitized subsets have different membership; MBPP+ and EvalPlus variants are not assigned this guide.
Everyday Python building blocks
The original dataset contains 974 crowdsourced tasks intended for entry-level programmers. Problems exercise short programs, programming fundamentals and standard-library use. Each release entry includes a natural-language description, a reference solution and three tests.
Program Synthesis with Large Language Models · MBPP methods and analysis · Google Research MBPP · Dataset files, splits and prompts
Use the declared test split
The official split uses task IDs 11–510 for testing; other IDs serve prompting, validation or training. A separate hand-checked sanitized file also exists. A score on every available task is not automatically the published test protocol.
Passing tests and generating many candidates
The study evaluates whether generated Python satisfies the tests, including settings with many sampled candidates. Report how many candidates were allowed and how a final program was selected; a success found after many attempts is different from one-shot reliability.
Program Synthesis with Large Language Models · MBPP methods and analysis
Examples change the task
The reference setup uses few-shot examples, and the paper studies the effects of prompt examples and visible tests. Keep these inputs fixed when comparing models; supplying assertions gives additional information beyond the prose request.
Google Research MBPP · Dataset files, splits and prompts · Program Synthesis with Large Language Models · MBPP methods and analysis
Three tests do not cover every case
The paper reports cases of programs overfitting assertions and examines edits to ambiguous tasks. Inspect failures and edge cases rather than equating test success with complete correctness. MBPP's short tasks offer limited evidence about multi-file engineering work.
Program Synthesis with Large Language Models · MBPP methods and analysis