Model reports

ProSearch

Find an experimental method that meets every requested condition within the same setup. A paragraph mentioning the right ingredients can still describe the wrong experiment.

Original paper Official repository Official dataset Catalog source

AI Benchmark Radar / Equality Machine · Reviewed 2026-11-02

Paper v1, submitted 30 October 2026; method-paragraph retrieval and explicitly labeled alternative settings.

What must be retrieved?

Rank experimental-method paragraphs by joint constraint satisfaction. The queries specify 3–5 requirements; matching values scattered across different experimental variants does not qualify.

ProSearch paper v1 · Official ProSearch code and reproduction instructions

Queries, domains and evidence

The evaluation has 1,515 queries across six laboratory domains. Each domain supplies paragraph pools, queries and relevance judgments. Thirty-six generated queries without judgments were excluded.

ProSearch paper v1 · Official ProSearch dataset and files

How to reproduce the evaluation

The author scripts encode queries and paragraphs, save retrieval runs, then score against qrels. Choose raw queries and the method-paragraph pool for the main setting; full-body retrieval changes the corpus.

Official ProSearch code and reproduction instructions · Official ProSearch dataset and files

Read the score with its setting

nDCG@10 evaluates the top ten ranks. The primary series uses the stated arithmetic mean across six domains, preserving decimals. Query and reranking configurations remain explicit; diagnostic and corpus-shift results appear separately below.

ProSearch paper v1 · Official ProSearch code and reproduction instructions

All 19 first-stage retrievers

The complete original-query results retain all six domains. The comparable primary series uses the average column.

Table 6 · original-query nDCG@10
Retriever	Cell culture	Immunotechnology	Electrochemistry	Organic synthesis	Inorganic materials	Polymer materials	Average
BM25	0.136	0.011	0.079	0.132	0.217	0.297	0.145
Sentence-BERT	0.082	0.012	0.064	0.122	0.263	0.065	0.101
GTE-large	0.222	0.041	0.127	0.184	0.441	0.182	0.199
BGE-M3	0.254	0.025	0.106	0.216	0.369	0.326	0.216
Harrier-OSS-v1-0.6B	0.261	0.049	0.159	0.198	0.467	0.177	0.219
NV-Embed-v2	0.358	0.070	0.184	0.290	0.448	0.310	0.277
Qwen3-Embedding-8B	0.386	0.092	0.175	0.282	0.495	0.309	0.290
MatSciBERT	0.011	0.003	0.001	0.013	0.002	0.002	0.005
ChemicalBERT-uncased	0.043	0.001	0.027	0.026	0.040	0.055	0.032
SPECTER2	0.077	0.005	0.003	0.069	0.157	0.060	0.062
SciNCL	0.098	0.034	0.036	0.099	0.190	0.020	0.080
PubMedBERT-base-embeddings	0.069	0.001	0.039	0.104	0.213	0.059	0.081
BioClinical-ModernBERT	0.142	0.011	0.053	0.093	0.272	0.084	0.109
ChEmbed-prog	0.161	0.007	0.056	0.179	0.401	0.147	0.158
ReasonIR-8B	0.079	0.006	0.021	0.120	0.092	0.043	0.060
INF-X-Retriever	0.295	0.039	0.134	0.208	0.328	0.193	0.200
ReasonEmbed	0.268	0.066	0.150	0.201	0.425	0.164	0.212
BGE-Reasoner-Embed	0.285	0.069	0.152	0.240	0.435	0.215	0.233
DIVER-Retriever	0.284	0.072	0.170	0.262	0.476	0.267	0.255

Domain columns are separate subsets. Only the six-domain average enters the reported-results series. Values remain decimals.

ProSearch Table 6

Query formulations with one retriever

Every row uses Qwen3-Embedding-8B and the method-paragraph corpus. Query expansion changes the input, not the model checkpoint.

Table 7 · query-formulation nDCG@10
Query formulation	Cell culture	Immunotechnology	Electrochemistry	Organic synthesis	Inorganic materials	Polymer materials	Average
Original	0.386	0.092	0.175	0.282	0.495	0.309	0.290
Query Decomposition	0.317	0.068	0.091	0.205	0.454	0.254	0.232
Structured Query	0.324	0.080	0.138	0.250	0.494	0.189	0.246
INF-Query-Aligner	0.382	0.075	0.141	0.233	0.487	0.196	0.252
HyDE	0.359	0.098	0.154	0.262	0.501	0.255	0.272
TongSearch-QR	0.385	0.092	0.172	0.265	0.505	0.288	0.285
Reasoning-augmented	0.408	0.111	0.179	0.309	0.494	0.359	0.310
DIVER-QExpand	0.374	0.117	0.205	0.327	0.558	0.318	0.317

The Original row repeats the baseline for reference; it is not duplicated in the primary series.

ProSearch Table 7

What changes after reranking?

Rerankers reorder Qwen3-Embedding-8B’s top 100 candidates. Their scores include that first-stage retrieval; they are not standalone retriever scores.

Table 8 · nDCG@10 after top-100 reranking
Retriever	Cell culture	Immunotechnology	Electrochemistry	Organic synthesis	Inorganic materials	Polymer materials	Average
Qwen3-Embedding-8B	0.386	0.092	0.175	0.282	0.495	0.309	0.290
BGE-Reranker-v2-m3	0.481	0.192	0.247	0.336	0.705	0.416	0.396
Rank-R1	0.521	0.227	0.269	0.408	0.622	0.478	0.421
Rank1	0.592	0.266	0.241	0.416	0.598	0.457	0.428
DIVER-GroupRank-7B	0.586	0.255	0.262	0.468	0.705	0.531	0.468
ReasonRank-8B	0.577	0.276	0.314	0.483	0.637	0.547	0.472
Qwen3-Reranker	0.579	0.301	0.294	0.485	0.692	0.490	0.474

Domain columns are separate subsets. Only the six-domain average enters the reported-results series. Values remain decimals.

ProSearch Table 8

Changing the candidate corpus

Full-body retrieval is harder because it includes paragraphs outside the methods section. These are separate corpus settings, displayed outside the main method-paragraph series.

Table 9 · six-domain mean by corpus
Retriever	Method-only nDCG@10	All-body nDCG@10
BM25	0.145	0.089
BGE-Reasoner-Embed	0.233	0.164
NV-Embed-v2	0.277	0.197
Qwen3-Embedding-8B	0.290	0.206

Preserves the four directly reported average pairs.

ProSearch Table 9

Open-world search on 30 queries

This small experiment samples five queries from each domain. Its success, precision and reciprocal-rank metrics have different meanings from nDCG@10.

Table 10 · open-world evidence retrieval
Method	Success@10	P@10	MRR
Tavily Search	0.300	0.077	0.209
Tavily Agent	0.800	0.353	0.694

No values from this table enter the nDCG@10 series.

ProSearch Table 10

Structured information diagnostic

Structured matches query constraints within one extracted protocol record and averages 20 randomized tie orderings. Hybrid breaks ties with Qwen3-Embedding-8B. Both consume extracted protocol information and are not text-only retriever baselines.

Table 11 · diagnostic nDCG@10
Method	Cell culture	Electrochemistry	Immunotechnology	Inorganic materials	Organic synthesis	Polymer materials	Average
Qwen3-Embedding-8B	0.386	0.175	0.092	0.495	0.282	0.309	0.290
Structured Diagnostic	0.789	0.698	0.682	0.875	0.773	0.829	0.774
Hybrid Diagnostic	0.858	0.741	0.723	0.910	0.795	0.859	0.814

Diagnostic values are shown separately and excluded from the main results series. Note the source-specific domain order.

ProSearch Table 11

Constraint-role subsets

Queries are split by whether Condition or Method constraints are present. Each cell is a subset mean across six domains, not the full-query-set average.

Table 39 · role subsets
Retriever	Condition absent	Condition present	Method absent	Method present
BM25	0.181	0.136	0.132	0.159
Qwen3-Embedding-8B	0.356	0.285	0.288	0.293

Qwen3 is expanded to the checkpoint identified in the source caption. Subsets are excluded from the primary series.

ProSearch Table 39

What this does not establish

Templates and LLM-assisted pooled judgments can miss real needs or relevant paragraphs. Expert label agreement was 89.1%. Coverage is limited to selected procedures and single-paragraph evidence, not complete experimental reproducibility.

ProSearch paper v1

Version and model identity

The publication date is the v1 submission date. Checkpoints follow Appendix C.1/C.3; per-run dates are not provided. No later leaderboard values were substituted.

ProSearch paper v1 · arXiv submission and version history

Original publication

Original v1 publications; paper results retain their stated configurations.

Reviewed 2026-11-02

Evidence available for this benchmark

This source record contains 32 numeric observations. The source distinguishes 25 model identities or configurations. A model may appear more than once; check the version, tools and run budget before comparing. Read results and row-level evidence →

Reported metric: Average nDCG@10 (six domains)
Recorded unit: ratio
Metric direction: higher is better

Start with these resources

Catalog source · arxiv.org

Cited documents and dates (1)

ProSearch paper v1 · retrieval, query formulation and reranking results · 2026-10-30

Evaluation context and evidence

Source: Model reports
Publisher: Not recorded
Benchmark release: 2026-10-30
Collected: Not recorded
Modality: Not recorded

These are source-reported observations, not independent reruns by this site. Check the task version, evaluation protocol, unit and reporting date before comparing models. Results from different setups are not a universal ranking.

Reported results 32

Shown in source observation order, not ranked. Dates retain the precision provided by the source.

Model	Value	Unit	Protocol	Reported date and basis	Source
DIVER-Retriever	0.255	ratio	Original query; first-stage retrieval; checkpoint or implementation AQ-MedAI/Diver-Retriever-4B; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
GTE-large	0.199	ratio	Original query; first-stage retrieval; checkpoint or implementation Alibaba-NLP/gte-large-en-v1.5; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
BGE-M3	0.216	ratio	Original query; first-stage retrieval; checkpoint or implementation BAAI/bge-m3; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
BGE-Reasoner-Embed	0.233	ratio	Original query; first-stage retrieval; checkpoint or implementation BAAI/bge-reasoner-embed-qwen3-8b-0923; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
ChEmbed-prog	0.158	ratio	Original query; first-stage retrieval; checkpoint or implementation BASF-AI/ChEmbed-prog; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
PubMedBERT-base-embeddings	0.081	ratio	Original query; first-stage retrieval; checkpoint or implementation NeuML/pubmedbert-base-embeddings; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.29	ratio	Original query; first-stage retrieval; checkpoint or implementation Qwen/Qwen3-Embedding-8B; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
SPECTER2	0.062	ratio	Original query; first-stage retrieval; checkpoint or implementation allenai/specter2_base; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
ReasonEmbed	0.212	ratio	Original query; first-stage retrieval; checkpoint or implementation hanhainebula/reason-embed-qwen3-8b-0928; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
INF-X-Retriever	0.2	ratio	Original query; first-stage retrieval; checkpoint or implementation infly/inf-retriever-v1-pro; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
MatSciBERT	0.005	ratio	Original query; first-stage retrieval; checkpoint or implementation m3rg-iitd/matscibert; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
SciNCL	0.08	ratio	Original query; first-stage retrieval; checkpoint or implementation malteos/scincl; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Harrier-OSS-v1-0.6B	0.219	ratio	Original query; first-stage retrieval; checkpoint or implementation microsoft/harrier-oss-v1-0.6b; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
NV-Embed-v2	0.277	ratio	Original query; first-stage retrieval; checkpoint or implementation nvidia/NV-Embed-v2; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
BM25	0.145	ratio	Original query; first-stage retrieval; checkpoint or implementation rank_bm25.BM25Okapi; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
ReasonIR-8B	0.06	ratio	Original query; first-stage retrieval; checkpoint or implementation reasonir/ReasonIR-8B; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
ChemicalBERT-uncased	0.032	ratio	Original query; first-stage retrieval; checkpoint or implementation recobo/chemical-bert-uncased; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Sentence-BERT	0.101	ratio	Original query; first-stage retrieval; checkpoint or implementation sentence-transformers/all-mpnet-base-v2; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
BioClinical-ModernBERT	0.109	ratio	Original query; first-stage retrieval; checkpoint or implementation thomas-sounack/BioClinical-ModernBERT-base; model-specific query handling in Appendix C.1; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 6	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.317	ratio	Query formulation DIVER-QExpand; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.272	ratio	Query formulation HyDE; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.252	ratio	Query formulation INF-Query-Aligner; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.232	ratio	Query formulation Query Decomposition; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.31	ratio	Query formulation Reasoning-augmented; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.246	ratio	Query formulation Structured Query; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Embedding-8B	0.285	ratio	Query formulation TongSearch-QR; Qwen/Qwen3-Embedding-8B; original method-paragraph corpus; Appendix C.2 transformation and retrieval procedure; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 7	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
DIVER-GroupRank-7B	0.468	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint AQ-MedAI/Diver-GroupRank-7B; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
BGE-Reranker-v2-m3	0.396	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint BAAI/bge-reranker-v2-m3; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Qwen3-Reranker	0.474	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint Qwen/Qwen3-Reranker-8B; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Rank-R1	0.421	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint ielabgroup/Rank-R1-7B-v0.1; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
Rank1	0.428	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint jhu-clsp/rank1-7b; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results
ReasonRank-8B	0.472	ratio	Rerank the same top-100 candidates from Qwen3-Embedding-8B; checkpoint lwhlwh/reasonrank-8B; ranking format and prompts in Appendix C.3; evaluated group: six-domain arithmetic mean; raw nDCG@10; original GPT-5.4 pooled relevance judgments; source: Table 8	2026-10-30 · document_publication	ProSearch paper v1 · retrieval, query formulation and reranking results

What must be retrieved?

Queries, domains and evidence

How to reproduce the evaluation

Read the score with its setting

All 19 first-stage retrievers

Query formulations with one retriever

What changes after reranking?

Changing the candidate corpus

Open-world search on 30 queries

Structured information diagnostic

Constraint-role subsets

What this does not establish

Version and model identity

Original publication

Evidence available for this benchmark

Start with these resources

Evaluation context and evidence

Reported results 32

Explore related benchmarks

Put this benchmark in context

Six new benchmark pages: context, agents and executable tasks

Six Navers Lab / Einsia benchmarks and the work they test

AI benchmarks and leaderboards are different things

How to compare AI benchmark scores without mixing protocols

Four new agent benchmarks: adaptation, safety and video

Continue exploring