SimulacraBench

NeurIPS 2026 · Competition Track

SimulacraBench Closing the Simulacra Gap in Development Data

SimulacraBench is a Situated Evals competition to validate AI representations of hard-to-reach populations, evaluated on unreleased UN microdata.

Public launch
August 17, 2026
Tracks
Cross-respondent
~40,000 respondents
15 countries
Register interest Explore Competition Platform

Starter notebook

  • English

In partnership with

Stanford HAI · UN Behavioural Science

Data providers

UNICEF · UNHCR · World Bank

About

AI is already filling data gaps for hard-to-reach populations. Nobody knows how well.

UNICEF, UNHCR, and humanitarian programs increasingly rely on rapid behavioural surveys to set policy and programming. Field data is slow, expensive, and gappy — driving interest in using LLMs as simulacra of specific populations to pre-test instruments, impute non-response, and run subgroup what-ifs.

~40,000
respondents across 15 countries
0
in the pretraining corpus
3
behavioural surveys from UNICEF, UNHCR and the World Bank

Closed evaluation

How the competition works

Evaluation is closed. Code travels to the data; only aggregate scores travel back.

  1. 01

    Submission

    A zip holding main.py and requirements.txt, implementing predict(). Code, not predictions.

  2. 02

    Execution

    Organizers install your requirements, then run predict() on the held-out respondents with the network removed.

  3. 03

    Return

    One skill score, and how long the run took. No per-item breakdown, no cell counts, nothing else.

The microdata are never released to participants.

Impact

In classical survey practice, survey participants are asked about questions, and we take means to estimate. If we have good predictions from algorithms, we can augment human labels. This allows us to achieve classical-level accuracy across the full sample at a fraction of the time and cost.

  1. Classical survey practice

    Every cell is an observed response. Estimates are sample means over collected data.

  2. Algorithmic estimates

    Every cell is model-predicted. Estimates inherit the model’s bias, and nothing in the design reveals it.

  3. Augmented

    Model predictions scale the sample efficiently, while a small human subsample removes bias to recover classical accuracy with significantly less manual effort.

Three construction workers sift and shovel sand inside an unfinished concrete building, a UNHCR-branded bag resting on a ledge behind them.
© UNHCR

A clean dataset, a proper score, the grader in your hands.Apples-to-apples.

  1. A new dataset, not benchmaxed. Three unpublished behavioural-science instruments from UN and World Bank programs, not in any pretraining corpus.

  2. Calibration-first, log-score evaluation. A proper rule that rewards truthful probabilities, not accuracy on the modal class — normalised so a uniform guess scores 0 and perfection scores 1.

  3. Apples-to-apples comparison against two published baselines — a trivial crowd marginal and TabPFN — on identical held-out respondents.

  4. Open-source starter kit. MIT-licensed: the three schemas, a sandbox generator, two baselines, a tutorial notebook, and the grader itself.

A UNICEF field worker in protective coveralls stands with arms folded in front of a UNICEF-branded tent hung with public health posters.
© UNICEF

Data

One UNICEF asset, one World Bank asset,
one UNHCR instrument.None publicly available.

The UNICEF Faith & Immunisation survey covers adults in ten countries on religious influence and vaccine uptake; the World Bank SkillCraft instrument is a single-session online skills assessment of job seekers in South Africa; and the UNHCR ERPIS instrument is a household survey of Syrian refugees in four countries of asylum. Participants do not receive the microdata. You receive (i) a schema-only specification — for each item, the question wording, its class, its allowed answers, and the earlier answer that gates it; (ii) a generator that writes practice data of the schema's exact shape, with invented marginals and invented dependencies; (iii) the grader itself, so you can run a submission locally exactly as it will be run.

Note on the specification

The three schema files in the public repository are the specification: the figures below are read off them, and a submission has to work from the schema it is handed rather than assume one instrument's shape. Every question string in the UNHCR schema is a gloss reconstructed from the variable name and the observed response labels, not verbatim instrument wording. Any change to the specification is announced to all registered teams and posted here.

Faith & Immunisation SkillCraft ERPIS 2025
Custodian UNICEFWorld BankUNHCR
Countries 1014
Waves 112
N total 19,8006,90010,300
Items in schema 19106285
Always visible (GIVEN) 7917
Scored (PREDICT) 1273213
Mean options per scored item 4.97.03.8
Uniform reference U (nats) 1.5281.7581.241

  • Faith & Immunisation

    Custodian
    UNICEF
    Countries
    10
    Waves
    1
    N total
    19,800
    Items in schema
    19
    Always visible (GIVEN)
    7
    Scored (PREDICT)
    12
    Mean options per scored item
    4.9
    Uniform reference U (nats)
    1.528

  • SkillCraft

    Custodian
    World Bank
    Countries
    1
    Waves
    1
    N total
    6,900
    Items in schema
    106
    Always visible (GIVEN)
    9
    Scored (PREDICT)
    73
    Mean options per scored item
    7.0
    Uniform reference U (nats)
    1.758

  • ERPIS 2025

    Custodian
    UNHCR
    Countries
    4
    Waves
    2
    N total
    10,300
    Items in schema
    285
    Always visible (GIVEN)
    17
    Scored (PREDICT)
    213
    Mean options per scored item
    3.8
    Uniform reference U (nats)
    1.241

Starter kit

Everything except the data.Public, MIT-licensed, today.

The repository holds the three schemas, the sandbox generator, the grader, the two baselines, and a tutorial notebook that builds the submission format up one piece at a time. It holds no microdata: the schemas describe settings, questions, and answer options only.

Starter kit on GitHub Leaderboard on Codabench

The notebook is linked from the top of this page in English; translations into the other five official UN languages are in preparation. The schemas, the grader, and the submission contract are English-only.

A UNHCR staff member in a blue vest hands a printed form to a man at the counter of a reception and registration office.
© UNHCR

Submission & rules

Submit code, not predictions.Organizers run it inside the sandbox.

How submission works
  • Submit a zip with main.py and requirements.txt at the top level, implementing predict(frame, schema).
  • Return one probability vector per blank cell, in the schema's item order, non-negative and summing to 1.
  • The harness type-checks the data, blanks the held-out cells, installs your requirements, downloads models from Hugging Face, runs predict(), and returns a single score.
  • Code can train on the visible rows of the matrix before predicting.
  • predict() must be pure: no files, no sockets, no processes, no imports outside requirements.txt. Requirements install in a build step that has the network but no data; the run has the data but no network.
Compute & quotas
  • Single H100 GPU, 16 GB. 15 minutes per instrument in the development phase; 60 minutes per instrument in the final phase.
  • Development phase (Aug 17 – Oct 30, 2026): 1 leaderboard submission per team per day, with no cap over the phase, scored on the same 30% of respondents throughout. Scores come back noised and rounded to 0.01.
  • Final phase (Nov 1 – Nov 14, 2026): 1 submission per team, scored exactly, on respondents the development phase never touched. The phases do not nest.
  • Pretrained external weights are allowed if publicly downloadable at a fixed commit hash declared before the final phase opens; any fine-tuned weights must ship inside the submission.
What ships with the starter kit
  • Two baselines. The trivial one predicts each item's smoothed crowd shares and ignores everything about the individual respondent; it is what you have to beat. The second is TabPFN.
  • A sandbox generator that writes practice data of the schema's exact shape, obeying the skip logic, with invented marginals and invented dependencies. A model tuned to it will not transfer; a pipeline debugged against it will.
  • The grader, so you can score a submission locally the way it will be scored: fresh virtual environment, dependencies installed, network cut.
  • A tutorial notebook, the three schemas, and the config that fixes the phases, time limits, and privacy parameters.
  • All of it under an MIT license. No microdata, and no real survey answer, appears anywhere in the repository.
Eligibility & ethics
  • Open to teams from academia, industry, and independent research, except where precluded by sanctions or law. An individual may appear on at most one team.
  • Winning teams supply a 4-page method description and source code under a license permitting non-commercial research reproduction. Everyone else submits code alone.
  • Microdata are not released. Attempting to access, reconstruct, or re-identify the underlying records — successfully or not — is grounds for disqualification, and so is helping anyone else try.
  • Development-phase feedback may be used for ordinary model development, but not to infer hidden labels, reconstruct the evaluation set, or exploit repeated score feedback.
  • Teams whose final scores are not distinguishable under the pre-specified paired-bootstrap test are treated as tied.

The full rules, the forum, and the leaderboard live on Codabench. All competition communication goes through team@simulacrabench.org.

Health workers in gowns and masks walk together down the central aisle of a field treatment tent lined with empty beds.
© UNICEF

Recognition

Two prizes, and a talk to the people whose data this is.

Grand prize Top of the leaderboard — one ranking, the plain mean of three skills.

Roots of the Data Prize The highest-scoring team affiliated entirely in one of the 15 countries where the data was collected.

  • UNICEF skill

    On the Faith & Immunisation instrument. 12 scored items

  • World Bank skill

    On the SkillCraft instrument. 73 scored items

  • UNHCR skill

    On the ERPIS instrument. 213 scored items

How teams are recognized

Every winning team gives a talk: once at NeurIPS, and once to the UN teams who are the custodians of these datasets. Prize money will be announced soon.

Organizing team

Organizing teamThirteen people across Stanford, UNICEF, UNHCR, the World Bank, and the UN Innovation Network.

Lead organizers

  • Andreas Haupt Stanford HAI · Digital Economy Lab

    Digital Fellow at Stanford HAI, working across Stanford's Economics and Computer Science departments. PhD from MIT; co-author of the forthcoming textbook Machine Learning from Human Preferences.

  • Mary MacLennan UN Innovation Network · Stanford HAI

    Senior Advisor on Behavioural Science to the Executive Office of the UN Secretary-General; leads the UN Behavioural Science Group. Digital Fellow at Stanford HAI. Convenes the UNICEF and UNHCR data-custodian counterparts.

Academic supervisors

  • Sanmi Koyejo Stanford CS · STAIR

    Associate Professor of Computer Science at Stanford and director of Stanford Trustworthy AI Research (STAIR). Methodological expertise on trustworthy evaluation and benchmark design.

  • Alex Pentland Stanford HAI · MIT

    Toshiba Professor Emeritus at MIT, Professor (Research) at Stanford HAI. Long-standing engagement with multilateral institutions on data governance for development and humanitarian contexts.

Data partners

  • Ahmed Galal Abukhashaba UNHCR

    Data and innovation specialist at UNHCR; supports the governance, preparation, and quality assurance of refugee and asylum-seeker microdata contributed to the benchmark.

  • Ukasha Ramli UNICEF

    Behavioural science global lead at UNICEF; data steward for the Faith & Immunisation Survey.

  • Rebeca Moreno Jiménez UNHCR

    Leads innovation data work at UNHCR over refugee and asylum-seeker microdata; owns the technical specification and ingestion pathway for UNHCR-contributed data.

  • Rifat Zahir UNHCR

    Leads regional data science innovation for UNHCR in MENA, applying data science and predictive analytics to strengthen refugee protection.

  • Samantha de Martino World Bank

    Economist at the World Bank's Mind, Behavior, and Development Unit (eMBeD).

  • Michelle Dugas World Bank

    Behavioural scientist at the World Bank's Mind, Behavior, and Development Unit (eMBeD).

Contributors

  • José Ramón Enríquez Stanford

    PhD researcher at Stanford working at the intersection of economics and machine learning; coordinates the competition design, evaluation protocol, and participant operations.

  • Yegor Denisov-Blanch Stanford

    Researcher at Stanford on the measurement and evaluation of AI systems; builds and maintains the closed-evaluation harness, submission API, and leaderboard.

  • Vani Bhatnagar UN Innovation Network

    Designer leading the visual identity and competition theme for SimulacraBench — brand, web, and participant-facing materials.

A field officer helps a group of schoolchildren in uniform wash their hands at a UNICEF handwashing station in a school yard.
© UNICEF

Behavioural Science and AI Exchange

Interested in behavioural and global applications of AI?

The Behavioural Science and AI Exchange brings together researchers, practitioners, governments, civil society, and technology organisations working on AI, behaviour, and deployment in complex settings. Previous sessions have featured speakers from Anthropic, OpenAI, the World Bank, UNICEF, and leading academic experts.

Join the Behavioural Science and AI Exchange

Public launch

Be ready for August 17.

Registration opens August 17, 2026.

Registration opens with the public launch on August 17, 2026. The starter kit is already public — clone it, run the grader on the sandbox, and register your interest to be notified when the leaderboard goes live.

Register interest