Physical-Engineering’s Last Exam

Physical-Engineering’s Last Exam (PLE) is an 85-task benchmark of real-world engineering work across structural, civil, electrical, and mechanical engineering. Each task runs in an isolated container and asks for a project deliverable—a calculation package, design review, drawing, or technical memo—graded from 0 to 1 by automated checks, an LLM rubric judge, or both.

In the PLE competition, you submit a GitHub commit containing your harness (the server that drives a model through each task), the Google-sponsored model it calls, and the artifacts/ it produced on the 29 public dev tasks (dataset). For each submission, we:

1. Grade your committed artifacts/ to compute your dev score.
2. Run your harness live on a sample of dev tasks to verify that your dev score reproduces.
3. Run your harness three times on the 56 hidden test tasks; the mean of the three runs is your test score, and the leaderboard ranks participants by their best test score (rules and scoring).

Quick start

You need Docker and uv.

The dev tasks are in a gated Hugging Face dataset: accept its terms on the dataset page, then create a read token.

# 1. Install Harbor (runs the dev tasks; the kit's agent needs websockets)
uv tool install harbor==0.20.0 --with websockets
export HF_TOKEN=<your Hugging Face read token>

# 2. Copy a starter entry into its own Git repository outside this clone
cp -r sample_submissions/pi-opus ../my-harness
git -C ../my-harness init -q

# 3. Start the local proxy for the google-cloud models (billed to your Google Cloud project)
gcloud auth application-default login
export VERTEXAI_PROJECT=<your-gcp-project> MODEL_PROXY_KEY=$(openssl rand -hex 24)
kit/model_proxy/start.sh &
export MODEL_PROXY_URL=http://localhost:4000

# 4. Build, start, test the model, and run the smoke and MCP checks
uv run python -m kit check ../my-harness

# 5. Run and score the dev set; writes ../my-harness/artifacts/
uv run python eval/run.py ../my-harness

See kit/README.md for local checks and container limits, and eval/README.md for dev-set evaluation options and the full list of dev tasks.

Before you submit

1. Commit your harness and artifacts/:

   git -C ../my-harness add -A && git -C ../my-harness commit -m "harness and dev artifacts"

2. Run the readiness check and confirm it finishes with ready to submit (see docs/artifacts.md for artifact validation and offline regrading with eval/run.py --artifacts):

   uv run python -m kit check ../my-harness

3. Push and submit: Push the commit to a GitHub repository owned by your personal account (public, or private with the competition's GitHub App installed), register, and submit on the competition site.

How an entry runs

Your harness runs as a WebSocket server in its own container alongside an isolated task container:

 your server (serve.py)  <---- websocket ---->  evaluator  ---- runs your requests ---->  task container
         |                                                                                (project files,
         +---- egress proxy ----> your model's API, and no other host                      no network)

For each task (up to 6 hours), the evaluator sends the task prompt over the WebSocket, and your harness interacts with the task container via a small JSON protocol: run a shell command, read or write a file, or call one of the task's local MCP servers (documents, spreadsheets, email, CAD). See kit/README.md for the container contract and docs/protocol.md for the WebSocket protocol.

Models

Set model_api in submission.yaml to any of the eight Google-sponsored google-cloud/<id> models (free on the evaluation server, which supplies the key, so the competition takes no API keys; run locally via kit/model_proxy). For local runs only, the kit and eval/run.py can also run your harness on a commercial model with your own key or on a model you serve, with the key in your environment or --env-file; such an entry is not accepted for submission. See docs/models.md for the models and runtime environment variables.

Sample submissions

sample_submissions/ includes eight working starter entries, all on Google-sponsored models except local-model, which is for local runs:

• Compact custom starters: minimal, plan-verify, and node-protocol (Node.js)
• Full-featured custom harnesses: custom-openai and local-model (the same harness on a model you serve, local runs only)
• Adapters for existing agents: hermes-opus (Hermes Agent), pi-opus (Pi), and agy-gemini (Antigravity CLI)

The benchmark

Discipline	Tasks	Typical deliverables
Structural engineering	10	Gravity, wind, and seismic load checks, member and connection checks, seismic design bases, drawing revisions from review comments
Civil engineering	18	Settlement and bearing capacity checks, foundation strategy, site layout, grading and drainage, construction sequencing and temporary works, cost estimates and fee positions
Electrical engineering	27	Load and capacity calculations, plan-review responses, energization checks, cable sizing, specifications, drawing records
Mechanical engineering	30	Cooling and controls, clash-free MEP routing, equipment equivalency, storage-tank service life, fire-test validation

Citation

If you use PLE in your research, please cite the paper:

@misc{han2026ple,
  title         = {{PLE}: Physical-Engineering's Last Exam},
  author        = {Rujun Han and Yufan Zhuang and Arijit Ray and Han Yu and
                   Zhongying CuiZhu and George Lee and Yifei Ming and Binz Roy and
                   Varun Sakalkar and Vishy Tirumalashetty and Burak Gokturk and
                   Tomas Pfister and Chen-Yu Lee},
  year          = {2026},
  eprint        = {2610.30537},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2610.30537}
}

License

Apache 2.0. See LICENSE and CONTRIBUTING.md.

Disclaimer

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.