One harness for every benchmark

omegaeval plays benchmarks such as HLE, ARC-AGI-3 or FreeCiv through an OmegaClaw agent's own chat channel. It records what the agent saw, thought, remembered and did, and keeps an append-only history you can cite. It answers one question: did this change to the agent make it better, worse, or both, and in which respects.

Open the console demo Browse the registry Read the guide

See cognition step by step

The console shows every loop turn of an episode: the context sent to the model, NAL and PLN inference, memory writes, actions and errors, beside what the agent saw.

Plug in any benchmark

A question set, a memory test or a live game loads from a data file, a Python file or a package, and is named the way people say it: hle, arc-agi-3, freeciv.

Keep results you can cite

Every episode is sealed with checksums, replays exactly, and freezes into a release. A registered release is re-verified from its files before its numbers are shown.

Try it

pip install git+https://github.com/khellar/agi-benchmarking-suite
omegaeval baseline random qa-sample --seeds 1-10
omegaeval baseline oracle qa-sample --seeds 1-10
omegaeval export qa-sample --html report.html

The two reference agents need no model and no server. Connect an OmegaClaw agent when you are ready.

Register a benchmark or your results

Anyone with a GitHub account can register. Fill in a form, and a pull request is opened for you once the checks pass. Registered benchmarks load with one command, omegaeval add <id>.

Fill in the form. A benchmark needs its id, a description and the config block that loads it. Results need a release bundle made by omegaeval bundle, hosted at an https address.
The checks run. The entry is validated, and a result's bundle is downloaded and verified checksum by checksum. Problems come back as a comment on your issue.
A maintainer merges. The site rebuilds, and the entry appears in the registry with its numbers recomputed from the bundle.