Human-Agent-Society/reef

What Reef is and the problem it targets

Reef is an open-source project from the Human-Agent-Society group, described in one line as infrastructure for continually self-improving agents. According to its README, it is the first open-source infrastructure for this kind of continual learning. The central idea is to connect four things that usually live apart: agent inference, feedback, learning, and versioned delivery of the updates that learning produces.

The problem will sound familiar to anyone who has run an agent in production. A live agent generates a great many interactions, yet most of them sit in logs and never make the agent better. To improve it, a team normally builds its own pipeline: collect data, match feedback to individual responses, run training or prompt edits, evaluate, then redeploy. Reef packages that whole chain into one system with explicit versions.

The README says Reef can be used in two directions. The first is training model weights with Slime and SGLang. The second is improving the agent's harness, meaning its prompts, rules and skills. The two differ a lot in infrastructure cost, and a later section explains why that matters when you consider adopting it.

Basic facts about the repository

According to the GitHub metadata collected for this edition, the repository has 7639 stars and is licensed under Apache-2.0. Its official description is infrastructure for continually self-improving agents. The attached topics are agent-infrastructure, ai-agents, continual-learning, inference, llm, llm-training, reinforcement-learning and self-improving-agents, which places the project at the crossing of agent operations and reinforcement-learning-based training.

A high star count says nothing about technical quality, but it does show the community is paying attention. The README also lists a Discord server, a WeChat group, GitHub Discussions, a contribution guide, an RFC process for proposing designs, and a security policy for reporting vulnerabilities privately. For a young infrastructure project, having these channels in place is a good sign of how open the authors are.

When to use Reef

The README gives a table that maps goals to learning paths. If you want a model that keeps getting stronger and is designed for you, the path is model weight training, and you need a trainable model, a supported GPU stack and feedback that your recipe can use. If you want the agent's harness to improve itself, the path is harness optimization, and you need a model endpoint, representative tasks and an evaluator, with no local training GPUs.

The third path in the table is scientific discovery through test-time training. It needs an execution environment, a correctness checker and a measurable objective. In short, Reef asks users a first question: what should your agent learn, its weights, its harness, or the solution to one hard problem? Only after that does configuration begin.

For people building agents at small or medium scale, the harness optimization path looks like the more practical entry point because it does not require GPU infrastructure. The README states plainly that this path only needs a model API. That is the author of this article reading the requirements table, not an official recommendation from the maintainers.

Where Reef sits in an existing stack

The README includes a table comparing Reef with two familiar groups of tools. The first group is inference engines such as vLLM and SGLang. The second is reinforcement-learning training frameworks such as Slime, veRL and AReaL. Inference engines serve live traffic but do not train weights. Training frameworks do the opposite. According to the table, only Reef serves live traffic, trains weights, manages versions, stays live through updates, and evolves beyond weights into skills and harnesses, all at once.

Read the table for what it is: the authors positioning their own product, not an independent comparison. Even so, it clarifies the role. Reef does not replace an inference engine or a training framework. The README also thanks SGLang for high-performance inference, slime for model weight training and cordis for harness evolution, which means Reef relies on those projects for the specialised parts.

How it works: a four-step learning loop

Each learning cycle in Reef has four steps. The first is Serve: handle agent requests and record the interactions. The second is Observe: match feedback to the recorded interactions. The third is Grow: produce an update from the eligible records. The fourth is Commit: apply the configured selection policy and publish the updates that are accepted.

The README ties each step to source directories. Serving lives in service for agent requests and interaction records, and in runtime for inference and artifact updates. Observing lives in storage for stored interactions and feedback records, and in record2record for matching feedback and checking eligibility. Growing lives in recipe for recipe integration, and in train for the training pipeline, batches and update jobs. Committing lives in evaluation for candidate evaluation, artifact for version history, and surface for artifact delivery.

One detail deserves attention: the Commit step has a selection policy. An update is not put into use merely because it was produced; it first has to pass candidate evaluation. In any self-improving system this is where you learn whether the agent truly progressed or merely changed, so readers should look closely at the evaluation part when judging the project.

Installation and first run

The README explains installation from PyPI or from source and recommends uv for package management. The source checkout is intended for development and for the training examples. One important note: Reef's artifact and checkpoint features require the git-lfs system package, and Reef initialises Git LFS locally for its artifact repositories.

The smallest example is to start Reef as a pure inference server. From there, Reef's inference endpoint is compatible with both OpenAI and Anthropic, taking the provider's own request body at the chat completions, responses and messages paths. Every request carries a header naming the scenario; a new name creates a new scenario using the recipe configured for the deployment. Requests cannot select recipes.

Responses keep the OpenAI-compatible format, and Reef adds a header carrying a receipt for the interaction. A later report uses that receipt to say which interaction it is judging. A report can contain a numeric score, textual or structured feedback, and the list of receipts being evaluated. The endpoint validates reports against a dedicated schema.

Two deployment styles: weight training and harness evolution

For weight training, the README uses the SAO example deployment, tied to a paper with the arXiv identifier 2607.07508. The example must be run from a Reef checkout in an environment that meets the GPU requirements. You send inference requests through Reef and report a score for each response; the SAO recipe uses every eligible scored rollout to run a training step. Once enough feedback has arrived, the recipe runs the step and synchronises the updated weights to the serving runtime. Later requests use the current version without restarting Reef.

For harness evolution, the README describes refining a coding harness from plain-language requests, using a model API instead of GPUs. Reefine is the built-in harness-refinement recipe and ships with a deployment configuration; you specify the provider address and the model. By default Reef listens on port 8901 on the local machine without authentication, unless you set the REEF_TOKEN variable before starting it to require a token. State is kept under the .reef/reefine directory.

In a second terminal you create a scenario, install the harness and ask for a change. Inside a reef-pi session, the /reefine command files the same kind of request. The served model writes the change as a skill, a rules entry, an agent command or a pi extension. You review versions with /versions and install one with /versions install, so there is a human review step before a changed harness is used.

Safety notes when agents may rewrite their own harness

The README spends a good number of words on risk, and this is the part readers should study. The install root must sit outside the project the agent works in, because a session can write files in its project; if the install root were inside it, the agent could change what the next session runs. The same applies to the Python environment baked into reef-pi and to the Reef checkout it imports, because code there runs as you when reef-pi starts.

For codex and dsh, the README advises keeping those locations out of the system temporary directories too, since their sandboxes let a command write there. The pi tool runs commands with no sandbox. The change proposer only works as a coding agent that runs the changed harness before handing it back where the host can isolate it, specifically Linux with bwrap and pasta, as a non-root user. Otherwise you can set REEF_PROPOSER_SANDBOX to none on a machine you trust.

The practical lesson is that a system letting an agent write its own skills and rules is a new attack surface and a new source of bugs. Treat sandbox configuration and version review as a mandatory part of the design, not an option.

Recipes and examples

The README organises recipes by task type and by what should evolve, weights or harness. Weight recipes need the GPU training stack; harness recipes need only a model endpoint. Reefine ships with the reef-infra package, while the other implementations live in the recipes cookbook of the repository, are chosen by dotted class reference, and are not shipped in the Reef wheel.

For scientific discovery, the model-evolving recipes are TTT-Discover and Guidance-TTT, and there is no harness-evolving recipe yet. The measured benchmarks are TriMul, circle packing and Erdős minimum overlap. For continual learning on a task stream, the model recipe is SAO and the harness recipes are Meta-Harness and GEPA, with the benchmarks AIME 2025, IMOAnswerBench, CEO-Bench and Terminal-Bench. For learning from usage, the model recipe is OpenClaw-RL and the harness recipes are SkillClaw and Reefine, with a simulated student on a GSM8K task stream and WildClawBench as benchmarks.

The README says each results page documents its task, evaluation setup, measurements and limitations. This article does not repeat benchmark figures, because the README only lists benchmark names and links to separate results pages. The recipes/basic directory is the record-only starting stack and sits outside the catalog.

Using Reef in an agent-building workflow

Picture a team running an internal coding agent. They could place Reef in front of the model as an OpenAI-compatible proxy so that every request passes through it and is recorded. When a user corrects the agent's output or a test passes or fails, that signal is sent back as a feedback report tied to a receipt. On that basis a harness-refinement recipe proposes a new skill or rule, and the person in charge reviews the version before installing it. This is an illustrative scenario inferred from the components the README describes, not a workflow published by the authors.

The project's documentation is arranged in order: a Quickstart to install Reef, connect a client and inspect version history; the HTTP API for reporting feedback; a guide to writing recipes; a guide to evolving a harness; a guide to evolving a model; the recipe catalog; a description of the core loop; and a glossary. If you are new, following that order avoids jumping straight into GPU training.

Limits and what the README does not say

The README gives no figures for the latency added by placing Reef in front of a model, for operating cost, or for stability in production. It also says nothing about handling sensitive data inside interaction records, even though the system stores interactions and feedback. If your agent handles customer data, that is a question you must answer yourself before turning recording on.

According to the metadata the project is young, and the README has a roadmap entry whose details live on another page. Self-improvement also depends heavily on the quality of the feedback signal and of the evaluator: if scores are wrong or sparse, the updates produced may not be better. The README stresses representative tasks and an evaluator for the harness optimization path, which suggests the authors treat that as a prerequisite too.

Conclusion: who should try it and who should wait

Reef suits teams that already run an agent for real, have a reliable way to collect feedback, and want to turn a manual improvement loop into a versioned process. If you only have a few fixed tasks and no scoring signal, the cost of setting up this infrastructure may exceed the benefit. If you want to train weights, prepare GPUs and read the model-evolution guide carefully.

The most cautious way to try it, following the README, is to run the harness evolution path on an isolated machine, keep the install root outside the project, enable the sandbox where possible, and review every version by hand. That is the cheapest way to watch a Serve, Observe, Grow, Commit cycle run for real without investing in training infrastructure.