Autoresearch for Codebases: Turning a Goal into Measured Experiments

Autoresearch gives an agent a small goal and a fixed test. The agent tries a change, reads one score, and keeps only a valid gain. A person still owns the goal, limits, and final choice.

The original project is narrow on purpose

Karpathy's autoresearch project gives an agent one file to change. Each training run lasts five minutes. The agent reads one score after each run and keeps or rejects the change. A fixed time and score make the runs easier to compare.

The score is validation bits per byte, or val_bpb. Lower is better. The project tests a small language model on one graphics card. It does not prove that the same loop works for every codebase.

A codebase version needs a stable score

A third-party Codex Autoresearch project adapts the loop for software work. Its examples include test time, build time, bundle size, memory use, and page speed. This is a separate project, not part of Karpathy's first test.

Choose a score tied to the business problem. It might be the time needed to finish a report. It could be page load time on a target phone. Query delay and failed browser steps can also work.

The score must not hide damage. A smaller JavaScript bundle has no value when the contact form breaks. Regression tests protect the parts that must not change.

Define the experiment before the agent starts

Write the rules before the first run:

Record the current state and its score.
State the desired result and target score.
List the files or systems the agent may change.
List the tests that must keep passing.
Set the time or cost for each try.
Write the keep-or-reject rule.
Name the changes that need human approval.

OpenAI's long-running work guide calls for a clear result, limits, and checks. Those rules also make a code trial easier to review.

Browser and computer tools can test the real task

Unit tests can miss a broken user task. A browser tool can use links, menus, fields, and forms. Computer Use can check a desktop app when no direct tool exists.

OpenAI's Computer Use guide says to prefer a direct tool when one exists. Keep the task narrow. Review every permission. A graphical user interface, or GUI, tool can change real data. Keep private apps closed. Require a person for account, payment, security, or login changes.

This rebuild uses a small code experiment

The old GLCO homepage used Pixi and WebGL for a ripple effect. The new goal is a faster static hero. It must keep the Great Lakes visual identity.

The agent may change the hero, site styles, and package list. The site must still build. The Pixi client bundle must be gone. The main image needs a known width and height. Keyboard use and the full contact task must still work.

The agent may try more than one fix. GLCO keeps a change only when every required check passes. This is one tested code loop. It is not a claim about every site or agent.

A skilled operator remains the force multiplier

Modern models can test many ideas. They may find a result that beats a person's first try. The result is more useful when the person in charge sets the score, supplies the tools, spots a false win, and stops unsafe work.

The roles stay clear. The operator defines value and risk. The agent searches within the limits. Tests reject damage. A person approves the result.

See Agentic Software Improvement and the Codebase Improvement Loop for the GLCO offer and implementation record.

AI assisted with research, implementation, and drafting. GLCO checked the cited project scope, wrote the acceptance tests, and reviewed the final claims.

Sources

Next action

Record the current score for one costly task. Then list the allowed edits, time limit, tests, and keep-or-reject rule before an agent runs the first trial.

Test the method on one business process.