Is Luna on max effort the solution to coding cost?

1 - Summary

On July 30th OpenAI dropped the price of their Luna model by 80% and released compelling benchmark results indicating GPT-5.6 Luna with max effort exceeds Claude Sonnet 5 at high effort at ~1/8 of the cost (figure 1).

This has several major implications. This experiment seeks to test the recently popular hypothesis:

To answer this I ran a test to compare multiple harness setups against a task of a migration of a vanilla JavaScript front end to a React front end.

The front end is that of a live site debatebot.com.au, and includes 7 pages with a dynamic carousel based UX including accordions, skeleton loading, streaming, state management and text data entry.

15 Playwright tests were held out to test success and were not shown to the models performing the task.

Test cells configurations were a 2x2 block comparing both a vanilla Opus run vs Opus steering Luna (via Codex CLI), as well as a basic multi agent builder/reviewer & fixer orchestration plugin Adventure Party - shown below alongside results.

Results

Medians over 3 runs per cell on one fixed task (vanilla-JS → React migration), scored by a frozen 15-test Playwright suite.

There are three major findings:

1. Counterintuitively a plain Opus5 run dominated all success metrics. Whilst a medium length task, it was one which Opus5 was able to create its own playwright suite and directly build against its own test driven development plan with success without requiring external review.
2. Simply orchestrating from Opus to Luna (via Codex CLI) as a dumb builder actually increased cost. Detailed analysis showed this was due to the orchestration tax of independently verifying the diff and repairing issues.
3. The Adventure Party sub agent orchestration method (Opus5+Fable) cost materially the same as hiring out the build (Luna+Fable). Detailed analysis showed the build cost dropped to $0.20c with Luna, however the orchestration and review tax kicked in hard and Fable had significant review to complete including multiple bug fixes which fully counteracted the savings of using Luna.

This experiment did not support the usage of Luna within a builder/reviewer multi agent orchestration framework.

This is one experiment and as such is limited in its observations. Shorter single feature based sessions may show material benefits of using Luna in a builder/ reviewer framework.

Multiple technical observations were made for optimising harness setup and are covered in the technical notes section.

2 - Hypothesis and Test Setup

In order to test the hypothesis a 2x2 grid was established to test multiple harness combinations to perform a medium sized build task - the migration of a Vanilla JavaScript front end to React.

The experiment cell block

The following grid shows the 4 cells tested including a baseline Opus build, baseline Opus+Luna, and then a basic builder reviewer orchestration flow with and without Luna.

Each test was run 3 times on an automated loop with the same commencement prompt and automatic nudges if the loop paused for user feedback.

Scoring the runs

A frozen 15-test Python Playwright suite drove the built app in a real browser: lobby, validation, start, card gating, the three rounds' content and edges, the verdict, the jury's 6–6 draw, deep-link restore, browser back/forward, modal-closing, and the untouched podcast page. Alongside it, a static check confirmed whether the app was React-built.

Prompt and Nudge Approach

The following prompt was provided to kick off each run.

Migrate this app's vanilla-JS debate frontend to React.

MIGRATION_SPEC.md at the repository root is the binding statement of the task. Read it first, in full, before you touch anything. It fixes the stack, the file layout, the data-testid contract, the behaviours that must survive, and what is off-limits. Where the spec and this message disagree, the spec wins.

Do the work yourself, end to end.

How to run the app while you work:

    {App run details here}

Before you call this done, verify it yourself in a real browser: start a debate, click through all six cards, call the jury, use the browser Back button, and deep-link to /debate/<id> in a fresh tab. Do not report success on the strength of the code looking right.

Commit as you go, with real messages.

You are running fully autonomously. There is no one to answer questions, approve a plan, or choose between options — anything you ask will go unanswered and the run will simply end. Make the call yourself and keep going.

In cases where runs are paused waiting for human input the following nudge was provided. Each run was allowed a maximum of two nudges.

Continue. Make reasonable assumptions and don't ask again.

3 - Results

Here are the detailed results tables with commentary. Notes: (1) all costs shown are based on API costs in USD; (2) detailed breakdowns are rounded to the nearest cent; aggregate totals are derived from unrounded values.

Overall results by median

Analysis of cost profile between cell 3 and cell 4

Counterintuitively cell 3 and cell 4 had almost identical cost profiles. Detailed analysis shows Luna (via Codex CLI) did indeed have much lower cost however the review and fix agent required more than double the tokens in order to review and repair Luna's output due to fixing gaps and bugs. This is illustrated within the total cost per model table in this section.

Detailed per run results

Analysis of cost blowout in 2-vanilla-codex/run-2

Topline results clearly indicate the blowout is driven from turns and tool calls. Detailed analysis showed Claude had budgeted a Codex run for 10minutes in foreground mode (which doesn't need polling to confirm its still running) and the Codex run exceeded the 10minutes resulting in an error. In the next turn Claude executed Codex in background mode and polled the next Codex run every 4 seconds to check it was running - which was responsible of the cost blowout.

There is a solution for this which is expanded in the #2 within the technical notes section.

Analysis of cost blowout in 4-party-codex/run-2.

This was as the result of two unexpected orchestration strategies by the AI. (1) The guide (Opus5) allocated Fable to orchestrate the Luna session. (2) In this case the orchestrator chunked the work into 4 different packets to give to Luna vs other runs which provided a longer single execution plan.

Costs per run by model

Total cost per model

4 - Technical Notes

Technical Note #1 Setup details for calling Luna (via Codex CLI) from Claude Code

There are multiple approaches to orchestrate Codex CLI from Claude Code CLI. At a high level the two options are (1) let Claude figure it out with prompt guidance; (2) install an executable plugin such as openai/codex-plugin-cc from GitHub.

The prompt guidance approach is a little less deterministic and might occasionally need a nudge, the plugin from OpenAI is a system based approach that works very well until it can sometimes break with updates and it does have a large number of issues raised against it at time of writing. Choose your poison.

The prompt guidance approach should involve setting the model, setting effort, identifying the conversation id command, identifying the continue conversation command (in case of technical issues, simply trigger a continue), and approach to understanding when the Codex CLI agent is done.

If you'd like to take the prompt guidance approach the Adventure Party plugin has a prose based process of first probing and testing the Codex CLI, logging details, and then using it as a subagent. Feel free to use the plugin, fork or just generally copy.

Technical Note #2 In one run costs blowout occurred solely due to harness orchestration of Codex by Claude

In 2-vanilla-codex/run-2 a foreground run of the Codex session didn't require polling to remain running and in all other test runs worked well. In one test run Claude assigned a low time budget for the Codex session call and Codex exceeded that budget - resulting in the session killed and a move to using background Codex sessions and directly polling every 4 seconds racking up high token costs.

Compared to the median run for this group the cost was +53%.

This was unnecessary as the harness was already equipped with a monitor tool which monitors in the background and returns back to the model when the task is finished. This is an interesting observation as it is a "rookie error" model fault which can be fixed by prompt instructions either via skill or subagent.

Technical Note #3 Luna tiering issue with budget impact

At the time of writing the tier 1 level of direct API credits allow a limit of 200k tokens per minute (confirmed on the OpenAI user dashboard as different to the 500k publicly advertised). Breaching this limit is easy and causes rate limit errors until the window resets, which occurred for this author and killed runs which required to be restarted from scratch.

The net effect is a user must spend $50 USD to access tier 2, which grants sufficient tokens per minute to code with the model. For Luna the user will take some time to spend this much money. It is recommended to check the user dashboard for limits prior to purchases.

5 - Updates to Adventure Party Plugin

This plugin is used as a base configuration to test a typical engineer workflow using multi agent orchestration and setup of unit testing suite. Over time the tests from fresh-worktree will inform new updates into Adventure Party.

Updates added to the backlog:

(1) Additional instructions will be added to the Guide to allocate a smaller model for hireling orchestration management the actual orchestration of the Codex call is not complex however receiving the Codex session and automatically rereading context is by default an expensive task which blows out cost if being run particularly by Fable.

(2) Additional instructions will be added to the hireling sub agent to mitigate background run issues identified to optimise for more generous foreground budget, as well as implement fix for background sessions. This simple fix will significantly reduce risk of an occasional cost blowout.

6 - Future Tests

Thank you for reading this far!

My focus for this blog is to test the latest novel approaches to the outer harness.

If you have any pressing ideas, lets me know - I'll probably run them!

Also, I am currently building a better scheduling and test automation system, allowing me to run the system longer and smarter - collecting much higher samples than this run.

Instead of 3 runs per cell, 10 per cell will unlock another depth of insight.