Are better models replacing Superpowers?

1 - Summary

This fortnight’s outer harness evaluation rests on Superpowers - perhaps the most popular AI skills framework in the world.

Superpowers is well known for being a comprehensive spec/test driven AI multi agent development framework that just works.

Advancements in recently model evaluations and large jumps in benchmark scores have led some to question Superpowers.

Some say, “surely the major labs are encoding the basic tenets of Superpowers directly into the model training data?”

The following major model benchmark would certainly indicate so.

One contact of the author was recently granted access to a frontier model and stopped using Superpowers, only to later lose access to that frontier model and lament having to "go back to superpowers".

The evaluation will occur across two parts:

Part 1) Greenfield build of a gym booking application (this article), based on a plain English specification.

Part 2) Brownfield fix and migration of the same gym booking application, after a migration was abandoned mid way through by a poor developer.

For part 1, the evaluation compared three different cells; a plain Opus5 agent, a basic orchestration plugin called Adventure Party (Opus5 as builder, Fable as reviewer/fixer), and the latest Superpowers plugin.

Instructions were provided in the form of a written specification document (refer to section 2-Hypothesis and Test Setup), and a 66 test suite setup.

Results

Table of results by cell

There are several major findings;

1) Based on raw cost to deliver, the plain Opus5 setup outperformed both Superpowers and Adventure Party.

2) The Opus5 plain run never wrote a single unit test, resulting in the least established run for future build enhancements. In real life, future upgrades would likely result in multiple unnoticed bugs. Additional instruction to simply write unit tests may have solved this and will be tested in the future - we are yet to arrive on completely “trusting the model”. Both Adventure Party and Superpowers setup a comprehensive test suite.

3) Log analysis showed that all failures except one centred on the admin screen. Whilst a small sample this will be followed in future evaluations - admin screens follow a very different function to other app features and may benefit from special treatment in the outer harness.

4) In the plain and Superpowers runs direct front end Playwright tests were used compared to http auditing in the Adventure Party run. Playwright tests were significantly more effective at catching bugs and did so early, whereas the party run was saved by the Fable model review and fix agent at an expensive cost. This will be added to the Adventure Party backlog.

Bonus: Of some novel interest regarding the recent hack of a gym website in the Australian news - none of the runs featured any obvious “vibe coded” security gaps as appeared to be the case in the news event.

I’ll note that for some time, sub agent driven development (of which Superpowers is a leading framework) has long been agreed as best practice to combat context bloat and improve performance. I have also found this to be true in the past.

Whilst I’ll hold my opinion for further compounding evidence, these results strongly suggest models are now becoming much better at managing context within single or small number agent runs, which may mean that frameworks in the future should use significantly less sub agents.

Next edition - we’ll run the same test cells on a complicated brownfield project.

2 - Hypothesis and Test Setup

In order to test the hypothesis three different cells were established to test multiple harness combinations to perform a medium sized build task - the greenfield build of a gym booking app.

The experiment cell block

The following grid shows the 3 cells tested including a baseline Opus build, Superpowers with Opus, and then a basic builder reviewer orchestration flow using the Adventure Party plugin.

Each test was run 5 times on an automated loop with the same commencement prompt and automatic nudges if the loop paused for user feedback.

Scoring the runs

A frozen 66-test suite matching the build spec driven from the live app:

Prompt and Nudge Approach

The following prompt was provided to kick off each run.

TASK: Build the IronForge Gym class-booking web app from scratch.

TASK_SPEC.md is the binding spec and outranks everything else: user roles, class types, seed data, exact UI labels, API requirements, business rules, run contract.
    
GIT Clean branch. Commit as you go, with real messages.

RUN Stack is the agent's choice; the contract is not:
bash setup.sh
PORT= bash run.sh
App live on localhost:, all spec'd accounts, classes,
bookings and leave requests self-seeded.

VERIFY Reading the diff is not verification — run the app and
exercise it by hand (every role, book/cancel, admin) before
claiming done.

AUTONOMY Questions go unanswered. Make the most reasonable reading of
the spec, record assumptions in commits, stop when verified.

PER-CELL ORCHESTRATION SECTION — the only thing that varies 

Plain Opus5 none; core.md alone.

Adventure Party "The party is hereby explicitly summoned" — muster;
fighter builds, cleric always reviews; the summons changes how the work is orchestrated, never what.

Superpowers "Follow your superpowers skills workflow" + "I am asking
you to dispatch subagents"

The task specification includes:

• Three-role auth — member, instructor, admin log in through one email/password page; each role strictly supersets the last
• Weekly class schedule — Mon–Sun calendar of Weightlifting / Yoga / Cardio classes across seven fixed time slots, browsable for the current week and one week ahead only
• Booking engine — book and cancel classes with rules enforced: 30-person capacity, no double-booking, no booking past weeks, cancelling frees the spot
• Instructor accreditation — five seeded instructors; a class can only be allocated to an instructor accredited for its type
• Annual leave workflow — instructors request leave; admin approves or denies; approved leave blocks allocation on that day, pending leave doesn't
• Admin dashboard — add/delete classes (with accreditation and leave checks), manage leave requests, view all bookings
• Attendance — instructors take a Present/Absent roster for their own classes only (admin excepted), re-saveable, exposed via the API
• Email outbox — no real email; reminder (class start − 8 h), birthday, and missed-class emails are queued as inspectable records, with cancellation and no-duplicate rules 
• Pinned REST API — 15 exact routes (`/api/auth/login`, `/api/bookings`, `/api/classes`, `/api/leave`, attendance, admin outbox…) with uniform 401/403/404/409 semantics
• Exact UI labels — every nav link, heading, button, and "X / 30 spots" string pinned verbatim so the frozen test suite can find them
• Self-seeding — 9 named accounts, 20 scheduled classes (5 full at 30/30, 15 partially booked), one approved and one pending leave request, dates of birth — all created on first start
• Run contract — `setup.sh` installs everything; `run.sh` serves in the foreground on `$PORT`; no working scripts, no marks

In cases where runs are paused waiting for human input the following nudge was provided. Each run was allowed a maximum of two nudges.

Continue. Make reasonable assumptions and don't ask again.

3 - Results

Here are the detailed results tables with commentary. Notes: (1) all costs shown are based on API costs in USD; (2) detailed breakdowns are rounded to the nearest cent; aggregate totals are derived from unrounded values.

Overall results by median

Detailed per run results

Analysis of cost blowout of Superpowers run 2

In this run Superpowers hit a bug in the build and in this run only invoked the debugging skill which resulted in both higher turns and tool calls and causing context and cost blowout for the orchestrator agent. Given the debugging was successful and this run passed every test this is arguably a positive outcome and feature for Superpowers.

Cost per run by model

Total cost per cell by model

4 - Technical notes

Unit tests setup by Opus5 plain run

Notably the plain Opus5 run did not write a single unit test in any run, leaving a problematic code base to build on later, to say the least. Future tests will include a single line prompting Opus5 to including a test suite, albeit it is surprising at this point this is not coded into the model.

Test failures largely centred on the administrative screen

Outlined in the following table are all the test failures. The majority centred on the admin screen. Whilst a small sample this is a potential item to note and compare across other test exercises. The admin screen occurs later in the build and this may be a context degradation issue or an area which requires a specific sub-agent spawn as administrative screens serve a very different function in applications.

Usage of front end inspection using Playwright vs HTTP tool

Analysis of logs showed that a significant driver of cost for the Adventure Party runs was the builder agent using the HTTP tool to self QA the front end. This resulted in costly fixes later required by the review/fixer Fable agent at higher billing.

Similar initial bugs were caught by the plain and Superpowers runs early in the test using Playwright instead of HTTP.

5 - Updates to the Adventure Party Plugin

This plugin is used as a base configuration to test a typical engineer workflow using multi agent orchestration and setup of unit testing suite. Over time the tests from fresh-worktree will inform new updates into Adventure Party.

Updates added to the backlog:

(1) Additional instructions will be added to the agents to prefer front-end testing via Playwright as opposed to HTTP. Only the Adventure Party in this test used HTTP due to an ambiguous agent instruction “always use your eyes to check work”. This was noted in all Adventure Party runs, and while they were later solved by the Fable reviewer agent there were unnecessary costs incurred in doing so.

6 - Future Tests

Thank you for reading this far!

My focus for this blog is to test the latest novel approaches to the outer harness.

If you have any pressing ideas, lets me know - I'll probably run them!

In this run I implemented updates to the test engine to allow both longer and a larger number of runs to be completed - which were put to full test with the long Superpowers runs.

Next run we will again test Superpowers however on a much more difficult brownfield suite. The rigour around Superpowers may tell a very different story next time!