What it costs to work in the open

We published the Remora protocol before we had run it, and then published every run against it, including the four that told us nothing. Six weeks in, the method is not the one we started with. Some of that is the work; some of it is the audience. This is an attempt to separate the two, while we can still remember which was which.

The short version: working in the open made the protocol better and the schedule worse, and neither effect came from where we expected.

The commitment

The rule we set was narrow enough to be checkable. Before a run starts, the protocol that describes it is public — the hypothesis, the metric, the stopping condition, and what would count as the result being negative. After the run ends, the numbers go up whether or not they support the protocol’s guess. Nothing is held back for a write-up.

We did not commit to publishing continuously, and it matters that we did not. The log records state changes and results; it does not narrate. A lab that streams its thinking is doing something else, and something harder.

What it changed about the method

Protocols became falsifiable

Writing the stopping condition down in front of strangers is a different act from writing it down for yourself. Our first draft of the Remora protocol said we would measure “routing quality”. That survived exactly as long as it took to imagine someone reading it and asking what number would make us stop.

The rewrite named two: the fraction of sub-tasks routed to a large model, and end-to-end task accuracy against a single-large-model baseline. Both were already implied by the work. Neither was written down, and one of them we had been quietly computing three different ways depending on which notebook was open.

We would like to claim this as a discipline we could have imposed on ourselves. We had six months of private notebooks saying otherwise.

The negative results stopped disappearing

Four of the first eleven Remora runs produced nothing usable. In a private project, three of those four would have been a bad afternoon and a deleted branch. Published, they are four rows in the log, and two of them turned out to share a cause — a context-window truncation in the router that only bit on long inputs, which we found because the failures were sitting next to each other in a list rather than scattered across six weeks of memory.

That is the strongest argument we have for any of this, and it is not the one we expected to be making. It is not about honesty. It is that a public log is a sorted, deduplicated record of your own failures, and a private one is not a record at all.

Where it cost us

The rewrite tax

A protocol that is already published cannot be edited silently, so every change is a revision with a note attached. Four revisions in six weeks, and by our count the notes took about a day and a half of writing in total.

That is not nothing, but it is not the real cost either. The real cost is that a published protocol makes you reluctant to revise, and reluctance is not a neutral force. Twice we ran an extra experiment to avoid amending the document, which is precisely backwards: the experiment was cheaper to justify than the edit, so we bought the experiment. Both were wasted.

We do not have a fix. We have a habit now of asking, before a run, whether it exists to answer a question or to protect a document.

The audience in the room

The second cost is harder to write about without sounding ungrateful. Someone is reading, and knowing that changes what feels worth doing.

The specific failure mode is not people being wrong or unkind. It is that attention is unevenly distributed across the work, and the parts that draw it are not the parts that need it. Remora’s routing heuristic drew questions. The evaluation harness — which is where the actual uncertainty lives, and which we have rewritten twice — drew none. It took us about three weeks to notice that we were spending time roughly in proportion to interest rather than to risk.

What we would keep

Publishing the protocol first. All of it, including the stopping condition, including the definition of a negative result. It is the cheapest part of this and it carried most of the benefit.

Publishing failed runs in the same place as successful ones, in one list, in order. Not as a gesture — because the list is a tool, and it stops being one the moment it is filtered.

What we would drop

The implied promise of pace. Nothing we wrote committed us to a cadence, but a log with dates on it makes one anyway, and we felt it in weeks four and five when nothing was ready to say. Two of those four wasted experiments date from exactly that stretch.

If we run the next project this way, the log will say when the lab is thinking and not shipping, in as many words. A gap that is explained is a fact about the work. A gap that is not is a fact about the lab, and readers are right to read it that way.

@misc{nullsilver2026workingintheopen,
  author = {Nullsilver},
  title  = {What it costs to work in the open},
  year   = {2026},
  month  = jun,
  url    = {https://nullsilver.com/writing/working-in-the-open}
}