For about a week I've been tuning the drafting side of a writing app I'm building. The machine writes a chapter, a second pass edits it, a third pass reads it cold and flags what a stranger wouldn't follow. The goal was simple to say: make the prose stop sounding like a machine wrote it.
So I measured it. Triplets per thousand words ("cold, quiet, and still"). Sentence length variation. Paragraph variation. Vocabulary spread. The first chapter came in at 4.3 triplets per thousand words when good published prose in the genre sits between 1.4 and 3.1. Its rhythm was flat. I gave the editing pass targets, ran it again, and watched the numbers walk into range. Triplets dropped to 0.8. Rhythm variation climbed from 7.7 to 10.
By every instrument I had, the chapter was fixed.
Then I tried to read it, and I couldn't get through it.
The radiator radiates
The line I kept quoting to myself was something like "the radiator radiates the way a radiator does." Not literally that sentence, but that was the texture. Every door got opened. Every stair got climbed. The bus was described like a catalogue entry. Chapter two started by re-explaining chapter one, because the cold-read pass had read chapter two on its own, decided the premise was missing, and helpfully restated it.
None of that shows up as a tic. There are no adverb piles in it, no hedges, no "I noticed that I felt." The classic machine-prose tells were already low. The prose was clean in the way a hotel room is clean. Nothing wrong, nobody home.
That was the first lesson: I had built a very good detector for the problems that are easy to count, and then I'd mistaken the count for the reading.
Four pages and a pen
My wife reads on paper. I printed the revised chapter and she went through the first four pages with a pen.
She made nineteen marks.
Run-on sentences where ", and" was joining two different people doing two different things. Words that were fancy or too literal: someone "told her cell phone" something, the bus "knelt" at the curb. Details that clouded the point, like an exact medication count, the dates on sticky notes, a riff about funding credits that went nowhere. A "he" that could have been either of two men. A line of narrator wisdom that the scene hadn't earned.
I scored the system against her marks. How many of her nineteen edits would the editing passes make on their own?
Zero.
Not a few. Zero. Every metric I had said the chapter was in range, and the system agreed with none of the things a careful reader saw in four pages.
Her notes became the test
So I stopped treating her notes as feedback to apply once and started treating them as the benchmark. Her nineteen edits went into a scoring script. Every change to the drafting chain got run against them.
That changed the work. Some of the fixes were mechanical: one editing pass trying to cut padding, fix sentences, and remove tells at the same time was unreliable, so it became two passes. Split ", and" only when the subject changes. Cut ten to twenty percent, never more.
The first new score came back at fifteen of nineteen, and for about an hour I was pleased with it.
Then I caught myself. Some of the examples I'd put in the editing instructions were her actual edits. Of course the score was high. I'd handed the test the answer key. Once I pulled those out and rebuilt the passes on the kinds of mistakes instead of the specific ones, the score climbed back to fifteen, sixteen, seventeen. Same range, but now I trusted it.
That trap is worth naming because it's so easy to fall into. The moment a human judgment becomes a number, the number starts pulling on everything around it. I wanted the score to go up, and the fastest way to make it go up was to stop measuring the thing she noticed and start measuring whether the machine could recite her.
Cutting is not the same as seeing
The other surprise ran the opposite direction. When I first told the passes to cut padding, they cut everything. A 1,388-word chapter came out at 753. The ordinary life of the characters disappeared, and what was left was a story made entirely of clues.
It's a mystery with a twist, so this was fatal. The everyday texture is what hides the clues. Strip it away and every remaining detail points at the ending. A reader who wouldn't have guessed now can't help guessing.
The machine could follow "less" perfectly. It had no idea which parts of the less mattered. That's the same blindness as the radiator, pointed the other way. Too much and too little are both decisions about what a reader needs to see, and neither one is in the word count.
What the machine actually did well
I don't want to undersell the drafting. It wrote the chapters. It took the plot, the clue budget, the character beats, and produced readable scenes in minutes. Once it knew what to look for, it applied her kinds of edits across chapters she never read. That's real.
But every useful thing it learned this week came from a person reading on paper and making a mark. The second round of her notes brought new problems: a list of chores that read like a laundry list, compressed phrasing, exposition arriving before the moment that should have triggered it. Those became a new pass and a new test. The machine got better exactly as fast as she could notice.
This week I also built a way to photograph her handwritten pages and turn the marks into notes in the app, because retyping nineteen pages of margin notes was the slowest part of the loop. That felt like the right place to spend engineering. Not on replacing the reader. On getting her pen into the system faster.
The takeaway
The metrics weren't wrong. They measured something real, and fixing it made the prose better. They just measured the part of quality that holds still long enough to be counted.
The part that matters (is this detail helping, is this "he" clear, does this sentence earn its wisdom) only shows up when someone reads the thing like a reader. The machine can draft at any speed I want. What it can't do is want to turn the page.
So the benchmark for the whole system is now nineteen marks from one person with a pen. I can't think of a better one.