voice_analyzer

The trouble with asking a model to write in somebody's voice is that you have no way of knowing whether it managed it, and "that sounds about right" is not a measurement, and it is not something that you can iterate against either. voice_analyzer is the missing half of that loop, which is a number, and a ranked list of what to change in order to move it.

You feed it your own writing, and it archives all of it, and it pulls about 1,400 stylometric features out of every document, and then it will score a new draft from 0 to 100 on how likely it is to have come from you. It reads plain text, and markdown, and HTML, and Word documents, and PDFs. The whole thing runs on your own machine, and it makes no network requests at all.

The number is a probability, and not a similarity

This was the part that took the longest to get right. A distance between two feature vectors is a number, but it is not an answer, because nobody knows what 0.34 is supposed to mean, and so the score has to be calibrated against 2 populations before it means anything whatsoever.

The first of them is your own documents scored leave-one-out, which is to say each one measured against a profile built from all of the others, and that is what tells you how far your own writing normally sits from your own centre. The second is a few hundred documents written by other people. A logistic fit over those two populations is what turns a distance into a probability, and it is the reason that 80 can mean "about an 80% chance that this came from that voice" rather than "0.8 of some unit that I made up".

It also means the tool can say when it doesn't know. Below about 10 documents the fit has too few genuine points to be defined at all, which is not imprecise but undefined, and so it prints the diagnostic table and refuses the score, because a number that somebody is going to quote later is worse than an error is.

The adversary is on your side, which changes the defence

Everything in the literature on gaming a metric assumes an attacker, and here the thing optimising against the score is your own agent, working on your behalf, and you want it to succeed. So the risk is not attack at all, and it is self-deception, which is that the loop reports 92 and the prose reads like a competent robot doing an impression.

Secrecy is useless against that, and it is close to useless anyway, since anything holding the binary can recover the weights by probing it. So the tool discloses everything, and defends structurally instead:

• Randomised feature subsets on every scoring round, so that there is no fixed target to overfit to
• A verify pass that rescores with a fresh seed and a different comparison set, because a match that survives resampling is real, and one that collapses was fitted to one particular sampling of the metric
• A verbatim cap, since copying somebody's sentences moves every metric at once, and the threshold for it is derived from how much you repeat yourself across your own documents rather than from a constant
• Symmetric penalties, so that being far above somebody's em-dash rate is punished exactly as much as being far below it, which is what makes phrase-stuffing backfire

Measurement kept overruling me

I validated it against 6 public-domain authors, each one trained on one book and tested on a different one, so that topic and character names could not leak into the result. Two things came out of that which I had got wrong.

The reference corpus that everything is scored against matters enormously, and swapping a mismatched one for a well-matched one was worth +0.13 AUC, which was a larger effect than anything else in the whole tool. And a profile built from a single register does not recognise its own author writing in another one — Wilde trained on De Profundis scored Dorian Gray at barely above chance, while every other author scored between 0.85 and 1.00.

I had already decided, on reasoning, that per-author feature weighting wasn't worth the complexity. The measurement said otherwise, and so the decision got reversed, and the plan file records both the original reasoning and the number that overturned it.

Rust, for one specific reason

It is a single binary with an embedded part-of-speech tagger, and no runtime, and no interpreter. But the reason for Rust is narrower than that, which is that the calibration has to run on every single ingest, and that means N profile rebuilds and several hundred million float operations each time, and if that took minutes then calibration would become a batch step that somebody has to remember, and the scores would quietly drift out of date as the corpus grew. It takes well under a second, and so it just happens, and the score is never stale.