How to Detect AI Content Claims Without Fooling Yourself
Detector scores are probabilities, not proof. Here is what the research says about accuracy, and a five-step check that beats trusting one score.
By Galactic News Editorial · Published 2026-11-10T16:22:44+00:00 · Updated 2026-11-10T16:22:44+00:00
How to Detect AI Content Claims Without Fooling Yourself
A detector score is not proof that a person or a machine wrote a text. It is a probability from a statistical model. This guide explains how detectors work, what the research says about their accuracy, and a five-step check that beats trusting one score. The goal is simple: never let a single number decide what a writer did.
How detectors actually work
Most detectors measure two signals. Perplexity is how predictable the word choices are. Burstiness is how much sentence length and structure vary. Machine-written text tends to be predictable and even. Human writing tends to vary more. That pattern is real, but it describes style, not authorship.
Three things weaken the signal fast. Heavy editing by a human blurs it. Mixing machine and human writing blurs it. Paraphrasing blurs it. Short passages carry too little signal to read reliably, so detectors struggle below a couple of hundred words. A detector only sees the final version of a text. It cannot tell who typed it, whether a human revised it, or what the writer intended.
How to detect AI content: what the evidence says about accuracy
The strongest independent evidence comes from a 2023 Stanford study by Liang and colleagues, published in the journal Patterns. The team ran seven widely used detectors on 91 TOEFL essays written by non-native English speakers and 88 essays by American eighth-graders. The detectors handled the native-speaker essays almost perfectly. On the non-native essays, the average false-positive rate was 61.3 percent. Nearly one in five of those essays, 19.8 percent, was flagged by all seven detectors. Almost all of them, 97.8 percent, were flagged by at least one. The authors cautioned against using detectors in evaluative or educational settings, especially for non-native English speakers.
Vendor figures tell a different story, and they should be read as vendor figures. GPTZero reports that for results in its high-confidence category, 99.1 percent of human articles are classified as human, measured on its own test data. Its FAQ says results should not be used to punish students. Turnitin states a document-level false-positive rate under 1 percent for documents with more than 20 percent AI writing. Its sentence-level rate is around 4 percent, meaning roughly one highlighted sentence in 25 may have been human-written. Readings above zero and below 20 percent show only as an asterisk, with no highlights, because Turnitin found more false positives in that band. Turnitin's own guidance says the report should not be the sole basis for adverse action against a student.
Originality.ai shows the same gap between marketing and independent testing. The company advertises accuracy near 99 percent. Independent tests place real-world performance between 76 and 97 percent, with false-positive rates of a few percent rather than a fraction of one percent. Its scores run from 0 to 100, and a score of 85 percent AI means the model is 85 percent confident the text is machine-generated. It does not mean 85 percent of the words were written by a machine. Scans are credit-based, with one credit covering about 100 words.
GPTZero vs Originality.ai vs Turnitin compared
| Feature | GPTZero | Originality.ai | Turnitin |
|---|---|---|---|
| What it measures | Perplexity and burstiness of the text | Machine-learning classifiers over linguistic patterns | Sentence-level classification inside the Similarity Report |
| Vendor accuracy claim | 99.9 percent aggregate accuracy on its own data | About 99 percent accuracy | Under 1 percent false positives at document level above 20 percent AI writing |
| Independent evidence | Stanford study found 61.3 percent false positives on non-native essays across seven detectors | Independent tests find 76 to 97 percent accuracy, false positives of a few percent | Sentence-level false-positive rate around 4 percent; some universities switched the indicator off |
| Best use | Screening raw, unedited machine text | Publisher screening with sentence-level highlights | Institutional review alongside other evidence |
| Key limit | Weak on edited, mixed, or short text; bias against non-native writers | Strictest model flags formal human writing too | Only visible to instructors; asterisk band hides scores below 20 percent |
A five-step check that beats a single score
- Write down the claim. What exactly is being alleged, and what decision depends on it? A score matters more when a grade or a job hangs on it. Say so before you test anything.
- Run two detectors, not one. Different tools use different models and thresholds. If they disagree, that disagreement is information. Treat agreement as a weak signal, not proof.
- Test with known human text from the same writer. Run an earlier piece the writer is known to have written by hand. If that also scores as machine-written, the detector is misreading this writer's style, not catching cheating.
- Look for process evidence. Drafts, version history, notes, and search history say more about how a text was made than any score can. Ask for them before you conclude anything.
- Talk to the writer. Ask how the piece was written and what tools were used. A conversation catches honest mistakes, such as a grammar tool that rewrote whole sentences, that a score never will.
When a detector score should never decide anything
Never let a score alone decide a grade, a hiring choice, a firing, or a publishing ban. The error falls hardest on non-native English writers, whose careful, regular prose looks machine-made to these models. Short texts, formal writing, and heavily edited drafts are all high-risk for false flags. Several universities have switched detection features off entirely, citing false positives and bias concerns. That is the right instinct. A detector can start a question. It cannot answer one.
Bottom line
Detectors are screening tools, not judges. Run more than one, test them against known human writing, gather process evidence, and talk to the writer. If a score is the only evidence you have, you do not have enough evidence. The honest answer to "was this written by a machine" is usually "the score suggests it, and here is what else I checked."