Building Your Own AI Text Detector: What the Substack Demo Reveals
According to Ahead of AI, Substack has added an AI‑detector widget directly into its publishing UI. The move sparked a flurry of questions about how such detectors work and whether a small team can reproduce the idea without paying for a cloud service.
The answer is yes – you can assemble a functional detector on a laptop using a fine‑tuned language model. Below is a hands‑on look at the method, a quick comparison of the main detection families, and a realistic take on what changes when you go DIY.
How Substack’s New Detector Works (and What It Looks Like)
Substack’s feature returns a single score between 0 and 100 that indicates how likely the supplied text was generated by an LLM. The underlying algorithm is not public, but the author of the tutorial believes it resembles a pangram model: a classifier trained on short, overlapping text windows (or "chunks") that learns the subtle statistical fingerprints of AI‑generated output. In practice the model treats each chunk as an independent example, aggregates the chunk‑level probabilities, and presents a whole‑document score. The UI also highlights the chunks that contributed most to the final rating, giving writers a visual cue of which sentences look “too smooth.”
The DIY Approach: Fine‑Tuning DistilBERT
The tutorial builds a detector by fine‑tuning DistilBERT, a lightweight version of the BERT transformer that runs comfortably on a modern laptop GPU. The steps are straightforward:
- Collect a balanced corpus – equal numbers of human‑written paragraphs and AI‑generated samples (e.g., from GPT‑4). The author used the 2023 Substack post on detection methods as a reference for the data split.
- Chunk the text – break each paragraph into 50‑token windows. This mimics the pangram approach and gives the model many training points from a single document.
- Label the windows – assign
0for human,1for AI. The label reflects the training distribution, not a universal truth about any future text. - Fine‑tune – run a standard binary‑classification head on top of DistilBERT for a few epochs. The loss function is binary cross‑entropy, and the output is a probability that the window belongs to the AI class.
- Deploy locally – wrap the model in a tiny Flask API. The tutorial adds a browser UI that shows the overall score and highlights high‑scoring windows.
The result is an end‑to‑end pipeline that can be run offline, costs only electricity, and produces scores comparable to the Substack widget for the test set used by the author.
Comparison of Common Detection Techniques
| Technique | Core Idea | Typical Data Requirement | Strengths | Weaknesses |
|---|---|---|---|---|
| Supervised classifier (e.g., fine‑tuned BERT) | Learn statistical patterns that distinguish AI from human text | Labeled human and AI samples | Works out‑of‑the‑box, adaptable to new models | Needs continual retraining as LLMs evolve |
| Perturbation‑based probability test | Compare model‑assigned token probabilities before and after small text edits | Access to the generating LLM’s probability distribution | Model‑agnostic, can detect over‑confident generators | Requires query access to the original LLM, slower |
| Perplexity measure | Compute how surprised a language model is by the text | Large pre‑trained language model | Simple to implement, no labeled data | Human text can be low‑perplexity; AI can be high‑perplexity when prompted unusually |
| Watermarking | Embed hidden token patterns during generation | Control over the generation pipeline | Near‑perfect recall for watermarked text | Only works if the generator cooperates; easy to strip |
| Pangram‑style chunk classifier | Train on overlapping windows to capture local smoothness | Labeled chunks | Highlights specific problematic passages | May miss document‑wide tricks, needs chunk size tuning |
The table shows that the DIY DistilBERT detector falls into the supervised classifier bucket, but it also adopts the pangram‑style chunking to gain the granularity that Substack’s UI advertises.
What Changes When You Build Your Own
The headline change is control. Running the model locally means you decide how often to update it, which data you trust, and what privacy guarantees you need. That freedom comes with a hidden cost: the detector becomes a moving target. The tutorial warns that AI checkers are a cat‑and‑mouse game – a new LLM release may deliberately avoid the patterns the current classifier has learned, causing the score to drop dramatically until you retrain. In practice this translates to a maintenance schedule of “re‑collect data and fine‑tune every few months.”
Another trade‑off is false positives. A supervised model trained on a particular mix of prompts will flag any text that resembles that mix, even if a human author unintentionally mimics AI style (e.g., using overly formal phrasing). The Substack UI already shows occasional human‑written articles marked as high AI, a symptom of the same bias. Users who need a hard filter (spam detection, policy enforcement) should pair the score with a human review step; those who just want a writing aid can treat the score as a suggestion rather than a verdict.
Finally, the DIY route shifts cost from a subscription fee to compute time. DistilBERT inference on a CPU costs a few milliseconds per chunk, which is negligible for a typical blog post but can add up for batch processing of thousands of documents. If you plan to scale, you may need a small GPU server or a cloud function – at that point the expense begins to look similar to commercial APIs.
Try It at Your Desk Today
- Install Python 3.9+,
pip install transformers torch flask. - Grab the public GitHub repo linked in the tutorial (it contains a minimal data loader and training script).
- Run the script with a tiny sample set – you can generate AI text with the free ChatGPT web UI and copy a few of your own paragraphs for the human class.
- After training (≈10 minutes on a mid‑range laptop), start the Flask server (
python app.py). - Paste any paragraph into the browser UI; the highlighted lines will show you which parts the model thinks look AI‑generated.
- Experiment by running a grammar‑checker on the same text and observing how the score moves – this mirrors the “over‑polished” risk Substack warned about.
By following those steps you’ll have a working detector, a sense of its blind spots, and a concrete example to show teammates how easy it is to build a verification layer for any internal LLM workflow.
