Detecting LLM-Generated Texts with "Classical" Machine Learning
现代大型语言模型表现出明显的统计特征，使得用传统机器学习方法可以有效地区分其生成的文本与人类创作的文本。尽管许多在线 AI 检测服务宣称高准确率，但常被营销噪音或诸如文本困惑度（perplexity）之类复杂且不可靠的方法所掩盖。一种更直接且高效的做法是使用经典的分类模型，例如线性支持向量机（Linear SVM），以捕捉 AI 生成内容中特有的用词模式。
要构建有效的检测器，需要一个包含人类写作样本和经验证的 LLM 生成内容的可靠训练集。通过抓取大量 Pre-AI 时代的人类文本，并用多种主流 LLM 生成对应的文本，可以构建这样的数据集。训练流程通常是先对文本做 TF-IDF 向量化，然后用线性 SVC 进行训练。所得模型在句子级别往往能稳定达到约 85% 的准确率，为识别较长作品中的 AI 痕迹提供了坚实基础。
该项目不依赖资源密集的云 API 或庞大服务器，而采用适合浏览器运行的轻量实现：将训练好的模型导出为与 JavaScript 兼容的格式，在客户端即时执行，从而实现无服务器、注重隐私的工作流。最终系统通过七个不同二元分类器的多数投票机制来标记可疑片段，能够以高置信度检测出即便占比很小的 AI 生成内容，同时保持可忽略的误报率。
即便尝试通过机器翻译往返或特定的"反 AI"提示来规避检测，那些潜在的统计标记仍然相当顽固。这表明当前一代 LLM 在语言生成上依赖于与人类创作过程根本不同的可预测模式。尽管这些模型进步迅速，本实验表明经典的机器学习方法在区分真实人类表达与大规模语言模型生成的模式方面仍然非常有效。
Modern large language models exhibit distinct statistical patterns that allow them to be effectively distinguished from human-written text using traditional machine learning techniques. While many online AI detection services promise high accuracy, they are often obscured by marketing noise or complex, unreliable methods like measuring text perplexity. A more straightforward and effective approach involves using classic classification models, such as Linear Support Vector Machines (SVM), which can capture the specific word-choice patterns inherent in AI-generated output.
To build an effective detector, the process requires a robust training set containing both human-written articles and verified LLM-generated content. By scraping thousands of human-written texts from the pre-AI era and generating equivalent content using a variety of prominent LLMs, it is possible to create a reliable dataset. Training these binary classifiers involves processing text through TF-IDF vectorization followed by a Linear SVC. The resulting models consistently achieve approximately 85% accuracy at the sentence level, providing a strong foundation for identifying AI influence within longer works.
Rather than relying on resource-intensive cloud APIs or bulky servers, this project utilizes a lightweight implementation suitable for web browsers. By exporting the trained models to a JavaScript-compatible format, the detection logic runs instantly on the client side, maintaining a serverless and privacy-conscious workflow. The final system uses a majority-voting mechanism across seven different binary classifiers to flag suspicious segments, where even a small percentage of AI-generated content can be identified with high confidence and a negligible false-positive rate.
Despite attempts to bypass these detections through methods like machine translation round-trips or specific "anti-AI" prompts, the underlying statistical markers remain remarkably persistent. This suggests that the current generation of LLMs relies on predictable language patterns that are fundamentally different from human creative processes. While these models have seen rapid advancement, the experiment demonstrates that classical machine learning still holds significant power in distinguishing between authentic human expression and the patterns generated by large-scale language models.
167 comments • Comments Link
• 有人认为文本本身信息密度不足，无法可靠判定来源；也有人认为，现代检测器可以通过识别嵌入在模型训练与强化学习过程中的特定"破绽"来实现高准确率。
• 目前 AI 检测器的有效性高度依赖使用环境：在学术或法律等高风险场景中，误报率使其不宜作为唯一依据，但作为个人用来过滤在线信息流中低质量内容的工具，仍然很有价值。
• 一个重要挑战是，人类写作正越来越趋向于大语言模型所推广的那种平淡、过度结构化的风格，形成反馈循环，使得区分人类与机器产出变得愈发困难。
• 一些人主张采用"工作量证明"式的机制（例如可验证的编辑记录或带时间戳的草稿），作为在专业与学术场合确认作者为人类的更稳健、更客观的方式，而不是单靠检测器。
• 目前的检测格局更像一场军备竞赛：任何被识别出的信号（比如 em-dash 的使用或特定句式）都可能被利用来提示模型模仿更具"人类特征"的写作风格。
• 即便检测器非常准确，也会面临基准率问题：在学生抄袭等案例中，即使是极少数的误报也可能改变人生，因此必须非常谨慎并配以人工复核。
• 许多用户反映，他们凭直觉的"嗅觉测试"——识别那些被优化后索然无味的 AI 输出中的特殊、令人不适的语气——在快速筛弃低质量内容方面，往往与自动化工具效果相当。
• AI 公司目前的经济与战略激励并不倾向于"隐形"写作；它们通常更偏好面向互动优化的输出，这类产出相较于自然的人类表达，反而更容易被归类为合成内容。
• 有观点认为，这些工具的最终价值并非完全消除 AI 文本，而是帮助个人过滤掉大量自动生成的信息噪音，从而夺回时间与注意力。
• 归根结底，这场辩论凸显了数字信任的转变：验证"真实性"或"付出"这一负担，正从对内容本身的审核，转向对其创作过程与历史的验证。
总体而言，这场讨论反映出对长期检测 AI 生成文本可行性的深刻怀疑——原因在于语言本身是流动的，而且在技术军备竞赛中，生成器往往比检测器更占优势。尽管自动化检测在帮助个人过滤和管理数字噪音上有直接价值，但舆论普遍警告，不应在高风险纪律性场景中依赖这些工具，理由是误报不可避免且存在对抗性规避的潜力。对话还表明，未来的数字素养更可能从追逐"AI 签名"转向建立可验证、透明的流程，以证明人类的创作与努力。 • While some argue that text lacks sufficient information density to reliably determine provenance, others contend that modern detectors achieve high accuracy by identifying specific "tells" embedded in model training and reinforcement learning processes.
• The effectiveness of current AI detectors is often contextual; while they may be unsuitable for high-stakes academic or legal environments due to false positives, they remain highly valuable as personal tools for filtering low-effort "slop" from online feeds.
• A significant challenge is that human writing is increasingly drifting toward the bland, overly structured patterns popularized by LLMs, creating a feedback loop that makes distinguishing between human and machine output more difficult over time.
• Rather than relying solely on detection, some advocate for "proof of work" systems—such as verifiable edit histories or timestamped drafts—as a more robust, objective method for confirming human authorship in professional and academic settings.
• The current detection landscape functions as an arms race, where any identified signal (like em-dash usage or specific sentence structures) can be exploited by users to prompt models to adopt more "human-like" stylistic signatures.
• Even highly accurate detectors face a "base rate" problem where, if applied to situations like student plagiarism, even rare false positives can have life-altering consequences, necessitating significant caution and human oversight.
• Many users report that their own intuitive "smell test"—identifying the specific, insufferable tone of optimized, flavorless AI output—is often as effective as automated tools for quickly discarding low-quality content.
• The economic and strategic incentives of AI companies do not currently prioritize "stealth" writing; they often favor engagement-optimized outputs, which remain inherently easier to classify as synthetic compared to natural human expression.
• Some argue that the ultimate utility of these tools isn't the total elimination of AI text, but the ability for individuals to reclaim their time and attention by filtering out bulk-generated noise.
• Ultimately, the debate highlights a shift in digital trust, where the burden of verifying "truth" or "effort" is moving from the content itself to the process and history behind its creation.
The discussion reflects deep skepticism regarding the long-term feasibility of detecting AI-generated text, driven by the belief that language is fluid and that technological arms races favor the generator over the detector. While automated detection provides immediate value for personal filtering and managing digital noise, consensus warns against using these tools for high-stakes disciplinary actions due to the inevitability of false positives and the potential for adversarial evasion. The dialogue suggests that the future of digital literacy will likely shift away from chasing "AI signatures" and toward establishing verifiable, transparent processes that demonstrate human effort.