An update on residential proxies and the scraper situation
356 points
• 6 days ago
• Article
Link
互联网正面临一场持续且不断升级的危机：大量爬取网站内容以为大型语言模型和其他由 AI 驱动的项目提供训练数据。这类活动已演变成一场复杂的军备竞赛，幕后黑手动用数以百万计的独立 IP 地址冲击网站基础设施。爬虫常常借助住宅和移动网络，甚至劫持普通用户设备作为指挥与控制节点。由僵尸网络驱动的流量使传统的 IP 封锁失效——攻击者很少重复使用相同来源，而且能模拟人类行为，足以绕过表面上的检测手段。
这些住宅代理网络的运营者通常可分为两类：一类是通过恶意软件入侵设备的犯罪分子，比如攻破安全性较差的媒体流设备来执行爬取；另一类则伪装成合法企业，宣称其数据来源"伦理合规"，并诱导用户安装 VPN 或软件开发工具包，将个人设备变成公司代理网络的出口。通过这种方式，运营者得以在毫无防备的用户网络中路由大量未经授权的流量，其带来的安全风险远超单纯的网页抓取。
尽管一些知名 AI 公司保留可识别的爬虫并大体遵守 robots.txt 等既定规则，但它们无法解释当前激增的流量。更可能的情况是，秘密的 AI 项目、政府机构和犯罪组织共同推动了这波争夺训练数据的竞赛。为此，网站运营者被迫采取激进的防护措施，例如要求工作量证明、设置复杂的人工验证挑战以及采用数据投毒工具。然而，这些防护对开放网络造成了沉重负担，常常损害合法用户的访问体验。
站点所有者面临的难题是：在抵御这些攻击的同时，保持对真实读者的可访问性，并避免依赖可能无意中强化搜索引擎垄断地位的允许列表。一些网站通过激进的优化和防御策略在一定程度上缓解了影响，但这类办法在动荡的环境中只是权宜之计。近期政府和企业对大型僵尸网络的打击（例如对 NetNut 网络的瓦解）虽能带来短期缓解，却无法解决恶意功能被轻易引入应用生态系统的系统性问题。
总的来看，爬取行业对开放互联网可持续性表现出令人不安的漠视，将独立网站当作可掠夺的资源，不顾由此产生的附带损害。随着内容不断被推到付费墙和防御屏障之后，网络作为开放且富有创造力的平台这一基本特性正受到威胁。在没有更广泛的共识去建立并执行用于 AI 训练数据的伦理标准之前，网站运营者将不得不继续加固自身，以抵御一个把互联网视为待征服战场的行业。
The internet is currently facing a persistent and escalating crisis driven by the massive scraping of websites to fuel large language models and other AI-driven projects. This activity has evolved into a sophisticated arms race where shadowy actors deploy millions of unique IP addresses to overwhelm site infrastructure. These scrapers frequently utilize residential and mobile networks, often hijacking ordinary consumer devices to act as command-and-control nodes. This botnet-driven traffic makes traditional IP blocking ineffective, as the attackers rarely strike from the same source twice, and they mimic human behavior well enough to bypass superficial detection.
The operators behind these residential proxy networks generally fall into two categories. One group consists of outright criminals who use malware to compromise devices, such as poorly secured media-streaming hardware, to carry out their scraping operations. The second group masquerades as legitimate businesses, offering services that claim to be ethically sourced. These companies often incentivize users to install VPNs or software development kits that turn their personal devices into entry points for the company's proxy network. In doing so, these operators gain the ability to route massive amounts of unauthorized traffic through unsuspecting users' networks, creating a significant security risk that extends far beyond simple web scraping.
While some high-profile AI firms maintain identifiable scrapers that generally follow established protocols like robots.txt, they do not account for the overwhelming surge in traffic. Instead, the current assault is likely driven by a combination of undercover AI projects, government agencies, and criminal organizations, all competing in an arms race to secure training data. This scramble has forced website operators to implement aggressive defensive measures, such as proof-of-work requirements, complex human-verification puzzles, and data-poisoning tools. These protections, however, impose a heavy tax on the open web, often degrading the user experience for legitimate human visitors.
For site owners, the challenge is to defend against these attacks while maintaining accessibility for real readers and avoiding the need to rely on allowlists that might inadvertently entrench search engine monopolies. Some sites have successfully utilized aggressive optimization and defensive strategies to mitigate the impact, but there is a clear understanding that these are temporary fixes in a volatile landscape. Recent government and corporate efforts to dismantle large botnet networks, such as the disruption of the NetNut network, provide temporary relief but do not address the systemic ease with which malicious functionality is introduced into the app ecosystem.
Ultimately, the scraping industry displays a troubling disregard for the sustainability of the open internet, treating independent websites as resources to be pillaged regardless of the collateral damage. As this behavior continues to drive content behind paywalls and defensive barriers, the fundamental character of the web as an open, creative network is at risk. Until there is a broader commitment to establishing and enforcing ethical standards for AI training data, website operators remain forced to fortify their sites against an industry that views the internet as a battleground to be conquered.
387 comments • Comments Link
- 像 Anubis 这样的 Proof-of-work (PoW) 工具正越来越被视为 captchas 的必要替代，后者通常更慢、容易分散注意力，并对真人用户有较高的误判率。
- PoW 的批评者认为其难以扩展：scraper 操作者可以利用庞大的 botnets 或原生代码，比标准浏览器更高效地完成 PoW 挑战，从而削弱延迟惩罚对顽固攻击者的威慑力。
- 虽有人建议将基于浏览器的 PoW 作为向网站"支付"内容费用的方式，但怀疑者指出，在运行于 residential proxies 或被攻陷设备上的 scrapers 不需要承担这些计算的能源或硬件成本，因此经济负担对攻击者并无意义。
- residential proxy networks 的兴起，常通过 SDK 集成到移动应用或流媒体硬件中，已把普通的 residential IP addresses 变成绕过基于 IP 的速率限制和黑名单的手段。
- 对开放、可访问互联网的渴望，与网站运营者为保护基础设施免受那些无视 robots.txt 并猛烈攻击计算密集型端点的"vibe-coded" scrapers 侵害之间，存在明显紧张关系。
- residential proxy providers 常被描述为运营着类似 botnet 的基础设施，其关键差别往往只是隐藏在晦涩 EULAs 中的一层薄弱用户同意，而非用户对设备被二次利用的明确知情。
- 一些开发者主张回归开放标准，例如改进 Common Crawl 数据或标准化 static archival formats，以满足数据需求，而无需让每个站点都承受单独、资源密集的 scraper 负担。
- Site administrators 报告称，大规模 scraping 往往并非为有目的的数据采集，而是反映出编写欠佳、带有攻击性的脚本缺乏基本优化，结果无意中造成持续性的 DDoS attack 。
- 社区在屏蔽的道德问题上存在分歧：有人认为公共网络内容本质上可被抓取，另一些人则坚持认为对服务器资源的漠视和对网站可用性的破坏，使得采取更严厉的防御措施变得合理。
- 在 Hacker News 等平台上对 single-purpose accounts 的审查，凸显出当参与者过于热衷于宣传特定反 scraping 或反 AI 议程时，维持"curious conversation"的难度。
网站运营者与激进 scrapers 之间的斗争揭示了曾经支撑开放网络的非正式社会契约正在破裂。有人认为 residential proxies 的兴起是在日益封闭、由企业主导的平台世界中获取信息的必要工具，另一些人则将其视为将个人设备武器化、针对独立出版商的手段。当前的防御措施——无论是 PoW 、激进的 IP blocking 还是 proxy-aware firewalls——往往不可避免地影响合法用户的体验。总体而言，该行业陷入一种不稳定循环：用于提取数据的工具促使防御门槛愈发侵入性，从而可能使网络进一步背离曾以无状态、可访问为荣的理想。 • Proof-of-work (PoW) tools like Anubis are increasingly seen as a necessary alternative to captchas, which are often slower, more cognitively distracting, and prone to high false-positive rates for human users.
• Critics of PoW argue that it does not scale effectively, as scraper operators can leverage massive botnets or native code to execute PoW challenges far more efficiently than standard browsers, rendering the latency penalty ineffective against determined attackers.
• While some propose browser-based PoW as a way to "pay" websites for content, skeptics point out that scrapers running on residential proxies or compromised devices do not pay the energy or hardware costs of these computations, making the financial burden irrelevant to the attacker.
• The rise of residential proxy networks, often integrated into mobile apps or streaming hardware via SDKs, has turned standard residential IP addresses into a resource used to bypass traditional IP-based rate limiting and blacklisting.
• There is a clear tension between the desire for an open, accessible internet and the necessity for site operators to defend their infrastructure against "vibe-coded" scrapers that ignore robots.txt and hammer computationally expensive endpoints.
• Residential proxy providers are frequently characterized as operating botnet-like infrastructure, with the key distinction often being the thin veneer of user consent embedded in obscure EULAs rather than explicit awareness of the device's secondary usage.
• Some developers advocate for a return to open standards, such as improved Common Crawl data or standardized static archival formats, to satisfy the demand for data without forcing every site to host individual, resource-heavy scrapers.
• Site administrators report that high-volume scraping is rarely about purposeful data gathering, but often reflects poorly written, aggressive scripts that lack basic optimization, effectively acting as an accidental, perpetual DDoS attack.
• The community is divided on the ethics of blocking; some argue that public web content is inherently open for extraction, while others maintain that the lack of respect for server resources and the disruption of site availability justifies increasingly draconian defensive measures.
• The moderation of single-purpose accounts on platforms like Hacker News highlights the difficulty of maintaining a "curious conversation" when participants become heavily invested in promoting specific anti-scraping or anti-AI agendas.
The struggle between website operators and aggressive scrapers reveals a breakdown of the informal social contract that once characterized the open web. While some view the rise of "residential proxies" as a necessary tool for accessing information in a world of increasingly gated, corporate-controlled platforms, others see it as the weaponization of personal devices against independent publishers. Current defensive measures—whether through PoW, aggressive IP blocking, or proxy-aware firewalls—frequently end up collateralizing the experience of legitimate users. Ultimately, the industry remains in a precarious cycle where the tools used to extract data necessitate ever-more intrusive barriers, potentially pushing the web further away from the stateless, accessible ideal that previously defined its success.