Threats to the Open Web: IETF Standards and Automated Access

• Automated web access (crawling/scraping) is essential for archiving, journalism, consumer tools, and research, but faces growing threats from publishers and tech firms seeking to restrict bot access to protect advertising and licensing revenue.
• Proposed changes to IETF standards threaten the neutral, open nature of the internet by creating restrictive, potentially legally binding protocols to block or monetize access.
• EFF and allies are actively opposing these efforts, arguing that they undermine the right to access the public web for legitimate societal benefits.

Current IETF Initiatives and Risks

• AI Preferences Working Group: Aims to create "preference signals" via robots.txt, allowing publishers to formally prohibit AI training or content generation, which could be enforced legally.
• Web Bot Auth Working Group: While intended to mitigate excessive, resource-straining bot traffic, its secondary goal of cryptographically identifying bots poses a significant risk.
• Risks of Authentication: Cryptographic identification could allow websites to create "pre-approved" lists, forcing researchers, non-profits, and startups to pay for access that was previously free and open.

Implications for the Open Web

• Veto Power: Website operators could effectively gatekeep public information, hindering:
  • Non-profit archival efforts (e.g., Internet Archive).
  • Investigative reporting and watchdog activities.
  • Accessibility tools for disabled users.
  • Research aimed at government and corporate accountability.
• Financial Barriers: By enforcing paywalls on automated access, sites may exclude anyone unable to pay for licensing, fundamentally altering the democratic nature of the internet.