Building web agents at scale reveals a puzzle: some sites are dramatically easier to automate than others, in ways unrelated to their technical sophistication. A React application with the latest framework requires fragile selectors that break constantly. An older site navigates through structure that survives every redesign.

The difference is invisible to most developers. But it's critical to anyone building systems that need to work reliably across thousands of sites. The infrastructure that makes automation possible emerged from a crisis nobody could see—one that reshaped how the web communicates meaning.

Building web agents at scale reveals a puzzle: some sites are dramatically easier to automate than others, in ways unrelated to their technical sophistication. A React application with the latest framework requires fragile selectors that break constantly. An older site navigates through structure that survives every redesign.

The difference is invisible to most developers. But it's critical to anyone building systems that need to work reliably across thousands of sites. The infrastructure that makes automation possible emerged from a crisis nobody could see—one that reshaped how the web communicates meaning.

In 2006, someone invented infinite scrolling to keep humans engaged. No more clicking "Next Page." Just scroll. Content appears. Social media platforms adopted it because engagement metrics soared.

Google's crawlers don't scroll. They land on a page, index what's visible, and move on. Content that loads only after scrolling might never exist for search engines. In 2020, Google's Martin Splitt confirmed what developers already knew: "Googlebot does not scroll."

The fix? Maintain two navigation systems. Infinite scroll for users. Paginated URLs for crawlers. Your engineering team now supports parallel architectures because a 2006 engagement optimization never considered machine speed. Web automation hits the same wall. Scrapers see initial content, miss everything below the fold. A UX pattern optimized for human behavior created permanent friction with how machines traverse the web.

Hybrid implementation: Sites maintain infinite scroll for users while exposing ?page= URLs for search engines to crawl sequentially.

Automation blindness: Web scrapers face identical challenges since they don't execute JavaScript or scroll, capturing only initial viewport content.

Universal adoption: Now standard on TikTok, Instagram, Twitter, Apple News, Nike.com wherever continuous content streams trump discrete pages.

Performance tradeoff: Lazy loading accelerates initial page speed but creates indexing gaps, forcing balance between experience and discoverability.

Architectural consequence: Optimizing for human scrolling behavior permanently complicated machine-based web traversal at production scale.