Valuable News – 2025/11/10

RE: bot feeding

Interesting because for the last few weeks I've been working on something for bots/scrapers. They are relentless! The idea of a `mmap` is interesting but "raw speed" can be your own enemy too (I'm thinking a few milliseconds will actually help your jumbled response seem more normal--unless, I guess, you typically only serve plain text).

Scrapers use some pretty sophisticated tactics like site/header fingerprinting, response timing heuristics, etc., so I'm not totally convinced just feeding them a constant/repeating stream of jumbled text will really fool/feed the more sophisticated ones. However, I too am building a "text jumble" that will make random outputs (with a more constant header) in conjunction with automatic adjusting rate limiting. I think a system should tackle this growing problem on several fronts because some of these scrapers seem very good at trying to appear normal.
 
Back
Top