I host a few sites and I haven’t had a problem yet. Sure, logs suggest a majority of traffic is bots, but the cheap VPS I’m hosting on hasn’t melted so I haven’t been motivated to do anything about it.
Lemmy is particularly susceptible. Hosting a static site with a limited number of pages is fine. But Lemmy needs to check all posts, find the votes, sort them, and compile the page. It does that for practically every page load, and so loading a page once is quite “expensive”.
Then add on to that the almost unlimited pages of content for bots to crawl through, bots just get caught endlessly scraping pages. Even powerful servers struggle, which is why most Lemmy instances have implemented some sort of protection against bots.
Why not just use more static pages?
4chan has been ‘static’ forever, the backend builds each page every ~15s, pages served to users are always static. It’s ‘justwerked’ for 20 years now on a shoestring.
I believe you, but the linked article is not about hosting Lemmy.
No it’s not, it was more of a tangential comment.
I think this is the photography site he’s talking about.
He mentioned “so many scrapers hit my site that it saturated my upload with generated content feeding the them.” which seemed unusual to me. I’m not sure why such a site would have a public upload. It doesn’t feel like a site that would have a technical challenge with the scrapers, but I also get the impression the main issue is the thought of hosting all his photos just for AI scrapers to train their image generation models.
If I didn’t enjoy the cat & mouse of it, I’d definitely walk away from running a site. Just out of curiosity, I created an hourly graph today showing malicious bot, legitimate bot and human ratio graphs for one of my sites. The malicious hits make up around 80% hourly and human hits are usually well under 10%.
Yeah, internet is no longer the interesting place it used to be. Interesting, intriguing, exciting. Now it’s worse than going to a tax office… the day before taxes… On Christmas… 😂
I run a DaaS product Trace Jobs that serves job posting data.
My expectations are pretty low and am never disappointed. Self-hosting can be pretty affordable.
