Starting on November 16, 2026, conversations in Discoverable Servers will be easier to find for people searching the web. Admins can choose which channels are surfaced on the web. Private servers, DMs, and GDMs are not included.
A lot of article websites whitelist Google’s crawler, right? Can’t that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though
Now lemmy servers need to do the same.
Maybe some do, but they never show up in search queries whenever I tried. I even filtered by sites.
I see Lemmy/Piefed results through DuckDuckGo.
I’m using DuckDuckGo too, which server are you seeing on the results? I’ll try to filter specifically for that in the future.
PieFed.social, but I’ve occasionally seen some lemmy.world ones as well.
For example, just try searching for: “starcitizen steamdeck guide”
And third result currently goes to PieFed (a PieFed.social account posting to a Lemmy.world community).
A lot of lemmy servers have Human Verification in front of them maybe that’s why?
Could whitelist specific search engine crawlers
The bots will pretend to be search engine crawlers. And not all have their own IP ranges.
You can’t impersonate search engine crawlers, because the server can verify that it’s really a search engine crawler
See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot
A lot of article websites whitelist Google’s crawler, right? Can’t that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though