NullPointer@programming.devtoPrivacy@lemmy.ml•How to block AI Crawler Bots using robots.txt file
19·
3 months agorobots.txt will not block a bad bot, but you can use it to lure the bad bots into a “bot-trap” so you can ban them in an automated fashion.
deleted by creator