Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 年前How to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square63linkfedilinkarrow-up1112
arrow-up180external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 2 年前message-square63linkfedilink
minus-squareJackbyDev@programming.devlinkfedilinkEnglisharrow-up2arrow-down4·2 年前It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
minus-squarethanks_shakey_snake@lemmy.calinkfedilinkarrow-up3·2 年前Maybe? But it’s not like that’s the only alternative thing to say, lol
minus-squareReversalHatchery@beehaw.orglinkfedilinkEnglisharrow-up2arrow-down1·2 年前Is it, though? I mean, robots.txt is the Do Not Track of the opposite side of the connection.
It’s better than saying something like “there’s no point in robots.txt because bots can disobey is” though.
Maybe? But it’s not like that’s the only alternative thing to say, lol
Is it, though?
I mean, robots.txt is the Do Not Track of the opposite side of the connection.