Cynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agoHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizexternal-linkmessage-square64fedilinkarrow-up1109arrow-down132
arrow-up177arrow-down1external-linkHow to block AI Crawler Bots using robots.txt filewww.cyberciti.bizCynicus Rex@lemmy.ml to Privacy@lemmy.mlEnglish · 3 months agomessage-square64fedilink
minus-squarebreadsmasher@lemmy.worldlinkfedilinkEnglisharrow-up10·3 months agoIt isn’t an enforceable solution. robots.txt and similar are just please bots dont index these pages. Doesn’t mean any bots will respect it
It isn’t an enforceable solution. robots.txt and similar are just please bots dont index these pages. Doesn’t mean any bots will respect it