alyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 7 months agoThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comexternal-linkmessage-square10fedilinkarrow-up14arrow-down10cross-posted to: technology@lemmy.world
arrow-up14arrow-down1external-linkThe rise and fall of robots.txt: As unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.www.theverge.comalyaza [they/she]@beehaw.orgM to Technology@beehaw.orgEnglish · 7 months agomessage-square10fedilinkcross-posted to: technology@lemmy.world
minus-squaregayhitler420@lemm.eelinkfedilinkarrow-up1·7 months agorobots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.
robots.txt isn’t a basic social contract, it’s a file intended to save web crawlers precious resources.