The rise and fall of robots.txt
www.theverge.com
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
  • YTG123@feddit.chEnglish
    1·
    9 months ago

    We need laws mandating respect of robots.txt. This is what happens when you don’t codify stuff

    • patatahooligan@lemmy.worldEnglish
      1·
      9 months ago

      AI companies will probably get a free pass to ignore robots.txt even if it were enforced by law. That’s what they’re trying to do with copyright and it looks likely that they’ll get away with it.

    • Echo Dot@feddit.ukEnglish
      11·
      9 months ago

      It’s a bad solution to a problem anyway. If we are going to legally mandate a solution I want to take the opportunity to come up with an actually better fix than the hacky solution that is robots.txt