Making My Blog Readable by AI Agents

(deallocatedobjects.com)

29 points | by ingve 20 hours ago

5 comments

  • naw103 3 hours ago
    One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
  • noworld 53 minutes ago
  • Muhammad523 1 hour ago
    why? You're helping AI companies to take your writing for free
  • Hissigh 11 hours ago
    Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.

    https://www.rebootonline.com/geo/llms-txt-experiment/

    • nubinetwork 8 hours ago
      I get the odd one try on my personal site, but usually its just robots.txt... /shrug
  • aburayhan 1 hour ago
    [flagged]