AI can be good. Can. But it currently isn’t – and disrupts everything from news to your basic search query. To block your website from most bots, do the following:
First, create a file called
llms.txt
and add it to the root folder of your website, so that it reaches at yourdomain.com/llms.txt
Add this to the file:
# Block LLM and AI scraping
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Cohere-ai
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Then, go to your robots.txt file (f.ex. in SEOPress and Rank Math; or directly in your root). If you don’t have one, create one. Add the following line to your robots.txt:
Include: /llms.txt
This should stop every major AI Crawler and Bot from accessing your website. Please note: That includes inputs and prompts you do yourself. Once added, the crawler can’t access your website anymore, even when directly prompted to.
Leave a Reply