How to Block AI Crawlers That Slow Down Your WordPress Website

Do you want to block AI crawlers that can slow down your WordPress website or WooCommerce store? If you are looking for a simple guide, keep reading this article.

As artificial intelligence evolves, many AI crawlers actively scan websites to collect data.

While some of this activity may seem harmless, excessive crawling by these bots can slow down your WordPress website and strain server resources.

In this guide, we’ll explain why and how to block AI crawlers effectively to protect your site’s performance and maintain a better user experience.

But before going further, let’s see what AI crawlers are and what they do.

What Are AI Crawlers

AI crawlers are automated bots used by artificial intelligence tools to scan and collect website data.

These crawlers visit web pages, analyze their content, and often use that data to train language models, improve search capabilities, or build AI-powered services.

Unlike traditional search engine crawlers, AI crawlers may extract significant portions of content without directly driving traffic or providing SEO benefits, making them a growing concern for website owners focused on performance and content protection.

Now that you know what AI crawlers are, the next section will explain why blocking them is a good idea.

Why Blocking AI Crawlers Is a Good Idea

AI crawlers often scan websites to collect data for training language models or powering AI tools. While that might sound harmless, it can lead to increased server load, slower site performance, and unauthorized use of your original content.

For WordPress users who value speed and content protection, blocking AI crawlers helps maintain control over your bandwidth and how your site is accessed. It’s also an effective way to reduce unnecessary traffic from bots that offer little to no benefit to your SEO or user engagement.

By restricting these crawlers, you safeguard your content, improve loading times, and ensure your server resources are used efficiently for real visitors.

List of Common AI Crawlers

Here is a list of the top AI crawlers you need to know:

• GPTBot (by OpenAI)
• CCBot (by Common Crawl)
• Amazonbot (by Amazon)
• ClaudeBot / AnthropicBot (by Anthropic)
• FacebookBot (by Meta)
• Google-Extended (for Bard / Gemini data)
• Bytespider (by ByteDance / TikTok)
• Applebot (used by Apple for Siri and Spotlight)
• YandexBot (by Yandex, Russia’s search engine)

Next, let’s see how we can block these bots.

How to Block AI Crawlers that Slow Down the Website

There are three methods to block AI crawlers in WordPress:

• Via the robots.txt file (famous)
• Using the .htaccess file
• With a security plugin

We will show you all three methods below.

1. Robots.txt Method

The simplest method to block AI crawlers is to edit the robots.txt file. Search engine crawlers use the robots.txt file to understand what to crawl and ignore.

To finish this task, you must use a plugin to edit the robots.txt. Any SEO plugin like Rank Math or Yoast comes with this feature.

Here, we will use the File Manager plugin to view and edit the robots.txt file. After activating the plugin, open the settings.

And find the robots.txt file.

By default, the file’s contents will look something like this:

Now, you need to add this content to the file:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Make sure to edit the bot name to fit your needs. Once you have modified the code, you can paste it inside the robots.txt file like this:

Once you are done with it, save the file. You can try accessing the robots.txt file via www.yourdomain.com/robots.txt.

That’s it.

Check out the next section if you need to use the .htaccess method.

2. Htaccess Method

We will be using the same File Manager plugin to tweak the file. The code we are going to use is:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Amazonbot [NC]
RewriteRule .* - [F,L]
</IfModule>

Feel free to tweak the code and add more user agents. Once you are finished, open the .htaccess file and paste it inside.

Save the file, and all the bots you mentioned in the code will be blocked. Next section will show you how to use a plugin to block AI crawlers.

3. Using a Dedicated Security Plugin

For this tutorial, we will use the All-In-One Security plugin. First, install and activate the plugin on your website.

Once you are finished activating it, you can see the plugin’s settings on the left-hand side.

Under the Firewall settings, you can see the block and allow lists.

Now, scroll to the bottom and enable the blocking feature. In the box, enter the user agents and save the settings.

Once you have saved it, all the user agents in the list cannot access the website.

That’s it!

This is how you can use a plugin to block AI crawlers.

Possible Downsides of Blocking AI Crawlers

While blocking AI crawlers may protect your content and server resources, it’s essential to consider a few trade-offs:

• Overblocking Risks: If you misconfigure your robots.txt or firewall rules, you might accidentally block useful crawlers, affecting SEO, indexing, or third-party services like uptime monitors or analytics tools.
• Reduced Visibility in AI Search Tools: By blocking bots like GPTBot or Google-Extended, your content won’t be included in generative AI results like ChatGPT answers or Google’s AI Overviews, potentially reducing exposure.
• Overblocking Risks: If you misconfigure your robots.txt or firewall rules, you might accidentally block useful crawlers, affecting SEO, indexing, or third-party services like uptime monitors or analytics tools.
• Loss of Indirect Traffic Sources: Content used in AI training may sometimes lead users back to the source. You could cut off this indirect traffic by blocking AI crawlers.

Frequently Asked Questions

Next, see some frequently asked questions and answers regarding this topic.

Conclusion

If you’re serious about your website’s speed and data privacy, it’s crucial to block AI crawlers that do not add value to your site.

These bots can consume server resources, affect loading times, and potentially scrape your content without permission.

By setting up proper rules in your robots.txt file, using security plugins, or leveraging CDN-level tools, you can take better control over unwanted bot traffic.

Blocking unnecessary crawlers isn’t just a performance tweak—it’s a smart move for long-term site health and SEO. Keep your WordPress site fast, clean, and focused on human users who matter most.

Which method are you going to choose to block AI crawlers?

Let us know in the comments.