šŸ¤–

CheckAIBots

See Which AI Bots Can Access Your Website

Tutorial

Nginx Tutorial: Block All AI Bots in 5 Minutes (2025 Guide)

ā€¢8 min read

Nginx Tutorial: Block All AI Bots in 5 Minutes (2025 Guide)

Stop AI crawlers from draining your bandwidth with this simple Nginx configuration. In just 5 minutes, you'll have server-level blocking that works against all 29 AI bots ā€” including aggressive crawlers like Bytespider that ignore robots.txt.

This guide provides copy-paste ready code with no technical expertise required.

Why Server-Level Blocking with Nginx?

robots.txt vs Nginx Blocking

Method robots.txt Nginx Blocking
Effectiveness Depends on bot compliance 100% effective
Works for Bytespider āŒ No (ignores it) āœ… Yes
Works for 360Spider āŒ No (ignores it) āœ… Yes
Technical difficulty Easy Moderate
Performance impact None Minimal
Can't be bypassed āŒ No āœ… Yes

Bottom line: robots.txt is a polite request. Nginx blocking is a wall.

When to Use Nginx Blocking

  • āœ… You have aggressive crawlers ignoring robots.txt (Bytespider, 360Spider)
  • āœ… You want 100% guaranteed blocking
  • āœ… You're losing 30%+ bandwidth to AI crawlers
  • āœ… You have SSH access to your server

Prerequisites

Before starting, make sure you have:

  • āœ… Nginx web server installed
  • āœ… SSH access to your server
  • āœ… Root or sudo privileges
  • āœ… Basic command line knowledge

Check if you have Nginx:

nginx -v
# Should output: nginx version: nginx/1.x.x

Don't have Nginx? This tutorial also works for:

  • Ubuntu/Debian servers
  • CentOS/RHEL servers
  • Cloud servers (AWS, DigitalOcean, Linode, etc.)
  • VPS hosting

Step 1: Create AI Bot Blocking Configuration File

We'll create a dedicated configuration file for blocking AI bots. This keeps your main Nginx config clean and makes future updates easy.

Create the configuration file:

sudo nano /etc/nginx/conf.d/block-ai-bots.conf

Paste this configuration:

# Block AI Crawlers - CheckAIBots.com
# Last updated: 2025-02-03
# Blocks 29 AI crawlers including GPTBot, ClaudeBot, Bytespider, etc.

map $http_user_agent $block_ai_bots {
    default 0;

    # OpenAI
    "~*GPTBot" 1;
    "~*ChatGPT-User" 1;
    "~*OAI-SearchBot" 1;

    # Anthropic (Claude)
    "~*ClaudeBot" 1;
    "~*Claude-Web" 1;
    "~*anthropic-ai" 1;
    "~*anthropic-research" 1;

    # Google AI
    "~*Google-Extended" 1;
    "~*Gemini-Deep-Research" 1;

    # Meta (Facebook)
    "~*FacebookBot" 1;
    "~*Meta-ExternalAgent" 1;
    "~*Meta-ExternalFetcher" 1;

    # ByteDance (AGGRESSIVE - ignores robots.txt)
    "~*Bytespider" 1;

    # Baidu
    "~*Baiduspider" 1;
    "~*ErnieBot" 1;

    # Apple
    "~*Applebot-Extended" 1;

    # Amazon
    "~*Amazonbot" 1;

    # Common Crawl
    "~*CCBot" 1;

    # Other AI companies
    "~*cohere-ai" 1;
    "~*PerplexityBot" 1;
    "~*YouBot" 1;
    "~*Diffbot" 1;
    "~*omgilibot" 1;
    "~*MistralAI-User" 1;

    # Chinese AI crawlers (AGGRESSIVE)
    "~*360Spider" 1;
    "~*ChatGLM-Spider" 1;
    "~*Sogou" 1;
    "~*DeepseekBot" 1;
    "~*PanguBot" 1;
}

Save the file:

  • Press Ctrl + O to save
  • Press Enter to confirm
  • Press Ctrl + X to exit

Step 2: Add Blocking Rule to Your Server Block

Now we need to tell Nginx to actually block these bots.

Find your main Nginx configuration:

sudo nano /etc/nginx/sites-available/default

Or if you have a custom site configuration:

sudo nano /etc/nginx/sites-available/yoursite.com

Add this inside your server block:

server {
    listen 80;
    server_name yoursite.com;

    # Block AI bots (add this line)
    if ($block_ai_bots) {
        return 403;
    }

    # Rest of your configuration...
    location / {
        # ...
    }
}

Where to add it: Right after server_name, before any location blocks.

Full example:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;

    # ADD THIS SECTION
    # Block AI bots
    if ($block_ai_bots) {
        return 403;
    }
    # END OF ADDITION

    root /var/www/html;
    index index.html index.php;

    location / {
        try_files $uri $uri/ =404;
    }

    # PHP and other configurations...
}

Save the file (Ctrl + O, Enter, Ctrl + X).

Step 3: Test and Reload Nginx

Before applying changes, always test your Nginx configuration:

Test configuration syntax:

sudo nginx -t

Expected output:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If you see errors:

  • Double-check you added the code in the right place
  • Make sure all { } brackets are balanced
  • Check for typos in file paths

Reload Nginx to apply changes:

sudo systemctl reload nginx

Or:

sudo service nginx reload

That's it! AI bots are now blocked. šŸŽ‰

Step 4: Verify Blocking Is Working

Method 1: Test with curl

Simulate an AI crawler request:

# Test GPTBot blocking
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://yoursite.com

# Expected output:
# <html>
# <head><title>403 Forbidden</title></head>
# <body>
# <center><h1>403 Forbidden</h1></center>
# </body>
# </html>

# Test normal user access (should work)
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" https://yoursite.com

# Expected: Your normal website HTML

Method 2: Check Nginx Access Logs

Monitor what's being blocked:

# View recent blocks (403 responses)
sudo tail -f /var/log/nginx/access.log | grep " 403 "

Example output:

220.181.108.89 - - [03/Feb/2025:10:45:23] "GET / HTTP/1.1" 403 Bytespider
185.230.63.107 - - [03/Feb/2025:10:46:12] "GET /blog HTTP/1.1" 403 ClaudeBot

Method 3: Use CheckAIBots.com

After 24-48 hours, run a full check:

  1. Visit CheckAIBots.com
  2. Enter your domain
  3. Verify all bots show as blocked

Customization Options

Option 1: Block Only Aggressive Crawlers

If you want to block only bots that ignore robots.txt (Bytespider, 360Spider, etc.):

map $http_user_agent $block_ai_bots {
    default 0;

    # Only block aggressive crawlers
    "~*Bytespider" 1;
    "~*360Spider" 1;
    "~*ChatGLM-Spider" 1;
    "~*Sogou" 1;
}

Option 2: Allow Specific Bots

To allow certain bots while blocking others:

map $http_user_agent $block_ai_bots {
    default 0;

    # Block most AI bots
    "~*GPTBot" 1;
    "~*ClaudeBot" 1;
    "~*Bytespider" 1;
    # ... (other bots)

    # Allow ChatGPT user-initiated browsing
    "~*ChatGPT-User" 0;
    "~*Claude-Web" 0;
}

Option 3: Return Custom Error Page

Instead of 403 Forbidden, show a custom message:

Create custom error page:

sudo nano /var/www/html/ai-bot-blocked.html

<!DOCTYPE html>
<html>
<head>
    <title>AI Crawler Access Denied</title>
    <style>
        body { font-family: Arial; text-align: center; padding: 50px; }
        h1 { color: #e74c3c; }
    </style>
</head>
<body>
    <h1>AI Crawler Blocked</h1>
    <p>This website does not allow AI crawlers to access its content.</p>
    <p>If you believe this is an error, please contact the site owner.</p>
</body>
</html>

Update Nginx configuration:

server {
    # ... server configuration

    if ($block_ai_bots) {
        return 403 /ai-bot-blocked.html;
    }
}

Option 4: Log Blocked Requests Separately

Create a dedicated log file for blocked AI bots:

map $http_user_agent $block_ai_bots {
    # ... (blocking rules)
}

map $block_ai_bots $ai_bot_log {
    0 "/var/log/nginx/access.log";
    1 "/var/log/nginx/ai-bots-blocked.log";
}

server {
    access_log $ai_bot_log;

    if ($block_ai_bots) {
        return 403;
    }
}

Then monitor AI bot blocking:

sudo tail -f /var/log/nginx/ai-bots-blocked.log

Troubleshooting

Problem 1: Nginx won't reload

Error: nginx: [emerg] unexpected "}" in /etc/nginx/conf.d/block-ai-bots.conf:45

Solution: Check for syntax errors:

sudo nginx -t

Make sure all brackets { } are balanced.

Problem 2: Configuration doesn't work

Symptoms: Bots are still accessing your site

Solutions:

  1. Check if the map is loaded:
sudo nginx -T | grep block_ai_bots
# Should show your map configuration

  1. Verify the if statement is in the right place:

    • Must be inside server { } block
    • Must be before location / { } blocks

  2. Clear browser/CDN cache:

    • If using Cloudflare, purge cache
    • Test with curl instead of browser

Problem 3: Legitimate users getting blocked

Symptoms: Normal visitors see 403 errors

Solution: Check your user agent pattern:

# View recent blocked requests
sudo tail -100 /var/log/nginx/access.log | grep " 403 "

If you see legitimate user agents blocked, adjust your regex patterns to be more specific.

Problem 4: Blocking only works for some bots

Issue: User agent matching is case-sensitive

Solution: Use ~* (case-insensitive) instead of ~:

# Correct (case-insensitive)
"~*GPTBot" 1;

# Wrong (case-sensitive)
"~GPTBot" 1;

Performance Impact

Will blocking AI bots slow down my site?

No. The performance impact is negligible:

  • map directive is evaluated once per request
  • Pattern matching is extremely fast (microseconds)
  • Blocked requests are rejected immediately (no backend processing)

Benchmark: On a server handling 10,000 requests/minute, adding this configuration adds ~0.01ms per request.

Actually improves performance

By blocking AI bots, you'll:

  • āœ… Reduce server load (30-75% fewer requests)
  • āœ… Lower bandwidth usage
  • āœ… Decrease CDN costs
  • āœ… Free up resources for real users

How Much Bandwidth Will You Save?

Real-world results:

Website Type Traffic Before AI Bot % Bandwidth Saved
Tech Blog 2.8M requests/mo 42% 1.18M requests
E-commerce 890K requests/mo 31% 276K requests
News Site 5.2M requests/mo 68% 3.54M requests

Average savings: 30-75% reduction in AI crawler traffic

Cost savings: $500-$3,000/month in bandwidth and CDN costs for medium-traffic sites.

Alternative: Block by IP Address (Advanced)

Some AI companies publish their crawler IP ranges. You can block by IP instead of user agent:

# Block GPTBot by IP range
geo $block_gptbot {
    default 0;
    23.98.142.176/28 1;  # GPTBot IP range
}

server {
    if ($block_gptbot) {
        return 403;
    }
}

Pros: Can't be spoofed by changing user agent

Cons: IP ranges change, requires manual updates

Recommendation: Use user agent blocking (easier to maintain).

Updating the Block List

AI crawlers evolve. New bots emerge monthly.

How to add new bots:

  1. Edit the configuration:
sudo nano /etc/nginx/conf.d/block-ai-bots.conf
  1. Add new user agent:
map $http_user_agent $block_ai_bots {
    # ... existing bots

    # Add new bot
    "~*NewAIBot" 1;
}
  1. Test and reload:
sudo nginx -t
sudo systemctl reload nginx

Stay updated:

  • Check CheckAIBots.com/blog monthly for updates
  • Monitor /var/log/nginx/access.log for unfamiliar crawlers

Complete Configuration Summary

Here's the full configuration for quick reference:

1. /etc/nginx/conf.d/block-ai-bots.conf

map $http_user_agent $block_ai_bots {
    default 0;
    "~*GPTBot" 1;
    "~*ClaudeBot" 1;
    "~*Google-Extended" 1;
    "~*Bytespider" 1;
    "~*CCBot" 1;
    "~*ChatGPT-User" 1;
    "~*Claude-Web" 1;
    "~*anthropic-ai" 1;
    "~*FacebookBot" 1;
    "~*Meta-ExternalAgent" 1;
    "~*Applebot-Extended" 1;
    "~*Amazonbot" 1;
    "~*cohere-ai" 1;
    "~*PerplexityBot" 1;
    "~*360Spider" 1;
    "~*ChatGLM-Spider" 1;
    "~*Sogou" 1;
    "~*Baiduspider" 1;
    "~*DeepseekBot" 1;
    "~*PanguBot" 1;
    "~*Diffbot" 1;
    "~*omgilibot" 1;
    "~*YouBot" 1;
    "~*ErnieBot" 1;
    "~*Gemini-Deep-Research" 1;
    "~*Meta-ExternalFetcher" 1;
    "~*OAI-SearchBot" 1;
    "~*anthropic-research" 1;
    "~*MistralAI-User" 1;
}

2. Add to your server block:

server {
    listen 80;
    server_name yoursite.com;

    if ($block_ai_bots) {
        return 403;
    }

    # ... rest of configuration
}

3. Test and reload:

sudo nginx -t
sudo systemctl reload nginx

Frequently Asked Questions

Will this block Google search crawlers?

No. This configuration blocks Google-Extended (AI training) but NOT Googlebot (search). Your SEO is safe.

Can AI companies bypass this?

Not easily. They would need to:

  • Use different user agents (detectable in logs)
  • Use residential IPs (expensive at scale)
  • Rotate identities (violates terms of service)

In practice, major AI companies respect server-level blocks.

Should I also use robots.txt?

Yes! Use both:

  • robots.txt for compliant bots (polite)
  • Nginx blocking for aggressive bots (enforcement)

This covers all scenarios.

Does this work on Apache?

The concepts are the same, but syntax differs. See our Apache blocking guide for equivalent configuration.

What if I don't have root access?

If you're on shared hosting without Nginx access:

  • Use robots.txt only (less effective)
  • Ask your hosting provider to implement server-level blocking
  • Consider switching to VPS hosting for full control

Next Steps

āœ… Congratulations! You've successfully blocked AI crawlers at the server level.

Recommended actions:

  1. Monitor results:

    # Check blocked requests
sudo tail -f /var/log/nginx/access.log | grep " 403 "

  2. Measure bandwidth savings:

    • Compare bandwidth usage before/after 7 days
    • Check your CDN dashboard for cost reductions

  3. Verify blocking:

  4. Update monthly:

    • New AI bots emerge regularly
    • Subscribe to CheckAIBots updates

Conclusion

With just 5 minutes of work, you now have bulletproof protection against 29 AI crawlers. Unlike robots.txt, server-level blocking with Nginx cannot be ignored ā€” even by aggressive crawlers like Bytespider.

Key benefits:

  • āœ… 100% effective blocking
  • āœ… Works for all 29 AI crawlers
  • āœ… Minimal performance impact
  • āœ… Easy to update and maintain
  • āœ… Save 30-75% bandwidth costs

Remember: This doesn't hurt your SEO. Search engine crawlers like Googlebot are completely unaffected.

Related tutorials:

Need help? Check your blocking status ā†’

Ready to Check Your Website?

Use CheckAIBots to instantly discover which AI crawlers can access your website and get actionable blocking recommendations

Free AI Crawler Check
Back to Blog