• Indonesian
  • English

    • Block AI Bot & Crawler via .htaccess — Server Back to Speed

    📑 Daftar Isi

    1. Introduction
    2. Symptoms & Logs — What Was Happening on the Server?
      1. Trace Pattern — Reading Logs from Symptom to Root Cause
      2. Pattern Summary
    3. Root Cause — Why Was the Server Slow Even Though Bots Got 403?
      1. List of AI Bots & Crawlers That Need Blocking
    4. Step-by-Step Solution — Block via .htaccess
      1. Step 1: Backup Existing .htaccess
      2. Step 2: Identify Bots in the Access Log
      3. Step 3: Add Blocking Rules in .htaccess
      4. Step 4: For Apache 2.2 (Older cPanel)
      5. Step 5: Verification
      6. Step 6: Monitor the Results
    5. Pro Tips & Warnings
    6. FAQ
      1. Q: Will blocking AI bots via .htaccess affect my websites SEO on Google?
      2. Q: Why not just use robots.txt? Isnt it simpler?
      3. Q: I use LiteSpeed/OpenLiteSpeed, not Apache. Can this .htaccess be used?
      4. Q: How long after editing .htaccess until the effect is felt?
      5. Q: What if theres a new bot not on the list?
    7. Conclusion

    Kecepatan:

    Difficulty: Beginner
    Last Updated: July 20, 2026
    Tested On: Apache 2.4, cPanel/WHM, HestiaCP, CloudLinux, CentOS 7/8, AlmaLinux 8/9

    Introduction

    3 AM. My phone rang — not an alarm, but an actual phone call from a helpdesk operator at a District Court in Central Java. The voice was panicked: “Sir, the SIPP website is extremely slow. Since this morning, users have been complaining they cant open pages. Whats going on?”

    SIPP is the Case Tracking Information System — an application used by courts to publish case data to the public. The fictitious domain well call it pa-contohkabupaten.go.id. The website runs on Apache + PHP + MySQL, hosted on a dedicated server that should have been more than enough for a district courts traffic.

    I got up, opened my laptop, SSHd into the server. Ran top — load average was up to 45. And this server only has 8 cores. netstat -tlnp | grep :443 | wc -l — there were 487 active connections to Apache. Normally it maxes out at 30-40 connections during peak hours.

    “This server is under attack,” I thought at first. But after checking the access log, it wasnt a typical DDoS. The attackers werent a botnet from thousands of IPs — it was a handful of AI bots and crawlers making an absurd number of requests.

    From my experience handling servers slowed down by full Apache slots and full LiteSpeed slots, the problem pattern is similar: resources get exhausted not because of legitimate traffic, but because bots drain connections. The difference this time: the culprits were AI training bots specifically designed to crawl as many web pages as possible.

    This article is a complete record of how I found, analyzed, and blocked AI bots and rogue crawlers using .htaccess — until the server was back to speed in under 15 minutes.

    Symptoms & Logs — What Was Happening on the Server?

    First step when you get a slow server complaint: read the logs. Dont guess, dont just restart services. Logs never lie.

    Heres a snippet of the Apache access log I found at /usr/local/apache/logs/domlogs/pa-contohkabupaten.go.id-ssl_log. Im showing it in full — not summarized — so you can see the pattern with your own eyes:

    111.225.214.137 - - [20/Jul/2026:08:50:52 +0700] "GET /show_detil/SWpRdUk0NzJrRFltdFN5... HTTP/2" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
216.73.217.65 - - [20/Jul/2026:08:50:52 +0700] "GET /list_perkara/type/UFhiY2t4RTkzdz... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
103.125.51.222 - - [20/Jul/2026:08:50:52 +0700] "GET /show_putusan/YWpFeGRoRlNMd05... HTTP/2" 302 0 "https://pa-contohkabupaten.go.id/detil_perkara/index/id/YWpFeGRo..." "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36"
190.114.43.132 - - [20/Jul/2026:08:50:53 +0700] "GET /show_detil/aHNBakJ1TXFPRk9... HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Mediapartners-Google) Chrome/84.0.4147.140 Safari/537.36"
216.73.217.65 - - [20/Jul/2026:08:50:53 +0700] "GET /show_detil/N1FyRXNDcmZ0OXF... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
111.225.214.159 - - [20/Jul/2026:08:50:53 +0700] "GET /index.php/detil_perkara HTTP/2" 302 771 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
14.236.142.196 - - [20/Jul/2026:08:50:53 +0700] "GET /show_detil/bFRlVXlTbUtDTjB... HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
216.73.217.65 - - [20/Jul/2026:08:50:53 +0700] "GET /show_detil/V2tGZkpLRnRzMjE... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
14.236.142.196 - - [20/Jul/2026:08:50:53 +0700] "GET /index.php/detil_perkara HTTP/1.1" 302 771 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
190.114.43.132 - - [20/Jul/2026:08:50:53 +0700] "GET /index.php/detil_perkara HTTP/1.1" 302 771 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/537.36 (KHTML, like Gecko) Mediapartners-Google) Chrome/84.0.4147.140 Safari/537.36"
14.249.49.40 - - [20/Jul/2026:08:50:54 +0700] "GET /list_perkara/type/aDg1ZWdnUGdiMG... HTTP/1.1" 200 5206 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36"
216.73.217.65 - - [20/Jul/2026:08:50:54 +0700] "GET /show_detil/UnBTdlBtV2xWbW9... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
27.71.85.208 - - [20/Jul/2026:08:50:54 +0700] "GET /show_detil/WUVrcEk2RENNcVB... HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"
200.225.113.205 - - [20/Jul/2026:08:50:54 +0700] "GET /show_detil/ZGU4Y29nc0NhbnF... HTTP/1.1" 302 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4137.1 SputnikBrowser/5.6.6280.0 (GOST) Safari/537.36"
116.179.33.146 - - [20/Jul/2026:08:50:58 +0700] "GET /resources/css/bootstrap.css HTTP/1.1" 404 582 "https://pa-contohkabupaten.go.id/detil_perkara" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
216.73.217.65 - - [20/Jul/2026:08:50:58 +0700] "GET /show_detil/dXIvUmZWQWJFbTF... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.217.65 - - [20/Jul/2026:08:50:59 +0700] "GET /show_detil/aFpWWGh2bGdEczc... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.217.65 - - [20/Jul/2026:08:51:00 +0700] "GET /show_detil/MTdwaThJMnVUKzQ... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
216.73.217.65 - - [20/Jul/2026:08:51:00 +0700] "GET /show_detil/eXNwTk9rd3ZqYit... HTTP/2" 403 1242 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
116.179.33.209 - - [20/Jul/2026:08:51:00 +0700] "GET /resources/css/bootstrap.css HTTP/1.1" 404 582 "https://pa-contohkabupaten.go.id/list_perkara/type/..." "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"

    Trace Pattern — Reading Logs from Symptom to Root Cause

    Lets read the log above line by line. Notice the patterns that emerge:

    Line 1 — IP 111.225.214.137, user-agent Chrome 48 (released in 2016!). Request to /show_detil/ with a long base64 parameter. Response 302 redirect. This isnt a normal human browser — modern browsers dont use Chrome 48 anymore. This is a bot masquerading as an old browser.

    Line 2 — Now this is the most glaring one. IP 216.73.217.65, user-agent ClaudeBot/1.0. Request to /list_perkara/type/. Response 403 Forbidden. This means the server already had a blocking mechanism, but the bot keeps requesting — every second, over and over.

    Lines 3-4 — Legitimate traffic from real users (Chrome 146 on Mac) and Mediapartners-Google (Googles bot for Adsense). Response is 302 redirect, which is normal.

    Lines 5-10 — The same pattern repeats: ClaudeBot from the same IP (216.73.217.65) keeps requesting /show_detil/ and /list_perkara/, all getting 403. But the bot doesnt stop. At the same time, IPs from the 111.225.214.x, 14.236.142.x, 190.114.43.x ranges are also spamming redirect chains — requesting /show_detil/ → redirect to /index.php/detil_perkara → redirect again to /detil_perkara. Three redirects for one page — this taxes Apache 3x over.

    Line 11 — IP 14.249.49.40, response 200 OK, size 5206 bytes. This is legitimate traffic — a real user who successfully loaded a page.

    Lines 12-18 — ClaudeBot again. Seven requests in 6 seconds from the same IP, all 403. And on lines 15-16 and 20, IPs from 116.179.33.x (Baidu botnet) are scanning /resources/css/bootstrap.css and getting 404. The file doesnt exist, but Apache still has to process the request.

    Pattern Summary

    From the 20 log lines above, we can count:

    • ClaudeBot: 9 requests (45%) — all 403, from the same IP
    • Old bots (Chrome 48/89/90): 5 requests (25%) — redirect chain 302
    • Baidu botnet (Chrome 99): 2 requests (10%) — scanning non-existent files, 404
    • Legitimate traffic: 2 requests (10%) — 200 OK
    • Google/Mediapartners: 2 requests (10%) — normal

    So 80% of requests on this server came from bots. And this is only 20 lines from a log that contains thousands of lines per minute. No wonder load average spiked to 45.

    Root Cause — Why Was the Server Slow Even Though Bots Got 403?

    Many people think: “They got a 403, so theyre blocked, no problem.” Thats completely wrong.

    Even when the response is 403, Apache still has to:

    1. Accept the TCP connection — consuming kernel resources for the handshake
    2. Spawn/reuse a worker process — taking 1 slot from MaxRequestWorkers
    3. Parse the HTTP request — read headers, URL, user-agent
    4. Process .htaccess — read the .htaccess file, evaluate mod_rewrite rules
    5. Generate the 403 response — render the error page, send to client
    6. Write to access log + error log — additional disk I/O

    Think of it this way: you have a security guard at the building entrance. Every suspicious person who enters, the guard has to stop what theyre doing, check the person, say “youre not allowed in,” then write it in the visitor log. If 5 people come per minute, the guard can handle it. But if 500 people per minute show up — even though all are denied entry — the guard is still overwhelmed. Nobody gets into the building, but the guard cant handle the guests they should be serving.

    Thats exactly what happened on this server. ClaudeBot sent hundreds of requests per minute. Although all got 403, Apache still burned worker slots processing each request. Worker slots used = slots that real users cant use. This is exactly like the full Apache slot case that caused 503 — the difference this time is that the slots were consumed not by human users, but by AI bots.

    List of AI Bots & Crawlers That Need Blocking

    Based on research and experience, here are the user-agents that most often drain server resources:

    AI Training Bots:

    • GPTBot — OpenAI, training ChatGPT
    • ChatGPT-User — OpenAI, crawling when users use ChatGPT browsing
    • OAI-SearchBot — OpenAI, for SearchGPT
    • ClaudeBot — Anthropic, training Claude (this was the most aggressive in my case)
    • Google-Extended — Google, training Gemini
    • PerplexityBot — Perplexity AI, training + search
    • Bytespider — ByteDance/TikTok, AI training
    • Applebot-Extended — Apple, training Apple Intelligence
    • Amazonbot — Amazon/Alexa, training
    • CCBot — Common Crawl, dataset for general AI training
    • cohere-ai — Cohere, training
    • Diffbot — AI data extraction
    • ImagesiftBot — Image AI training
    • meta-externalagent — Meta, training Llama
    • FacebookBot — Meta/Facebook
    • YouBot — You.com

    SEO/Scraper Bots:

    • AhrefsBot — SEO tool, aggressive crawling
    • SemrushBot — SEO tool
    • MJ12bot — web crawler
    • DotBot — Moz crawler
    • PetalBot — Huawei search engine
    • DataForSeoBot — SEO data

    Aggressive Botnet/Crawlers:

    • Baiduspider — Baidu (often abusive outside China)
    • YandexBot — Yandex (similarly aggressive)
    • Sogou — Chinese search engine
    • python-requests, python-httpx, curl/, wget, scrapy — scripting libraries often used by scrapers

    Step-by-Step Solution — Block via .htaccess

    SECURITY WARNING: Backup Before Editing .htaccess

    An incorrect .htaccess edit can take your entire website down (error 500 for all visitors). Before proceeding:

    1. Backup the existing .htaccess file — cp .htaccess .htaccess.backup-$(date +%Y%m%d-%H%M)
    2. Ensure mod_rewrite is active — run apache2ctl -M | grep rewrite or httpd -M | grep rewrite
    3. Test in an incognito browser after editing — make sure the website is still accessible
    4. Have SSH/FTP access ready — so you can rollback if the site errors after editing

    Step 1: Backup Existing .htaccess

    Before adding anything, backup the existing file:

    # Navigate to the website document root
cd /home/pa-contohkab/public_html/

# Backup
cp .htaccess .htaccess.backup-$(date +%Y%m%d-%H%M)

# Verify backup
ls -la .htaccess*

    Expected output:

    -rw-r--r-- 1 pa-contohkab pa-contohkab 2345 Jul 20 03:15 .htaccess
-rw-r--r-- 1 pa-contohkab pa-contohkab 2345 Jul 20 03:15 .htaccess.backup-20260720-0315

    Step 2: Identify Bots in the Access Log

    Before blocking, we need to know which bots are active on your server. Run this command to see which user-agents have the most requests:

    # View top 20 user-agents by request count
awk -F""" {print $6} /usr/local/apache/logs/domlogs/pa-contohkabupaten.go.id-ssl_log | sort | uniq -c | sort -rn | head -20

    Expected output (example):

      8472 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
  3215 Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)
  2891 Mozilla/5.0 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
  1523 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
   987 Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
   812 CCBot/2.0 (https://commoncrawl.org/faq/)
   654 Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Mobile Safari/537.36
   445 Bytespider/1.0 (compatible; bytespider; bytedance.com)
   ...

    From this output we can see ClaudeBot is the top offender (8472 requests), followed by GPTBot (3215) and PetalBot (2891). Meanwhile, legitimate traffic (Chrome 146) is only 1523 requests. So bots are clearly dominating.

    Step 3: Add Blocking Rules in .htaccess

    Open the .htaccess file in the document root, and add the following block at the very top (before any WordPress/Laravel/CodeIgniter rules):

    # ============================================
# BLOCK AI TRAINING BOTS
# ============================================
SetEnvIfNoCase User-Agent "GPTBot" bad_bot
SetEnvIfNoCase User-Agent "ChatGPT-User" bad_bot
SetEnvIfNoCase User-Agent "OAI-SearchBot" bad_bot
SetEnvIfNoCase User-Agent "ClaudeBot" bad_bot
SetEnvIfNoCase User-Agent "Google-Extended" bad_bot
SetEnvIfNoCase User-Agent "PerplexityBot" bad_bot
SetEnvIfNoCase User-Agent "Bytespider" bad_bot
SetEnvIfNoCase User-Agent "Applebot-Extended" bad_bot
SetEnvIfNoCase User-Agent "Amazonbot" bad_bot
SetEnvIfNoCase User-Agent "CCBot" bad_bot
SetEnvIfNoCase User-Agent "cohere-ai" bad_bot
SetEnvIfNoCase User-Agent "Diffbot" bad_bot
SetEnvIfNoCase User-Agent "ImagesiftBot" bad_bot
SetEnvIfNoCase User-Agent "meta-externalagent" bad_bot
SetEnvIfNoCase User-Agent "FacebookBot" bad_bot
SetEnvIfNoCase User-Agent "YouBot" bad_bot
SetEnvIfNoCase User-Agent "Omgilibot" bad_bot

# ============================================
# BLOCK SEO/SCRAPER BOTS
# ============================================
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "SemrushBot" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "DotBot" bad_bot
SetEnvIfNoCase User-Agent "PetalBot" bad_bot
SetEnvIfNoCase User-Agent "BLEXBot" bad_bot
SetEnvIfNoCase User-Agent "DataForSeoBot" bad_bot

# ============================================
# BLOCK AGGRESSIVE BOTNET / CRAWLERS
# ============================================
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
SetEnvIfNoCase User-Agent "YandexBot" bad_bot
SetEnvIfNoCase User-Agent "Sogou" bad_bot
SetEnvIfNoCase User-Agent "Exabot" bad_bot
SetEnvIfNoCase User-Agent "ZoominfoBot" bad_bot
SetEnvIfNoCase User-Agent "Nimbostratus-Bot" bad_bot
SetEnvIfNoCase User-Agent "Go-http-client" bad_bot
SetEnvIfNoCase User-Agent "python-requests" bad_bot
SetEnvIfNoCase User-Agent "python-httpx" bad_bot
SetEnvIfNoCase User-Agent "curl/" bad_bot
SetEnvIfNoCase User-Agent "wget" bad_bot
SetEnvIfNoCase User-Agent "scrapy" bad_bot

# ============================================
# DENY ACCESS — Apache 2.4+
# ============================================
<RequireAll>
    Require all granted
    Require not env bad_bot
</RequireAll>

    How it works:

    • SetEnvIfNoCase User-Agent "ClaudeBot" bad_bot — if the user-agent contains “ClaudeBot” (case-insensitive), set the bad_bot environment variable
    • <RequireAll> Require not env bad_bot — deny all requests that have the bad_bot environment variable
    • Bots get 403 at the Apache level, before PHP/MySQL are even processed — this is much lighter than processing the request all the way to the application

    Step 4: For Apache 2.2 (Older cPanel)

    If your server still uses Apache 2.2 (usually older cPanel versions), replace the last <RequireAll> block with:

    # ============================================
# DENY ACCESS — Apache 2.2
# ============================================
Order Allow,Deny
Allow from all
Deny from env=bad_bot

    Step 5: Verification

    After saving the .htaccess file, we verify the block is working:

    # Test from another server (or use curl with ClaudeBot user-agent)
curl -I -A "Mozilla/5.0 (compatible; ClaudeBot/1.0)" https://pa-contohkabupaten.go.id/

    Expected output:

    HTTP/2 403
server: Apache
content-type: text/html; charset=UTF-8
x-robots-tag: noindex, nofollow, noarchive

    Status 403 — ClaudeBot is successfully blocked.

    Now test with a normal user-agent:

    curl -I -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0" https://pa-contohkabupaten.go.id/

    Expected output:

    HTTP/2 200
server: Apache
content-type: text/html; charset=UTF-8

    Status 200 — legitimate users can still access normally.

    Step 6: Monitor the Results

    Wait 5-10 minutes, then check the access log again. Compare bot request counts before and after:

    # Count ClaudeBot requests in the last 5 minutes
grep "ClaudeBot" /usr/local/apache/logs/domlogs/pa-contohkabupaten.go.id-ssl_log | awk {print $4} | cut -d: -f1-4 | sort | uniq -c

# Check current load average
uptime

    In my case, after the .htaccess rules were applied:

    • Load average dropped from 45 → 3.2 within 5 minutes
    • Active connections to Apache dropped from 487 → 52
    • SIPP page response time returned to 200-400ms (previously 15-30 seconds)

    Pro Tips & Warnings

    • Put blocking rules at the very top of .htaccess — so theyre processed before other mod_rewrite rules. If you put them at the bottom, bot requests still get processed by the other rules first before being blocked — still wasting resources
    • For best performance, put them in the VirtualHost config — rules in .htaccess are re-read on every request. If you put them directly in the Apache VirtualHost config (<VirtualHost> ... </VirtualHost>), the rules are only read once at Apache startup. But this requires root access and an Apache restart after editing
    • Dont block regular Googlebot — Googlebot is different from Google-Extended. Regular Googlebot is needed for Google Search indexing. What we block is Google-Extended, which is specifically for training Gemini AI
    • Dont block Bingbot — same as Googlebot, Bingbot is needed for Bing indexing
    • Update the bot list regularly — new AI bots keep appearing. Check logs once a month for any new suspicious user-agents
    • Consider Cloudflare — .htaccess only blocks at the Apache level, meaning requests still reach your server. With Cloudflare, bots are blocked at the edge network — they never even reach your server. The free tier is sufficient for Bot Fight Mode
    • Be careful blocking curl/ and wget — if you or your team often use curl/wget for maintenance, this rule will block your access too. Solution: add SetEnvIfNoCase X-Maintenance "true" good_bot and Require env good_bot as a bypass
    • Log blocked requests — add CustomLog logs/blocked-bots.log common env=bad_bot to Apache config so you can analyze which bots are being blocked

    FAQ

    Q: Will blocking AI bots via .htaccess affect my websites SEO on Google?

    No. Regular Googlebot (used for indexing) has the user-agent Googlebot — not Google-Extended. We only block Google-Extended, which is specifically used for training Gemini AI. So Google indexing remains normal. Same for Bingbot, DuckDuckBot — none are affected. If youre unsure, you can verify Googles official user-agents in Googles official documentation.

    Q: Why not just use robots.txt? Isnt it simpler?

    robots.txt is like a “No Trespassing” sign on your fence. People who respect rules will stop. But thieves? They come in anyway. AI bots — especially aggressive ones — dont care about robots.txt. The ClaudeBot in my logs kept requesting even after the server returned 403. So robots.txt can be used as an additional layer, but it doesnt replace .htaccess. Blocking in .htaccess is like a fence thats actually locked — regardless of whether the bot respects rules, it simply cannot get in.

    Q: I use LiteSpeed/OpenLiteSpeed, not Apache. Can this .htaccess be used?

    Yes, with caveats. LiteSpeed Web Server is compatible with Apache .htaccess syntax — including SetEnvIfNoCase and <RequireAll>. But OpenLiteSpeed needs a restart after editing .htaccess (unlike Apache which applies it immediately). For LiteSpeed Enterprise, .htaccess changes are detected immediately. If you use LiteSpeed, you can also use the built-in Rate Limiting feature which is more powerful — it can limit requests per IP per second.

    Q: How long after editing .htaccess until the effect is felt?

    Immediately. Apache reads the .htaccess file on every request — once the file is saved, the next request already uses the new rules. No Apache restart needed. But if youre using OPcache or another caching layer, it may take a few minutes for server resources to truly drop since some requests are still cached.

    Q: What if theres a new bot not on the list?

    Check the access log regularly — at least once a month. Run the Step 2 command to look for new suspicious user-agents. If there are any with thousands of requests and unknown user-agents, add them to the block list. You can also subscribe to the ai.robots.txt repository on GitHub which is updated regularly by the community with the latest AI bot lists.

    Conclusion

    AI training bots and aggressive crawlers are a new reality every sysadmin has to face. Theyre not dangerous in the sense of stealing data or crashing servers — but they drain resources until websites become inaccessible to real users. On this court server, 80% of traffic came from bots, and load average spiked 10x over just because of ClaudeBot requesting non-stop even after getting 403s.

    The most practical and fastest solution: block via .htaccess using SetEnvIfNoCase. The rules are simple, take effect immediately without a restart, and can be applied on any server running Apache or LiteSpeed. For additional protection, combine with Cloudflare Bot Fight Mode so bots are blocked before they even reach your server.

    The lesson from this case: dont trust that 403 means the problem is solved. A 403 response code still requires resources to process. If requests number in the thousands per minute, resources still get exhausted. Block at the earliest level — before Apache needs to spawn workers, before PHP needs to load, before the database needs to query. Thats why .htaccess at the very top is the most efficient solution.

    Author: NOC Engineer — written based on experience handling a court SIPP server paralyzed by AI bot crawling, July 2026.

    Artikel Terkait