1. Introduction and relevance of the problem
Fake search engine bots are one of the most covert and difficult-to-detect threats to web resources. Attackers massively forge the User-Agent header (as well as related HTTP headers such as Accept-Language, Referer, and X-Forwarded-For), disguising their scripts as legitimate crawlers like Googlebot, YandexBot, or Bingbot. This is done to bypass heuristic protection systems, freely scan content, dump databases, and overload application servers.
Disguising as official search engines gives attackers several architectural advantages:
| Disguise objective | Description and technical vector |
|---|---|
| Bypassing protection (WAF/anti-DDoS) | Exploiting weaknesses in basic WAFs that use regular expressions (Regex) to add substrings like *Googlebot* to global allowlists (allowlist). |
| Content scraping | Multi-threaded harvesting of unique texts, dynamic prices, product catalogs without triggering rate-limit rules. |
| Exploiting vulnerabilities | Exploiting IDS/IPS blind spots: directory scanning, looking for exposed .git, .env, and SQLi under the guise of legitimate indexing. |
| Click fraud and DDoS load | Simulating L7 (application layer) traffic, causing exhaustion of the Nginx/Apache worker pool and connection pool exhaustion to the database. |
Simply checking the User-Agent string in logs is categorically insufficient. This header is forged on the client side by modifying a single byte in the script configuration (for example, via cURL, Puppeteer, or Selenium). Reliable validation requires deep verification of the bot's IP addresses against autonomous systems (ASN) and validation via Reverse DNS, reinforced by behavioral analysis.
2. Analysis of request statistics
The analysis was conducted for two sites of similar subject matter, operating on an identical technology stack:
- International — in the
.comzone - Russian-language — in the
.ruzone
Blocking of fake bots was performed using the server-side module HaltClick, which analyzes every incoming request across three protection layers, preventing illegitimate headless browsers from getting through (see section 5).
2.1 Site in the. com zone
Request data (for July 14, 2026):
| Search bot | Number of requests | Share of all search requests |
|---|---|---|
| Googlebot / GoogleOther / google | 566,041 | 67.5% |
| YandexBot / yandex | 92,913 | 11.1% |
| bingbot / BingPreview | 179,415 | 21.4% |
| Total | 838,369 | 100% |
Blocking data (for July 14, 2026):
| Search bot | Blocked requests | Block share | Fake percentage |
|---|---|---|---|
| Googlebot / GoogleOther / google | 76,699 | 91.8% | 13.5% |
| YandexBot / yandex | 603 | 0.7% | 0.6% |
| bingbot / BingPreview | 9 | 0.01% | 0.005% |
| Total | 77,311 | 100% | 9.2% |
Technical interpretation: On the .com site, 77,311 blocks of imitation scripts were recorded, accounting for 9.2% of the total request pool. The extremely high concentration of Googlebot forgeries (13.5%) is explained by the fact that for international platforms, scrapers more often use AWS, DigitalOcean, and Hetzner data centers. Disguising as Google (whose infrastructure is also globally distributed) raises less suspicion in primitive monitoring systems than a sudden surge of Bing or Yandex traffic from European or American hosting provider servers.
2.2 Site in the. ru zone
Request data (for July 14, 2026):
| Search bot | Number of requests | Share of all search requests |
|---|---|---|
| Googlebot / GoogleOther / google | 305,333 | 65.1% |
| YandexBot / yandex | 128,775 | 27.5% |
| bingbot / BingPreview | 34,591 | 7.4% |
| Total | 468,699 | 100% |
Blocking data (for July 14, 2026):
| Search bot | Blocked requests | Block share | Fake percentage |
|---|---|---|---|
| Googlebot / GoogleOther / google | 72,284 | 97.7% | 23.7% |
| YandexBot / yandex | 17 | 0.02% | 0.01% |
| bingbot / BingPreview | 1,671 | 2.3% | 4.8% |
| Total | 73,972 | 100% | 15.8% |
Technical interpretation: On the .ru site, the situation shows signs of targeted scraping. Of 73,972 blocked forgeries (15.8% of all requests), almost every fourth request (23.7%) masquerading as Googlebot turned out to be a malicious script. The anomaly with Bing forgeries (4.8% versus 0.005% on. com) is associated with the mass collection of Russian-language text corpora for training AI models: many new scrapers use default User-Agent libraries that imitate BingPreview for rendering JS pages.
3. Comparative and analytical breakdown
3.1 Summary table for both sites
| Metric | .com site | .ru site | Delta |
|---|---|---|---|
| Total requests from «search bots» | 838,369 | 468,699 | -44.1% |
| Total blocks | 77,311 | 73,972 | -4.3% |
| Overall fake percentage | 9.2% | 15.8% | +6.6 pp |
| Googlebot forgery | 13.5% | 23.7% | +10.2 pp |
| YandexBot forgery | 0.6% | 0.01% | -0.59 pp |
| Bingbot forgery | 0.005% | 4.8% | +4.795 pp |
3.2 Analytical conclusions and attack patterns
- Googlebot is a universal bypass key. On both sites, Googlebot impersonation dominates (from 13.5% to 23.7%). Mathematically, the probability that a request with
User-Agent: Googlebotturns out to be a scraper correlates with the complexity of the site structure: attackers know that servers are less likely to return «stubs» (403 Forbidden) to «Google» so as not to lose search rankings. This is confirmed by Search Engine Journal industry data: up to 87% of «gray» traffic masquerades as Google. - Regional vulnerability of the RU segment. Despite the smaller total traffic volume (1.8 times less than. com), the absolute number of parasitic requests on the
.rudomain is almost identical to the international site (73k vs. 77k). Attack intensity (share of fakes) here is 71% higher. This indicates aggressive competitive scraping within the local market (competitor price monitoring, contact database harvesting). - YandexBot specifics. The low level of forgeries (0.6% and 0.01%) has a clear technical basis. Yandex has an extremely aggressive anti-spoofing policy: Runet infrastructure often verifies Yandex subnets by default (via the Yandex API or PTR records of the format
*.yandex.ru,*.yandex.com). Attackers avoid this vector due to the high cost of bypassing ASN 13238 and ASN 208722 checks.
4. HaltClick verification technology — engineering breakdown
To detect and block advanced scrapers (for example, written in Playwright with stealth plugins), the HaltClick server-side module was used. Unlike cloud WAFs that analyze requests by signatures and classic captchas that work at the DOM level, HaltClick integrates into the request processing pipeline on the web server.
4.1 Three-tier heuristic architecture
| Layer | Technology | Technical implementation and metrics |
|---|---|---|
| 1 | Cryptographic tag (Signed Cookie) |
Validation of an AES-256 encrypted cookie. Checks session lifetime (TTL) and binding to the browser's IP/subnet. If the token is valid — O (1) processing time, the request passes to the backend. |
| 2 | Hardware fingerprint (Hardware Fingerprint) |
Collection of rendering data: Canvas API (difference in GPU subpixel rendering), WebGL vendor, available system fonts, structure of the navigator object. Headless browsers often have empty values or emulator signatures (e.g., SwiftShader). |
| 3 | JS Challenge and behavioral entropy | Execution of asynchronous JavaScript solving a mini-task (Proof-of-Work, hash generation). Analysis of mouse/touch event entropy. Bots that cannot execute JS (cURL, Python Requests) drop off instantly. |
4.2 Verification algorithm for real search engines
Legitimate bots (Google, Yandex, Bing, Apple, Baidu) cannot execute JS challenges. For them, HaltClick implements a network validation protocol out of the box:
IP & CIDR Matching: Checking the initiator's IP address against the officially published network ranges of search engines (updated dynamically).
If there is a mismatch (fake bot), the module immediately terminates the connection, returning HTTP 429 Too Many Requests.
The module is autonomous:
- Synchronization of signatures with the HaltClick master server — once every 10 minutes (consumes < 15 KB of traffic).
- Updating CIDR lists of search engines — once every 6 hours.
- If the communication channel with the cloud fails, the module switches to Fail-Open mode (uses a local cache), eliminating site downtime (Zero-Downtime architecture).
4.3 Comparison: HaltClick vs Cloud WAF & Captchas
| Criterion | Cloud WAF / Captchas | Server-side HaltClick |
|---|---|---|
| Latency | Adds 100 to 500 ms of DNS routing | Runs locally (in memory), overhead < 2 ms |
| Resistance to ML/AI | Neural networks easily recognize images (solve rate > 85%) | Imitating hardware Canvas GPU and entropy without a real device is extremely difficult |
| Vulnerability to IP bypass | If an attacker discovers the real server IP, the WAF can be bypassed | Installed directly on the server (Endpoint), cannot be bypassed |
| Impact on SEO/UX | Captchas «kill» conversion (drop rate up to 20%) | JS check is transparent (0-click), does not block indexing |
5. Threat vectors and infrastructure consequences
The scale of detected mimicry (almost 16% of parasitic traffic) creates cascading risks for IT infrastructure and business:
| Threat vector | Infrastructure and business consequences |
|---|---|
| L7 DDoS (Application Layer) | Fake bots ignore cache and hit heavy SQL queries (search, filters). This leads to exponential growth in server CPU utilization and out-of-memory (OOM-killer) conditions. |
| Scraping and parsing | Intellectual property theft, real-time price dumping by competitors. Bots instantly download new catalog items before the real Google indexes them. |
| Big Data distortion | Data Lake pollution: fake traffic breaks cohort analysis, conversion rate (CR) calculation, and A/B testing, as it generates millions of «abandoned carts» and zero-duration sessions. |
| SEO crawl budget penalization | A server overloaded with scrapers serves pages to the legitimate Googlebot with a delay > 3 seconds. The search engine reduces the crawling budget, so new pages stop being indexed. |
| Click fraud | Bots clicking on ad links from Yandex.Direct and Google Ads. CPA metrics degrade: budget is burned in hours without real leads. |
According to cybersecurity data, from May 2024 to May 2025 the use of distributed botnets (Residential Proxies) to disguise as search engines increased by 18%.
6. Implementation results and technical metrics
HaltClick deployment intercepted 151,283 malicious requests in total across the two sites in one day (77,311 on .com + 73,972 on .ru).
Financial and technical impact of the implementation:
DBMS utilization reduction
Database load (MySQL/PostgreSQL CPU) decreased by 25–35% by cutting parasitic requests to «heavy» filter and search pages.
Throughput cleanup
Traffic savings up to 40%, as the server stopped serving multi-megabyte HTML structures and images to scrapers.
TTFB acceleration
Response time for real users improved due to freeing up the PHP-FPM / Node.js worker pool.
Marketing optimization
Reducing the percentage of fraudulent traffic (click fraud) in ad campaigns saved up to 18% of the budget (before connecting specialized click-fraud protection).
7. Technical protection regulations and recommendations
Based on the analytical data obtained, bot protection should be built in several echelons.
7.1 Echelon 1: Basic network protection (mandatory minimum)
| Method | Implementation mechanics | Effectiveness |
|---|---|---|
| CIDR verification | Configuring Nginx (geo module) to check IP addresses against public JSON lists of Google, Bing, and Yandex subnets. |
High (against primitive attacks) |
| Strict Reverse DNS | Configuring PTR checks. Requires server resources to perform DNS queries (adds overhead, requires a local caching DNS such as bind or unbound). |
High |
| Rate limiting | Using limit_req in Nginx (for example, no more than 50 requests per second from one IP). |
Medium (bypassed via proxies) |
7.2 Echelon 2: Behavioral and heuristic filtering (recommended)
| Method | Implementation mechanics | Effectiveness |
|---|---|---|
| Server-side antibot (HaltClick) | Multi-factor analysis at the backend level: JS token injection, WebGL/Canvas verification, emulator elimination. | Absolute (High) |
| Navigation pattern analysis | Log analysis (ELK/ClickHouse): identifying sessions without requests to static files (.css,. js) and bots that ignore robots.txt directives. |
Medium (post-analysis) |
7.3 HaltClick integration (engineering step-by-step process)
HaltClick architecture allows integration without interfering with the application source code:
- Initialization — obtaining the distribution (Zip archive) with a unique configuration key from the dashboard.
- Deployment — unpacking to the server root directory (for example,
/var/www/site.ru/haltclick-xxxxxxxxxxx). - Permissions — setting permissions
755or775to allow local caching of current black/white lists by the update daemon. - Server-level routing — connecting via the
auto_prepend_filedirective in.user.ini(for PHP-FPM),.htaccessrules (for Apache), or as Middleware for Node.js/Python applications. - Testing (Dry Run) — checking the correctness of JS challenge delivery. The verification interface automatically adapts to the client's
Accept-Languageheaders (40+ languages supported). - Configuration tuning (
config.php) — setting endpoints excluded from verification (skipUrl, e.g., payment system webhooks/api/webhook), configuring scoring thresholds.
8. Conclusion: Architecture of Trust
Analysis of logs and blocks on two high-load sites proved the critical scale of the problem of scrapers mimicking search engines. Simply checking the User-Agent header today is equivalent to having no protection at all.
- On the international domain
.com, every 11th request from a supposedly search engine was a malicious scraper (9.2%). - On the Russian domain
.ru, the situation is even harsher: every 6th request was fake (15.8%). - The maximum impact falls on Googlebot — up to 23.7% of parasitic traffic on the. ru domain masqueraded as Googlebot, making it the most popular vector for bypassing WAF.
Key engineering takeaways:
- Basic User-Agent substring filtering in Nginx or outdated WAFs is completely obsolete. Attackers easily bypass them by exploiting blind trust in the keywords
GoogleandBing. - Implementing CAPTCHA causes irreparable damage to conversion (UX degradation), while modern headless bots solve captchas via farms or AI vision more successfully than humans.
- Server-side verification through hardware fingerprints (Canvas/WebGL) and ASN network analysis detects scrapers with 100% accuracy, as proven by blocking >150,000 fake requests per day.
- HaltClick as an Anti-Bot / L7 Mitigation class solution provides:
- User-invisible verification (0-click JS challenge).
- Strict blocking of distributed scrapers, emulators, and harvesters.
- SEO indexing safety (native support for official search engine subnets).
- Reduction in server CPU and DB load by 25–35%.
- Significant advertising budget savings (up to 18%) by cleaning traffic from click bots.
Resolution for IT departments: Immediate rejection of User-Agent heuristics in favor of cryptographic verification is required. Integration of specialized solutions such as HaltClick is a necessary security standard for protecting infrastructure from L7 attacks, preventing content theft, and ensuring the validity of marketing analytics.