Website protection check: antibot, WAF and captcha

Check a site before you start a scraper or buy proxies. The service opens it like a browser and shows which antibot, WAF or captcha the site runs and whether it serves the page to a request without a browser.

For example:

Why check a site before scraping

A site’s protection decides how to scrape it and which proxies to take. The check takes a few seconds and costs nothing.

A website window behind layers of protection: some requests get through, others break against the antibot

  1. Pick the tool

    The site serves the page right away: curl, requests or Scrapy through proxies will do. The site demands JavaScript: plan for a browser or an antidetect from day one, or your scraper will hit the check.

  2. Pick the proxies

    DataDome, Akamai, PerimeterX and Qrator weigh IP reputation and request rate. For such sites take a pool with thousands of addresses and rotation, so each IP carries few requests.

  3. Test your own proxies

    After you buy, run the checker in target mode against the same site. It opens the site through every proxy and shows how many get through and how many hit a captcha.

    Proxy checker for LinuxProxy checker for Windows

What the check detects

Systems are recognised by response headers, cookies and page code, and CDNs and DDoS protection also by the site DNS and network (ASN). We extend the list as sites change their protection.

Antibot

  • Cloudflare Challenge
  • Cloudflare Bot Management
  • Akamai Bot Manager
  • DataDome
  • HUMAN (PerimeterX)
  • Kasada
  • Radware Bot Manager
  • Netacea
  • Variti
  • ServicePipe
  • Ozon Antibot
  • Wildberries Antibot
  • Avito Firewall
  • Imperva Advanced Bot Protection
  • Anubis
  • Imunify360
  • Fingerprint (FingerprintJS)
  • ThreatMetrix

WAF and DDoS protection

  • Imperva (Incapsula)
  • AWS WAF
  • F5 BIG-IP ASM
  • Reblaze
  • Sucuri
  • Vercel Firewall
  • DDoS-Guard
  • Qrator (Curator)
  • StormWall
  • Barracuda WAF
  • Citrix NetScaler
  • Fortinet FortiWeb
  • ModSecurity
  • Wordfence
  • BitNinja

Captcha

  • Cloudflare Turnstile
  • Yandex SmartCaptcha
  • Google reCAPTCHA
  • hCaptcha
  • GeeTest
  • Arkose Labs (FunCaptcha)
  • Friendly Captcha
  • MTCaptcha
  • ALTCHA
  • Prosopo Procaptcha
  • Tencent Captcha
  • Alibaba Cloud Captcha

CDN

  • Cloudflare
  • Akamai
  • Amazon CloudFront
  • Fastly
  • Gcore
  • Bunny CDN
  • CDN77
  • Azure Front Door
  • Google Cloud CDN
  • KeyCDN
  • Edgio (Edgecast, Limelight)
  • Alibaba Cloud CDN
  • Tencent EdgeOne
  • Vercel
  • Netlify
  • CDNvideo
  • NGENIX

How to read the result

The site served the page

Code 200 and a normal page. An antibot can sit here too: it watches the request rate and turns on a check later.

Browser check

Instead of the page came a script that tests the browser. Without running JavaScript you cannot go further.

Captcha

The site asks to solve a captcha. This usually happens when the IP has a poor reputation or there are too many requests.

Refusal

Code 403, 429 or 503 without a check page. The site is closed to such requests or to this network.

Questions about the protection check

01How does the service detect protection?
The server opens the address like a Chrome browser and looks at the response: headers, cookies and page code. Every protection system leaves its own traces, such as the __cf_bm cookie of Cloudflare or _abck of Akamai, and the check reads them.
02Why does my scraper see something else?
We send a single request from Cloudflare addresses. Antibots weigh the IP, the request rate and behaviour, so your scraper with a hundred requests a minute can get a captcha where our single request passed.
03What is a browser check?
A page with a script that tests whether a real browser opened it: it runs JavaScript, reads the fingerprint and sometimes asks you to wait. curl and requests cannot pass such a page.
04Do proxies help against an antibot?
Proxies give requests different IPs, and the antibot turns on checks less often. If the site demands JavaScript, proxies alone are not enough: you need a browser or an antidetect, with the proxies plugged into it.
05Does the service store the addresses I check?
No. The server opens the address once, returns the result to your browser and keeps neither the address nor the site response.

Other utilities

My IPYour IP, ASN, blocklists and WebRTC leakProxy checker for LinuxBash script: checks a list in hundreds of threads, RBL and sitesProxy checker for WindowsPowerShell script: checks a list in hundreds of threads, RBL and sites

Proxies for protected sites

MIX pools of up to 25,000 IPs with addresses from different sources, unlimited traffic, HTTP and SOCKS5 on one port. A test hour costs $1.

See packages
LOGIN/REGISTER