Skip to main content

Allow TrustedSite scanners through bot protection with Web Bot Auth

If your website is blocking our scanners, turn on Web Bot Auth for improved scan performance.

Written by Nick Hemenway

Some hosts and bot-protection layers block automated traffic by default, which can cause TrustedSite to miss pages or fail scans. Web Bot Auth is an optional setting that lets our scanners cryptographically sign their requests so your host can verify they came from TrustedSite and let them through.

Configure Web Bot Auth in TrustedSite

  1. Sign in to your TrustedSite dashboard.

  2. Open your site and go to Set Up → Advanced.

  3. Paste the three values from your host or CDN into the matching fields:

    • Signature

    • Signature-Input

    • Signature-Agent

  4. Save.

Allow our signed requests on your host

Shopify

In your Shopify admin:

  1. Go to Online Store → Preferences (https://admin.shopify.com/store/YOUR-STORE-ID/online_store/preferences).

  2. Under Crawler access, click Create signature.

  3. Fill in the form:

    • Signature name: TrustedSite

    • Domain: leave the default

    • Expires in: 90 days

  4. Copy the Signature, Signature-Input, and Signature-Agent values.

  5. Paste each into the matching field in your TrustedSite dashboard at Set Up → Advanced, then save.

For more on Shopify's verified crawler setup, see Crawling your store.

Other platforms

If your host or CDN supports web-bot-auth (Cloudflare, Akamai, and others are adding support), follow their documentation for verified bots.

When to turn this on

The configuration applies to three products:

Turn on web-bot-auth if:

  • Your certification scan is failing or skipping pages.

  • Diagnostics reports fewer pages than your site actually has.

  • Search Submission can't generate a complete sitemap.

  • You use Shopify, Cloudflare bot management, or another platform that supports the web-bot-auth standard.

If your scans already complete cleanly, you don't need this.

Did this answer your question?