Narwhal crawler

Purpose

Narwhal API runs a crawler that reads public pages of official statistics offices, central banks and regulators. We use them to build a searchable index and a database of published figures, and we link back to the source.

Identification

The crawler sends this user-agent: Mozilla/5.0 (compatible; NarwhalSearchBot/1.0; +https://crawler.narwhalapi.com)

Its robots.txt product token is NarwhalSearchBot.

Verification

The crawler signs its requests using HTTP Message Signatures (RFC 9421), following the Web Bot Auth scheme. A signed request carries Signature, Signature-Input and Signature-Agent headers. Its public Ed25519 keys are published at https://crawler.narwhalapi.com/.well-known/http-message-signatures-directory.

Respecting robots.txt

The crawler respects the Robots Exclusion Protocol. It follows the group addressed to NarwhalSearchBot. If you have not provided one, it follows the rules you have set for major search engines; otherwise it follows the * group.

Give the crawler its own rules (these take precedence):

User-agent: NarwhalSearchBot
Disallow: /private/

Block the crawler from your whole site:

User-agent: NarwhalSearchBot
Disallow: /

Limits

The crawler does not attempt to bypass logins, paywalls or CAPTCHAs, and it submits no forms.

Contact

If you have questions or need assistance, write to hello@narwhalapi.com.