oanor
Sign up free
Marketplace Pricing Features Sign in

API · /robots-api

robots.txt API

healthy 4,091 Subscribers

Fetch and evaluate any website's robots.txt. Pass a URL and a user-agent and the check endpoint tells you whether that URL is crawlable — selecting the most-specific user-agent group and applying the RFC 9309 longest-match Allow/Disallow rules (with * and $ wildcards, where Allow wins ties), and returning the matched rule, the group's crawl-delay and the sitemaps the site declares. The parse endpoint returns the whole file structured into per-user-agent groups (their allow and disallow lists and crawl-delay) plus the list of sitemaps. A missing robots.txt (404/403) means everything is allowed, exactly as the spec requires. The request is made server-side and private or internal targets are refused (SSRF-guarded). Built for SEO audits, crawler and scraper compliance, sitemap discovery and pre-flight "am I allowed to fetch this?" checks. A robots.txt evaluator — distinct from the on-page SEO audit (seo), the XML toolkit (xml) and link unfurling/preview (url). No upstream key, no cache.

#robots-txt #seo #crawler #rfc-9116 #web #sitemap
api.oanor.com/robots-api
Get an API key Try in playground → Contact provider

Machine-readable spec so AI agents can integrate this API.

/api/robots-api/openapi.json
/api/robots-api/llms.txt

Discovery: GET /api/index.json lists every API.

API health

healthy
Uptime
100.00%
Server probes · 24h
Avg latency
129 ms
Server probes · 24h
Subscribers
4,091
active
Total calls
0
last 7 days
status Full status page → · 3 probes/24h

Pricing

Pick a tier — billed monthly, cancel anytime.

Free

Free

  • 2,000 calls / month
  • 2 requests / second
  • Hard cap (429 above quota, no overage)
  • 2,000 calls/month
  • 2 req/sec
  • Crawlability + parse
  • No credit card
Sign in to subscribe

Starter

€5.90 /month

  • 37,500 calls / month
  • 8 requests / second
  • Hard cap (429 above quota, no overage)
  • 37.5k calls/month
  • 8 req/sec
  • RFC 9309 match + sitemaps
  • Email support
Sign in to subscribe

Pro

€19.70 /month

  • 202,000 calls / month
  • 20 requests / second
  • Hard cap (429 above quota, no overage)
  • 202k calls/month
  • 20 req/sec
  • Crawler & SEO pipelines
  • Priority support
Sign in to subscribe

Mega

€51.60 /month

  • 785,000 calls / month
  • 50 requests / second
  • Hard cap (429 above quota, no overage)
  • 785k calls/month
  • 50 req/sec
  • Crawl-platform scale
  • Dedicated SLA
Sign in to subscribe

Built by

P
PremiumApi

Related APIs

Other APIs with overlapping tags.

Sitemap API

Fetch and parse an XML sitemap (the sitemaps.org protocol). Pass a sitemap URL and the parse endpoint fetches it — following redirects and transparently gunzipping .gz sitemaps — and returns its type: a urlset with every URL and its lastmod, changefreq and priority, or a sitemapindex listing the child sitemaps, with offset/limit paging for large files. The urls endpoint goes further: when the sitemap is an index it fetches the child sitemaps too and flattens every page URL into a single list, with a configurable cap on URLs and child sitemaps and a truncated flag so you stay in control. The request is made server-side and private or internal targets are refused (SSRF-guarded). Built for SEO audits, building crawl queues and content inventories, change monitoring and migration checks. A sitemap fetcher and parser — distinct from generic XML-to-JSON conversion (xml), the robots.txt evaluator (robots) and the on-page SEO audit (seo). No upstream key, no cache.

api.oanor.com/sitemap-api

Readability API

Score how easy a piece of text is to read using the standard, peer-reviewed readability formulas — Flesch Reading Ease, Flesch-Kincaid Grade, Gunning Fog, SMOG, Coleman-Liau and the Automated Readability Index. Pass text and get all six scores back together with the underlying counts (words, sentences, syllables, complex and polysyllabic words, letters and characters), an averaged grade level, an estimated reading time and a plain-English interpretation of the reading ease. A second endpoint counts syllables for a word or for every word in a phrase. Supply text inline via ?text=, as a query parameter or in a request body; everything is computed locally with no network calls, so it is fast and deterministic. Built for content and copywriting tools, SEO and editorial workflows, education and accessibility (plain-language) checks, and UX-writing review. A readability scorer — distinct from sentiment/NLP analysis (nlp), spelling and grammar checking (grammar), the case and text utilities (text) and string similarity (similarity). No upstream key, no cache.

api.oanor.com/readability-api

SEO API

Run an on-page SEO audit on any URL — title and meta-description analysis, heading structure, content length, image alt coverage, canonical / viewport / Open Graph checks and an internal/external link breakdown — with an overall 0–100 score. Plus a dedicated link analysis endpoint.

api.oanor.com/seo-api

security.txt API

Fetch and parse any domain's RFC 9116 security.txt — the machine-readable file at /.well-known/security.txt that tells security researchers how to report vulnerabilities. Pass a domain and the service locates the file (the canonical .well-known path with a legacy root fallback), parses every field — Contact, Expires, Encryption, Acknowledgments, Preferred-Languages, Canonical, Policy, Hiring and CSAF — and reports whether it is valid (has at least one Contact and a single, non-expired Expires), whether it is PGP-signed, whether it has expired (with the number of days remaining) and a list of issues with concrete advice. A companion endpoint returns the raw file. The request is made server-side; private and internal targets are refused (SSRF-guarded). Built for security audits, vendor and third-party risk assessment, attack-surface reviews and vulnerability-disclosure-policy compliance checks. A security.txt parser and validator — distinct from the HTTP security-header grader (secheaders), the SSL/TLS certificate check (sslcheck) and host reachability (hostcheck). No upstream key, no cache.

api.oanor.com/securitytxt-api

Frequently asked questions

Quick answers about pricing, quotas, and integration.

How do I get an API key for robots.txt API?
Sign up for free at oanor.com, generate an API key from the developer dashboard, and call robots.txt API with the x-oanor-key header. No credit card needed for the free tier.
What's the rate limit for robots.txt API?
Free tier allows 1 request per second. Paid plans scale up to 50 requests per second on the Mega tier. Hard limits return HTTP 429 above the quota — no surprise overage charges.
How much does robots.txt API cost?
robots.txt API has a free tier with 100 calls / month. Paid plans start at €5.90 / month with higher quotas and faster rate limits.
Can I cancel my subscription anytime?
Yes. Plans are billed monthly and you can cancel anytime from your billing dashboard. No long-term contracts and no cancellation fee.
Is robots.txt API GDPR-compliant?
All requests to robots.txt API go through our EU-based gateway. Your upstream API key never leaves our server and no personal data is shared with the upstream provider beyond the request you send.

Pick an endpoint from the list on the left to see its details and try it.

Code snippets

Sign up to get an API key, then call any path under your slug.

curl https://api.oanor.com/robots-api/SOME_PATH \
  -H "x-oanor-key: oanor_test_..."
const res = await fetch("https://api.oanor.com/robots-api/SOME_PATH", {
  headers: { "x-oanor-key": "oanor_test_..." }
});
const data = await res.json();
$ch = curl_init("https://api.oanor.com/robots-api/SOME_PATH");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, ["x-oanor-key: oanor_test_..."]);
$response = curl_exec($ch);
import requests
r = requests.get(
    "https://api.oanor.com/robots-api/SOME_PATH",
    headers={"x-oanor-key": "oanor_test_..."},
)
print(r.json())

Ratings

Sign in to rate.

No reviews yet.

Discussion

Ask questions, share usage tips, get answers from the provider and other developers. Public — anyone can read.

Sign in to start a thread or reply.

Sign in

New thread

/ 4000

📌 Pinned 🔒 Locked

·

· ·

/ 4000

🔒 This thread is locked — no new replies.

  • No threads yet — start the discussion.

Support

Private 1:1 support with the provider — billing questions, integration issues, account problems. Only you and the provider team can see these threads.

Sign in to open a support ticket.

Sign in

Open new ticket

Describe what you need help with. The provider team gets an email and replies on the ticket page.

  • No tickets yet for this API.

Subscription active — calls can start immediately.

Send your first request —

Subscription active — copy a snippet and fire off your first call.