Arsenly
Tools
Recon & Discovery16 tools Web Security Audit10 tools Exploitation & OOB3 tools Payloads & Wordlists4 tools Cryptography & Generators3 tools Encoders & Converters5 tools Reference & Reports4 tools
Browse the full catalogue Open /explore →
Free Tools
Recon & Discovery
SiteMapper ProTurn a URL list into 4 views: hierarchical host/path tree, sortable fl… URL & Endpoint ExtractorMine URLs, JS endpoints, domains and parameters from pasted source cod… CIDR / IP CalculatorIPv4 and IPv6 subnet math. Expand CIDRs, parse ranges and netmasks, ag…
Web Security Audit
Cookie / JWT AuditorAudit cookie flags (Secure, HttpOnly, SameSite, __Host- / __Secure- pr… OpenAPI / Swagger AnalyzerPaste or drop an OpenAPI 3 / Swagger 2 spec and map its attack surface…
Cryptography & Generators
Hash ToolkitGenerate (MD5, NTLM, SHA) and identify 60+ hash types with ranked cand… Test Data GeneratorRealistic-looking test data across 22 countries: names, emails, phone … Code / Password / Token GeneratorCustom-charset strings (1-512 chars, up to 5000 at a time), passwords …
Encoders & Converters
Encoder / Decoder4-tab workbench: 20+ encodings (URL, double-URL, percent-all, Base64, … HTTP Request ConverterConvert HTTP requests between 6 formats: Raw HTTP (Burp), curl, fetch(… JSON WorkbenchFormat, validate (with line / col error), minify, sort keys, unescape … Text Suite7-tab text workbench: regex find/replace, sort/dedupe/sample, keep-dro… Text Comparer4-tab diff workbench: true LCS line / word / char diff, side-by-side, …
See all 13 free tools on /explore →
Pricing
Resources
ChangelogWhat's shipped lately API DocsAutomate scans from your CLI About XowiaLabsWho's behind this ContactBug reports, ideas, just say hi Terms & PrivacyAcceptable-use policy + privacy
Sign in Get started
← All tools

Robots & Sitemap Harvester

Recon & Discovery

Harvest robots.txt, recurse gzipped sitemaps, pull 14 well-known files plus ads.txt, classify disclosed paths into 13 categories with uniqueness scoring, and auto-generate probe variants. Batch up to 3 domains.

What is Robots & Sitemap Harvester?

Robots & Sitemap Harvester fetches a target’s robots.txt and every linked sitemap, plus the public discovery files at /.well-known/ and the ad-network manifests at /ads.txt and /app-ads.txt. Ironically, the very robots file meant to hide directories from crawlers often points testers straight at admin panels, staging areas and hidden endpoints.

The tool parses robots.txt per user-agent so a path blocked only for Googlebot still surfaces, recurses sitemap index files into their child sitemaps with gzip support, extracts <lastmod> for freshness sorting, and groups thousands of sitemap URLs into endpoint patterns. The disclosed paths are classified into 13 high-signal categories and scored by uniqueness so common patterns (like /wp-admin/) sink below site-specific finds.

What it pulls

  • Batch up to 3 domains with IDN/unicode accepted.
  • Per-user-agent rules: Disallow / Allow / Crawl-delay grouped by the bot they target.
  • Nested sitemaps: follows sitemap-index files to child sitemaps, decompresses gzip automatically.
  • `` parsing with freshness-sort of sitemap URLs.
  • 14 well-known files: security.txt, Android Asset Links, Apple App Site Association, OpenID Connect discovery, OAuth Authorization Server metadata, host-meta (XRD + JSON), change-password (RFC 8615), DNT policy, NodeInfo, WebFinger, JWKS keys, OpenID Credential Issuer.
  • ads.txt / app-ads.txt with top-vendor breakdown.
  • Interesting-path classifier: 13 categories (Source control, Secrets/config, Backups, Database tools, Admin, Auth, API, Internal, Upload, Logs, Dev/staging, API docs, Includes, CMS internals) with high/medium/low severity.
  • Uniqueness scoring: common disclosed paths (`/wp-admin/`, `/api/`, `/cdn-cgi/`) get a `common` flag and rank below site-specific finds.
  • Probe variants: each disclosed path generates 5-8 probe URLs (with and without trailing slash, common index files, `.bak` / `.old` / `~` suffixes).
  • Endpoint pattern grouping: 5,000 sitemap URLs collapse to a few hundred shapes like `/users/{id}` with counts.
  • security.txt RFC 9116 validation: checks for Contact, Expires, stale-file detection.
  • Category and source chip filtering: slice by category (Admin / Auth / Secrets) or source (robots vs sitemap).
  • Per-row actions: copy probe URL, archive replay, send single path to HTTP ProbeMaster.
  • Continue-recon handoff IN/OUT to CNAME, DNS Recon, Subdomain, ProbeMaster, SiteMapper, Wayback, URL Extractor, Security Headers, Vulnerability Scanner.
  • TXT, JSON and Markdown export.

Where it fits in your workflow

  • →Discover admin, source-control and staging paths the site discloses in robots.txt - uncommon ones surface first.
  • →Find OIDC / OAuth / mobile-app deep-link configuration at `/.well-known/`.
  • →See which 3rd-party ad networks the site uses via ads.txt vendor breakdown.
  • →Seed content discovery with the site’s own (often gzipped) sitemaps, sorted newest-first by lastmod.
  • →Hand probe-variants of the flagged paths to HTTP ProbeMaster to check which actually resolve live.
Use Robots & Sitemap Harvester

Run it from your dashboard.

Create free account Sign in Use via API

At a glance

CategoryRecon & Discovery
RunsServer-side
Token cost 3 / run (free tier)
Access Free
Status● Live

Frequently asked questions

Is reading robots.txt allowed?

robots.txt, sitemaps, /.well-known/ files and ads.txt are public files intended to be read by any client. Acting on what you find still requires authorisation for the target.

Does it test whether disclosed paths exist live?

No. The tool only reads the public discovery files and classifies what they reveal. The flagged paths are unverified leads. Use the Copy probe list / Copy probe variants buttons and HTTP ProbeMaster to check which are actually live.

What does the uniqueness score do?

Common disclosed paths like `/wp-admin/`, `/api/` and `/cdn-cgi/` are present on most sites and add noise to the report. Paths matching a curated common-list are flagged `common` and sorted below site-specific finds, so the gold floats to the top.

How does the lastmod sort work?

When a sitemap entry includes a `` element, it is parsed and used to sort the URLs newest-first. URLs without lastmod fall to the end. The date is displayed in the URL row.

Explore more tools →