CanuteTheGreat/security-txt-auditor
Watch
1
0
Fork
You've already forked security-txt-auditor
 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions
Audit robots.txt and security.txt (RFC 9116) for missing, expired or misconfigured fields (MIT, free; pipx install) https://errant.solutions/tools/security-txt-auditor.html
24 commits 2 branches 4 tags 170 KiB
  • Python 86.2%
  • Shell 13.8%
Find a file
Exact
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ronald Farrer f3ff5d4a25
All checks were successful
tests / test (3.14) (push) Successful in 1m2s
Details
tests / test (3.7) (push) Successful in 52s
Details
ci: pin pytest<9 for mutmut 3.8.0 (pytest 9 removed Item._nodeid: every nightly mutation job died at stats collection)
2026-11-06 09:15:31 -08:00
.forgejo/workflows mutation/chaos CI on every rw-man host again (owner decision 2026-10-18: colo backlog too big, GuardDuty suppression kept): EC2 guards are opt-in (refuse only with CI_COLO_ONLY=1, code kept), colo jobs move to the shared labels mutation-x86_64 / mutation-any (rw-man-01..05, 6g slots) 2026-10-18 13:45:42 -07:00
ci ci: pin pytest<9 for mutmut 3.8.0 (pytest 9 removed Item._nodeid: every nightly mutation job died at stats collection) 2026-11-06 09:15:31 -08:00
tests Expires parser pinned (offsets, leap second, lowercase t/z, fractions, invalid forms; mutation survivors); mutation floor 82 (measured 82.59%, nightly run 3701) 2026-10-17 00:39:32 -07:00
.gitignore nightly mutation: incremental (only functions changed since the last green run) except a weekly full run on the repo's own weekday; replace-queued concurrency; staggered cron 2026-10-18 12:22:54 -07:00
CHANGELOG.md Release 1.2.1: the test-depth fixes since 1.2.0 (see CHANGELOG) 2026-10-17 01:50:09 -07:00
claims.json Test depth: accuracy suite, nightly mutation + full Python matrix + install-channel checks, claims ledger; fixes found by it 2026-10-16 21:42:14 -07:00
LICENSE security-txt-auditor: free robots.txt + security.txt (RFC 9116) misconfiguration checker 2026-09-16 09:02:23 -07:00
pyproject.toml Mutation testing: copy all of ci/ into mutmut's sandbox (also_copy of ci/check_claims.py failed: no mutants/ci dir), keep real sleeps in fixture-server threads (--timeout test), per-repo mutant parallelism (ci/mutation-children) 2026-10-16 23:56:23 -07:00
README.md Release 1.2.1: the test-depth fixes since 1.2.0 (see CHANGELOG) 2026-10-17 01:50:09 -07:00
security_txt_auditor.py Release 1.2.1: the test-depth fixes since 1.2.0 (see CHANGELOG) 2026-10-17 01:50:09 -07:00

README.md

security-txt-auditor

Free, zero-dependency (Python stdlib only) tool from Errant Solutions.

Audits a domain's robots.txt and security.txt (RFC 9116) for common misconfigurations:

robots.txt

  • missing entirely
  • present but empty
  • references a Sitemap: that 404s
  • accidentally contains Disallow: / (blocks all crawlers site-wide — often a leftover staging config)

security.txt (checked at /.well-known/security.txt, the RFC 9116 location, falling back to the legacy /security.txt)

  • missing entirely (WARN: no disclosed vulnerability-reporting contact)
  • served as something other than text/plain, or without charset=utf-8
  • missing Contact:, or values that aren't mailto: / https: / tel: URIs, plus a note when contact is email-only or web-only
  • missing Expires:, repeated, not an RFC 3339 date-time with a timezone (e.g. 2027-01-01T00:00:00Z), already passed, or more than a year out (RFC 9116 recommends under a year)
  • Canonical: missing (informational) or not matching the URL the file was served from
  • Policy: / Encryption: / Acknowledgments: / Hiring: / Canonical: links that are http:// or don't answer 200 (--no-url-checks skips fetching them)
  • an OpenPGP cleartext signature is detected and the signed part is what gets parsed (the signature itself isn't verified); unsigned files get an informational note
  • unknown fields, and a file only found at the legacy location (informational)

Install

pipx install git+https://git.canutethegreat.com/CanuteTheGreat/security-txt-auditor.git        # latest release branch; puts `security-txt-auditor` on PATH
pipx install git+https://git.canutethegreat.com/CanuteTheGreat/security-txt-auditor.git@v1.2.1  # a pinned release
brew install canutethegreat/errant-tools/security-txt-auditor   # after: brew tap canutethegreat/errant-tools https://git.canutethegreat.com/CanuteTheGreat/homebrew-errant-tools
scoop install errant-tools/security-txt-auditor                 # after: scoop bucket add errant-tools https://git.canutethegreat.com/CanuteTheGreat/scoop-errant-tools

Or download the single file (stdlib-only Python 3.7+) and run it directly: curl -fsSLO https://errant.solutions/downloads/security_txt_auditor.py && python3 security_txt_auditor.py --help. Every release has a changelog, the wheel, the sdist and a SHA256SUMS file. Not on PyPI yet, so plain pipx install security-txt-auditor doesn't work until it's published there.

Usage

security-txt-auditor example.com [example2.com ...]
security-txt-auditor --json example.com           # machine-readable
security-txt-auditor --fail-on info example.com   # strictest CI gate

Python 3.7+. Exit codes (same convention as every Errant Solutions tool):

Code Meaning
0 nothing at or above --fail-on (default warn; info and never also accepted)
1 at least one finding at or above --fail-on
2 a target couldn't be reached (and nothing else failed the gate), or bad usage

No API keys, no signup, no third-party services involved: it only talks to the target's own web server. Source is MIT-licensed; read it before you run it against anything you don't own.

Pay what you want

If this saved you time: https://errant.solutions (payment link on the site).