Five Signals That Tell You Whether a Domain Is Ready for Machines

The problem behind the tool

42% of 400 domains in my August 2026 batch returned zero valid JSON-LD. 82% served no llms.txt file. These are domains from finance, healthcare, government, and e-commerce. The regulated industries. The ones building AI strategies with seven-figure budgets.

Every retrieval pipeline assumes the source delivers structured, machine-readable data. I stopped assuming two years ago. Since then I measure every source before it enters any pipeline. The gap between assumption and measurement is where extraction costs explode and hallucinations originate.

The five signals

1. JSON-LD. The homepage gets one GET request. Every <script type="application/ld+json"> block goes through json.loads. If the JSON parses and carries a valid @type, the check passes.

168 of those 400 domains had a JSON-LD block in the HTML. The block existed. It contained empty types, broken nesting, truncated strings. A malformed block is the more dangerous case: your pipeline sees data, tries to parse it, and extracts garbage.

2. llms.txt. The llms.txt standard (proposed by Jeremy Howard) gives language models a machine-readable summary of a website. Adoption sits at 18% across my 559 scanned domains. Two guard clauses catch false positives: a content-type check filters out HTML error pages served with a 200 status, and a prefix check catches soft-404 pages that return an HTML document at the /llms.txt path.

3. robots.txt. Two questions matter. Can a generic bot crawl the site at all? And does the site explicitly shut the door on known AI crawlers like GPTBot, ClaudeBot, or PerplexityBot? The tool parses every User-agent / Disallow pair and matches against a list of 10 known AI bot identifiers.

4. HTTP security headers. A domain that ships Strict-Transport-Security and X-Content-Type-Options runs a maintained stack. I track 10 sector lists. The pattern repeats in every single one: domains that fail on structured data also fail on security headers. Two of four checked headers must be present.

5. SSL/TLS. The shortest check. If the HTTPS handshake breaks, everything else is academic.

The verdict logic

A domain passes when at least 3 of 5 checks succeed. The threshold lives in a constant at the top of the script. I chose 3 because SSL alone accounts for one near-universal pass, and robots.txt presence is common enough to give most domains a second free point. The third signal separates domains that actively provide machine-readable infrastructure from those that exist on the web by accident.

What this tool covers and where it stops

Five binary signals. One question: can an automated system extract structured data from this source?

This is a pre-flight check. Content quality, factual accuracy, and freshness sit on a higher layer. This layer sits beneath all of them. If a domain fails here, your pipeline will burn tokens on extraction and get noise in return.

My full scanner covers 183 signals across 21 audit clusters: content quality, machine provisioning, infrastructure hardening, authority indicators, and anchoring. The pre-flight tool covers 5 of those 183. It tells you whether the door is open. What sits behind it requires the full scan.

Running it

The tool is a single Python file with one dependency (requests). It accepts single domains, multiple domains on the command line, or a file with one domain per line. Output goes to the terminal, to CSV, or to JSON.

# Single domain
python infra-check.py example.com

# Batch with CSV export
python infra-check.py --file domains.txt --csv results.csv

# JSON export for pipeline integration
python infra-check.py --file domains.txt --json-out results.json

# Quiet mode (verdict lines only)
python infra-check.py --file domains.txt -q --csv results.csv

The user-agent string identifies itself honestly as InfraCheck/1.0. The script waits 1 second between domains, follows redirects, and respects a 15-second timeout.

Sample output

Checking 3 domain(s)...

❌  example-bank.de  [FAIL]  (2/5 checks passed)
   JSON-LD:     missing
   llms.txt:    missing
   robots.txt:  open
   Headers:     HTTP 200, security headers: yes
   SSL/TLS:     valid

✅  example-saas.com  [PASS]  (4/5 checks passed)
   JSON-LD:     valid (Organization, WebSite)
   llms.txt:    found (2340 bytes)
   robots.txt:  open, blocks 2 AI bots
   Headers:     HTTP 200, security headers: yes
   SSL/TLS:     valid

❌  example-gov.de  [FAIL]  (1/5 checks passed)
   JSON-LD:     missing
   llms.txt:    missing
   robots.txt:  blocks all bots
   Headers:     HTTP 200, security headers: incomplete
   SSL/TLS:     valid

==================================================
Total: 3 | Passed: 1 | Failed: 2

What I learned from running this at scale

I ran the full batch twice: June 2026 and August 2026. The composite score across all 559 domains dropped from 45.3 to 44.0. The web got worse for machines in two months.

The five pre-flight signals track this decline. Domains that passed 4 or 5 checks in June now pass 3. The pattern concentrates in government portals and automotive. B2B e-commerce moved in the opposite direction: the only sector list that improved.

The pre-flight tool gives you the binary answer fast. If you need the full picture, with weighted scores, sector benchmarks, and signed attestations, that is what I built SOVP for.

Get the code

The complete infra-check.py with CLI argument parsing, CSV/JSON export, batch processing, and deduplication:

👉 github.com/litzki-systems/infra-check

MIT license. One file. Zero configuration.

One question

You feed domains into your pipeline. How many of them pass all five checks?

Run the script. Look at the CSV. That number is the actual size of your usable source list.