Robots.txt Evaluation Methodology

1. Background & RFC 9309 Standardization

For nearly three decades following Martijn Koster’s original 1994 draft, the Robots Exclusion Protocol operated as an informal consensus. In September 2022, the IETF ratified RFC 9309 as the official Proposed Standard.

SEO Free implements the exact parsing and evaluation rules established in RFC 9309, guaranteeing consistent, predictable behavior matching Googlebot and Bingbot.

2. Crawler Group Selection & Precedence

When a crawler inspects a robots.txt file, it applies the following sequence:

Product Token Search: The parser searches for a group containing a User-agent header matching the crawler’s product token (case-insensitively, e.g., Googlebot).
Group Merging: If multiple groups target the same crawler token, all Allow and Disallow directives from those matching groups are combined into a single unified rule set.
Wildcard Fallback: If no group matches the specific crawler token, the crawler falls back to the generic wildcard group (User-agent: *). If multiple wildcard groups exist, they are merged.
Unconditional Allow: If no matching group and no wildcard group are defined, all URLs are considered ALLOWED.

3. Directive Specificity & Tie Resolution (Section 2.2.2)

Unlike firewall rules that evaluate on a first-match basis, RFC 9309 dictates that the most specific matching rule wins:

Octet Length: Specificity is determined by the length of the matching pattern string. The rule with the greatest number of matching characters (octets) takes precedence.
Tie-Breaking: If an Allow directive and a Disallow directive match the identical path with the exact same octet length, the Allow rule takes precedence.
Wildcards (*): The asterisk matches zero or more valid UTF-8 characters.
End Anchors ($): The dollar sign indicates the end of the URL path string.

4. Essential Disclaimers & Technical Boundaries

Important: Allowed by robots.txt does NOT guarantee crawling, indexing, or ranking. Blocking crawling is NOT the same as using a noindex directive.

Robots.txt controls crawl access to server resources; it does not control indexation. If a blocked URL has inbound links from other websites, search engines may still index the URL with a bare snippet based on anchor text.