You update robots.txt, but search crawlers continue following the old directives. Your sitemap.xml works at the origin, yet a webmaster tool reports a 403, 404, invalid response, or unreadable content. When a CDN sits in front of the site, changing the file on the server is only half of the fix: crawlers may still be receiving an older or incorrect response from an edge cache.

The fastest way to resolve the problem is to answer three questions in order: which layer returned the response, which version is cached, and why the request still matches that object after a purge. This guide walks through that process without relying on repeated full-cache purges or risky changes to origin access controls.

Symptoms that point to stale or incorrect edge content

A CDN-related problem often appears inconsistent because different edge locations can hold different objects. Common signs include:

• the origin file has changed, but the public hostname still returns the previous content;

• robots.txt or sitemap.xml differs between networks, regions, or repeated requests;

• the origin returns HTTP 200 while the CDN returns 403, 404, 429, or a 5xx response;

• the response briefly looks correct after a purge and then reverts to the old version;

• headers such as Age, Via, X-Cache, or CF-Cache-Status show that an intermediary handled the request;

• the response body is an HTML error page, login page, or WAF challenge even though the URL ends in .txt or .xml;

• Google Search Console or another crawler tool reports a fetch failure while a direct origin test succeeds.

None of these symptoms proves that the cache is the only cause. Search engines can keep their own copy of robots.txt, and the same behavior can also result from DNS changes, a WAF rule, a wrong origin Host header, redirects, or multiple origin servers with inconsistent files. The useful test is a side-by-side comparison of the edge response and the origin response.

Compare the CDN response with the origin

Start by saving the headers and body returned by the public hostname:

curl -sS -D /tmp/robots-edge.headers \
  -o /tmp/robots-edge.txt \
  https://example.com/robots.txt

curl -sS -D /tmp/sitemap-edge.headers \
  -o /tmp/sitemap-edge.xml \
  https://example.com/sitemap.xml

Check the status code, body, redirect chain, and these headers when present:

• Age, which indicates how long an object has remained in a shared cache;

• Cache-Control and Expires, which influence freshness and revalidation;

• ETag and Last-Modified, which help identify the version and conditional-request behavior;

• CDN-specific cache status headers such as X-Cache or CF-Cache-Status;

• Content-Type, which should match plain text for robots.txt and XML for a sitemap;

• Content-Encoding and Content-Length, which can reveal an unexpected body or truncation.

Next, test the origin while preserving the real hostname and TLS Server Name Indication. Replace the example IP with the actual origin address:

curl --resolve example.com:443:203.0.113.10 \
  -sS -D /tmp/robots-origin.headers \
  -o /tmp/robots-origin.txt \
  https://example.com/robots.txt

curl --resolve example.com:443:203.0.113.10 \
  -sS -D /tmp/sitemap-origin.headers \
  -o /tmp/sitemap-origin.xml \
  https://example.com/sitemap.xml

If the origin content is correct and the edge content is not, focus on the cache object, cache key, routing, and edge security rules. If both responses are wrong, fix the application route, generated file, or origin configuration before touching the CDN.

Do not expose a previously private origin to the entire internet just for this test. Run it from an allowed management address, use an internal network, or temporarily permit a narrowly scoped source and restore the original access rule afterward.

Why robots.txt failures behave differently from sitemap failures

robots.txt is a control file at the site root. Crawlers use it to decide which paths they may fetch. Google documents that it generally caches the file for up to about 24 hours, and that the cache period can be extended when network or server errors occur. A successful CDN purge therefore does not guarantee that Google's previously stored copy disappears immediately.

Status codes matter as much as the file contents. A successful 2xx response lets the crawler process the rules. Most 4xx responses are treated as if no robots.txt file exists, while 429 is handled as a server error. Network errors and 5xx responses can temporarily affect crawling and may cause a crawler to keep using a previously cached version. Do not use 403 or 5xx responses as a substitute for a valid disallow rule; return an accessible file with explicit directives instead.

A sitemap has a different purpose. It should return a stable 200 response, valid XML, and URLs that use the site's canonical host and protocol. A CDN can accidentally cache an authentication page, WAF challenge, redirect target, or HTML error document under the sitemap URL. In that case, the filename and status code alone are misleading. Inspect the first part of the body, XML declaration, encoding, Content-Type, and redirects.

CDN settings that commonly cause the problem

Path rules and rule priority

Look for behaviors matching /robots.txt, /sitemap.xml, and any sitemap index or child pattern such as /sitemap*.xml. A broad rule that caches every “static” file for several days may unintentionally include these files. In rule engines where priority matters, verify that a specific exception is not overridden by a later wildcard behavior.

If the site uses more than one CDN or reverse proxy, inspect every layer. Purging the outer service does not remove an old object from an upstream cache, and purging the upstream service does not force every downstream layer to revalidate.

Cache keys

The cache key should separate sites and request variants that can produce different content. Check whether it includes the Host header, protocol where relevant, query-string handling, and only the request headers that truly change the response.

A dangerous configuration is a shared distribution for several domains whose key does not distinguish the hostname. That can allow one site's file to be served to another site. The opposite problem also occurs: unnecessary query parameters or headers can create many variants, so a purge that targets one URL does not remove the version crawlers are requesting.

Error caching

CDNs may cache 403, 404, 429, or 5xx responses according to provider defaults or custom rules. A short origin outage can then survive at the edge after the origin is healthy. Check both the normal object TTL and the error-cache TTL. For crawler control files, avoid caching temporary server errors for long periods.

WAF and bot controls

A security rule may challenge or block crawler traffic while allowing your browser. Test the exact hostname and path with headers close to the failing request, then review WAF events rather than assuming every bot-labeled request is malicious.

Avoid a broad allow rule based only on User-Agent; that value is easy to forge. Prefer documented verified-bot signals from the provider, IP or reverse-DNS verification where appropriate, and path-specific policies with logging.

Origin Host, rewrites, and redirects

An incorrect origin Host header may send the request to a default virtual host. A path rewrite can turn /sitemap.xml into an application route, and a redirect rule can send the file to the homepage or another domain. Trace HTTP-to-HTTPS and apex-to-www redirects and make sure they do not loop, cross to the wrong language host, or return HTML in place of XML.

Purge the smallest safe set of URLs

Once the origin is correct and the problematic cache behavior is understood, purge the exact public URLs first:

https://example.com/robots.txt
https://example.com/sitemap.xml

If sitemap.xml is an index, purge the index and any changed child sitemap files. Use a full-site purge only when targeted invalidation is unavailable or evidence shows that the problem affects a much larger set of objects. Clearing the entire cache increases origin traffic and removes protection from popular content that was not involved in the incident.

After invalidation, request each file several times. The first request may be a miss or revalidation, followed by a hit on the new object. Compare the public response with the origin and calculate a content hash:

curl -sS https://example.com/robots.txt | sha256sum
curl -sS https://example.com/sitemap.xml | sha256sum

If the old content returns, do not keep issuing identical purges. Check whether the purge used the exact scheme and hostname, whether a query-string variant exists, whether another CDN layer still has the object, and whether the origin pool contains a server with the old file.

Set a cache policy that will not recreate the incident

There is no universal TTL for these files. A site that changes them rarely can tolerate a longer freshness period; a publishing platform that updates sitemaps throughout the day needs a shorter, controlled edge TTL. The important part is to align the policy with the update process.

A practical setup usually includes:

• a short and predictable shared-cache TTL for robots.txt and sitemap files;

• targeted invalidation triggered after a successful deployment or sitemap generation job;

• very short or disabled caching for temporary 403, 429, and 5xx responses on these paths;

• consistent Cache-Control headers at the origin and no conflicting CDN override;

• a cache key that separates hostnames without creating unnecessary variants;

• monitoring for status code, content type, file size, redirect target, and content hash.

“Never cache these files” is not automatically safer. Short TTLs plus reliable, targeted purges preserve the CDN's ability to absorb bursts while keeping updates predictable.

Verify the fix from more than one vantage point

Use a checklist rather than trusting one browser refresh:

1. Compare the edge and origin bodies and confirm that their hashes match.

2. Test from at least two networks or edge regions to rule out a single stale location.

3. Confirm the final status code, cache status, Age, Content-Type, and redirect chain.

4. Check that robots.txt does not accidentally block CSS, JavaScript, or sections that should be indexed.

5. Validate sitemap XML, canonical hostnames, URL entries, and last-modified values where used.

6. Recheck robots.txt or resubmit the sitemap in Google Search Console, then allow time for another crawl.

If the CDN consistently returns the new file but a search platform still displays the old result, the remaining delay may be the crawler's own cache or recrawl schedule. Repeated full-site purges will not accelerate that process and can create avoidable origin load.

Prevent the next crawler-control outage

Treat these files as part of the deployment pipeline rather than static files that happen to exist. After generating or publishing them, automatically request the production hostname and verify the status code, content type, redirect target, and expected hash. Stop the deployment or alert the operator when the edge response differs from the approved origin version.

Keep several recent file versions and a record of purge operations. When crawl activity changes, that history makes it easier to separate a content mistake from a stale edge object, a security rule, a routing change, or a search engine that has not fetched the new version yet.

The reliable fix is not simply “purge the CDN.” It is to identify the responding layer, correct the rule or cache key that created the wrong object, invalidate only the affected URLs, and verify the result at both the origin and the edge.