Product Page Unavailable in Merchant Center: A Decision Tree for 404s, Bot Blocks and Redirect Chains

"Product page unavailable" is one of the more expensive messages Merchant Center prints, because it does not warn. Google disables the affected products, and they stop appearing in ads and free listings until its crawler can reach the page again (Google Merchant Center Help). One failed crawl on the mobile or desktop page disapproves the item on both devices, and once you fix it, the product can take up to 48 hours to come back (Google, Product crawl issues).

This guide is for feed managers, agency leads and developers who find landing-page errors in the Needs attention tab. By the end you will be able to sort the failure into one of six causes (the feed, robots.txt, a firewall or CDN rule, a redirect chain, the server, or Google itself) and fix it at the source.

What each landing-page issue name actually means

Google reports one underlying problem, "we could not fetch your product page", under several names. The table maps each to its API code and the first thing to check.

Issue codes come from the Content API severity mapping; names come from Google's help pages for product page unavailable, unable to check product pages and StoreBot access. The last row is account-level, not product-level.

First, count how many products are affected

Before opening any URL, check what share of the catalog is affected. The ratio tells you where to look.

• Nearly every product: something site-wide (robots.txt, a CDN or firewall rule, DNS, an outage) is blocking Google. Start with branches B, C and E.
• A clean subset (one category, brand or URL pattern): the feed is producing bad URLs. Start with branches A and D.
• A small, scattered share with no pattern, and the pages load fine: Google's crawl may simply have failed. FeedArmy's Emmanuel Flossie treats roughly 10% of products as the sign of a Google-side cause and all products as the sign of a website problem (PPC Land, March 2026). That is one practitioner's observation, not a Google benchmark, so use it as a starting hypothesis and not as a verdict.

Download the affected products as a CSV from the Needs attention tab (Google) and sort it by URL path; the pattern usually shows quickly.

The decision tree: six causes, in the order to test them

Run one affected URL through the URL Inspection tool, choosing smartphone or desktop as the fetch type to match the error, which is the test Google itself recommends (Google). Then follow the first branch that matches.

A. The URL in the feed is wrong (404 or invalid URL)

If the inspection returns 404, compare the feed URL with the live one. Usual causes: a changed slug, a staging domain, a missing locale prefix, or unencoded characters. Google requires URLs to start with http:// or https://, to use your verified domain, and to encode special characters, giving the example of a comma becoming %2C (Google). The link attribute must follow RFC 2396 or RFC 1738 encoding (Google, Product data specification).

This bites Turkish and Arabic stores whose platforms build slugs from product names. Illustrative rows, constructed for this guide:

Encode in a feed rule rather than by hand, and encode once. Double encoding turns %C4 into %25C4, a fresh 404. Fix a few products with Edit product; fix many by correcting link and mobile_link at the source and re-uploading (Google).

B. robots.txt is blocking Google, or cannot be reached

If the inspection reports the page as blocked, open /robots.txt. Google's fix is to let both its landing-page crawler and its image crawler fetch the whole site (Google):

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

Three details catch people out:

• AdsBot ignores the wildcard. AdsBot-Google and AdsBot-Google-Mobile ignore the User-agent: * group, so rules for them must name them (Google, special-case crawlers). Google uses AdsBot-Google for ads and Googlebot for organic, and blocking either can cause problems (Google).
• Storebot-Google has its own token. Its rules affect all Google Shopping surfaces (Google, common crawlers), so a Disallow: / added to save crawl budget shuts Storebot out of every product page.
• An unreachable robots.txt is a failure too. "Can't access your robots.txt file" is its own cause of product page unavailable (Google).

If you changed robots.txt recently, Google can keep showing affected products for up to 30 days using previously approved images (Google). That grace period hides the problem; it does not fix it.

C. Bot protection, a firewall or geo-blocking returns 403 to Google only

This branch confuses people because the page loads fine in a browser. Google names the pattern itself: firewalls or CDNs such as Cloudflare that return 403 or 404 only to automated bots, user-agent blocklists, and fingerprinting frameworks that flag crawlers by user agent, IP, browser variables or crawl rate (Google, StoreBot access).

1. Check the firewall or CDN log for blocked or challenged Googlebot, AdsBot-Google or Storebot-Google requests.
2. Verify by reverse DNS, not a static IP list. Google advises against static IP allowlisting and recommends reverse DNS (Google). Googlebot resolves to googlebot.com, AdsBot to google.com, and Google publishes ranges as common-crawlers.json and special-crawlers.json (Google, verifying requests).
3. Know what your bot product can and cannot exempt. Cloudflare's documentation says Bot Fight Mode cannot be bypassed or skipped with WAF custom rules or Page Rules, while Super Bot Fight Mode supports Skip rules (Cloudflare docs). If Google is being challenged under Bot Fight Mode, the fix is a configuration or plan change, not another rule.
4. Check geo-blocking. Google's instruction is to allow the AdsBot-Google user agent and, if you restrict regions, to allow crawling from the US (Google). Stores limiting traffic to Turkey or the Gulf hit this.

Do not test this by sending curl with a Googlebot user agent from your own machine. A bot product that verifies by IP will block your request as fake even when the real Googlebot is allowed. Use URL Inspection, which fetches from Google.

D. Too many redirects, or a broken redirect target

Google asks for no more than two redirects (Google). A typical three-hop chain is http:// to https://, bare domain to www, old slug to new. Submit the final URL. Redirects that return an invalid, empty or overly long URL are a separate failure (Google). If you use the regional availability and pricing add-on, URLs carrying a region_id parameter must also be crawlable (Google).

E. 5xx responses, timeouts or DNS failures

These point at the server, not the feed; Google lists 5xx, unresolvable hostnames, malformed responses and timeouts as separate causes (Google). Its peak-season guidance: if your site is down or times out, items may be disapproved, so confirm the server can take a traffic spike and check speed with PageSpeed Insights (Google, preparing for big events). Do this before a Black Friday or Ramadan peak, not during one.

F. Nothing is wrong on your side

If URL Inspection returns 200, robots.txt allows Google, the firewall log is clean and only a scattered minority is affected, request a website check. It speeds up re-approval; Google may take up to 12 hours and accepts no second request in that window (Google). The issue can return, sometimes the next day (PPC Land); if it does, escalate to Merchant Center support with your inspection results.

How long recovery takes after the fix

Google gives different timings for different stages. Use this table to set expectations with the budget owner:

A prevention checklist you can score

Score one point per item you can confirm today. Seven or more: errors should be rare and quick to diagnose. Four or fewer: the next robots.txt or CDN change will probably take products offline.

1. The feed exports the final canonical product URL, never a URL that redirects more than once.
2. A feed rule percent-encodes non-ASCII characters and commas exactly once.
3. No staging, preview or test domains can reach the production feed.
4. robots.txt allows Googlebot and Googlebot-Image across product and image paths.
5. Nobody has written a Storebot-Google or AdsBot-Google disallow without a reason someone can explain.
6. Bot protection exempts verified Google crawlers by reverse DNS, and your bot product actually supports exemptions.
7. Geo-restrictions allow crawling from the US.
8. Every robots.txt, CDN or WAF change is tested on three product URLs with URL Inspection.
9. Someone checks Needs attention weekly, and daily in peak season.
10. Products that go out of stock or are deleted leave the feed before their URLs start returning 404.

Items 1 to 3 and 10 live in the feed, where a feed tool can enforce them. Feedance's Feed Protection runs its checks on every feed update and drops items with broken image links before they reach a channel. Our Google Merchant Center integration sends the link values your rules produce. The image side of the same problem is covered in our guide to fixing product image links.

Where this guide breaks down

Feed tooling, ours included, does not crawl your site as Googlebot does. Feed Protection validates feed data on each update, is not real-time monitoring, and does not fetch landing pages from Google's network. Branches B, C and E stay with your developer, host or CDN administrator, and URL Inspection plus your server or CDN logs are the right tools there; no feed platform replaces them.

This guide covers Google only; Meta and TikTok Shop use their own crawlers and error names. Google's own pages disagree on timings, so treat the recovery table as a range. And the "10% means Google's side" rule comes from one practitioner, and we have no first-party data to confirm it.

Frequently asked questions

Does "Product page unavailable" mean my website is down?

Not necessarily. It means Google's crawler could not fetch the submitted URL, which can be a wrong feed URL, a robots.txt block, a bot-only firewall block, too many redirects or a Google-side failure.

Why are my products disapproved on desktop when only the mobile page failed?

Google says a crawl error on either the mobile or desktop landing page disapproves the item on both devices until the page can be reached. Fix the failing version and both come back.

Is allowing Googlebot in robots.txt enough for Shopping ads?

Not always. AdsBot-Google ignores the User-agent: * group and Storebot-Google has its own token for Shopping surfaces. Check that no rule disallows either.

Should I allowlist Google's IP addresses in my firewall?

Google advises against static IP allowlisting and recommends reverse DNS verification. If you need ranges, read Google's published JSON files rather than a list copied once.

How long until disapproved products come back after the fix?

A requested website check takes up to 12 hours; Google's pages then cite up to 48 to 72 hours for products and the Needs attention tab to update. Plan for two to three days.

Do Turkish or Arabic characters in product URLs cause disapprovals?

They can, if they reach Google unencoded or double-encoded. Encode non-ASCII characters and commas exactly once in a feed rule.

Can a landing-page problem suspend my whole account?

Yes. Google has a critical-severity, account-level issue for landing pages not working that suspends the account, so a full-catalog spike in these errors needs same-day attention.

Check your feed's links before Google does

The feed-side causes in this guide (wrong domains, unencoded characters, URLs that should not be in the feed at all) are cheaper to catch in the feed than in Merchant Center. Run your feed through the free product feed audit tool, which scores data completeness, data quality and technical health, and work through what it reports before your next peak.

Sources

• Google Merchant Center Help: How to fix: Product page unavailable
• Google Merchant Center Help: Product crawl issues
• Google Merchant Center Help: How to fix: Unable to check product pages
• Google Merchant Center Help: How to fix: Unable to do quality & policy checks on product pages
• Google Merchant Center Help: How to fix: Google StoreBot crawler can't access your in-store product page
• Google Merchant Center Help: How to prepare for big events and product launches
• Google Merchant Center Help: Product data specification
• Google Content API for Shopping: Issue severity and Merchant Center Diagnostics
• Google Search Central: List of Google's special-case crawlers
• Google: List of Google's common crawlers
• Google: Verifying Google crawlers and fetchers
• Cloudflare docs: Bot Fight Mode
• PPC Land: Google Merchant Center's "product page unavailable" error kills ads (March 2026)