Large Indian catalogues (D2C Shopify Plus, custom marketplaces, B2B parts sites) waste crawl on filters, sort orders, and session URLs while money SKUs wait. Crawl budget is not mystical — it is which URLs Googlebot fetches usefully. Start from the technical SEO audit checklist.
Diagnose before you Disallow
Use log file analysis for two weeks:
- Top paths Googlebot hits
- Status codes (soft 404s, 302 chains)
- Share of crawl on faceted / parameter URLs
- Whether sitemap URLs are actually fetched
If bots never reach new products, fixing copy will not help.
robots.txt rules that are usually safe
| Pattern | Intent |
|---|---|
| Block internal search | Prevent index bloat |
| Block cart / checkout / account | No SEO value |
| Allow product / category canonicals | Money paths |
| Avoid blocking CSS/JS needed to render | Rendering still matters |
Never Disallow a URL you still want indexed. Prefer canonical + noindex + parameter handling for borderline filters — see faceted navigation SEO and index bloat cleanup.
Sitemap discipline
Split product vs category vs content sitemaps. Only include canonical 200 URLs. Keep lastmod honest. Guide: XML sitemap best practices.
Parameter playbook
- Inventory tracking params (UTM should not create indexables)
- Decide indexable facets (rarely more than brand + one primary attribute)
- Canonical to clean URL
- Confirm in GSC URL Inspection after release
14-day fix sequence
Days 1–3: Log sample + GSC “crawled – currently not indexed” themes.
Days 4–7: robots + parameter rules + sitemap cleanup.
Days 8–11: Facet templates noindex/canonical.
Days 12–14: Re-check bot hits on new SKUs.
FAQ
Will Disallow fix ranking? No — it only frees crawl. Rankings need content and links.
Shopify? Theme apps create parameter mess; pair with Shopify technical SEO checklist.
Need help? Technical SEO consulting — proof: fintech organic growth.
Related reading on Fundaking
- Technical SEO Audit Checklist (2026)
- Faceted Navigation SEO for Ecommerce
- Index Bloat Cleanup Playbook
- Log File Analysis for SEO: A Practical Guide
- Technical SEO consulting
- Fintech organic growth case study
Rahul Agarwal — Fundaking Media. SEO consultant based in Pune / Baramati, serving founders across India.
Rahul Agarwal
