Direct answer
If a URL requires a session, it is not a marketing page. Keep it out of the XML sitemap, add noindex where a login URL might still be discovered, and disallow operator and API paths in robots.txt.
Google’s robots documentation is specific: named user-agent groups replace the * group. If you allow an AI crawler by name, restate the private disallows in that group.
Sitemap is an allowlist
A sitemap is not a dump of every path your framework can render. Include canonical, indexable, trailing-slash marketing URLs only. Login, reset-password, and /app are not ‘content’.
Structured data on a login wall violates Google’s structured-data quality guidelines: markup must match visible content.
A practical sequence
Treat indexing as a product rule, not an afterthought.
- Inventory public vs authenticated routes
- Generate the sitemap from the public list only
- noindex login and thin utility pages
- Disallow /operator/, /api/, and app prefixes
- Restate disallows in every named crawler group
Common leaks
Staging hosts left indexable. Preview query strings. OG images that contain private names. Search Console coverage for ‘Indexed, though blocked by robots.txt’ on a login URL — fix the page, do not celebrate the index.
What to send if you want a senior review
Share the public host, the app host, and any staging URLs. Related work at Kiwi: technical SEO and performance foundation.
Frequently asked questions
Should /login be in the sitemap so people find it?
No. Link it from the site. Sitemaps are for canonical public content, not for utility URLs.
Does noindex replace authentication?
No. noindex is a crawler hint. Authorization is a server check. You need both.
Can AI crawlers see our portal?
Not if the routes are authenticated and disallowed. Do not grant a named bot a looser policy than Googlebot on private paths.
Sources
- Google Search Central — Block indexing with noindex
- Google Search Central — robots.txt introduction
- Google Search Central — Structured data guidelines
Figures cited above are drawn from the linked publications and are the responsibility of their sources; we date and scope them rather than presenting them as universal guarantees.