A Deploy Put noindex on Eight Pages

In October, Search Console started listing pages on our site under Excluded by noindex tag. Eight of them, all real pages, and every one of them answered with index, follow when we fetched it. The tag Google reported was not in the page we served.

Hashed files, deleted on every build

A SvelteKit build writes its scripts into /_app/immutable/ with a content hash in each file name. The HTML for a page names the exact files it needs. Our deploy built into a fresh folder, so after every deploy, the files from the previous build no longer existed. Files whose content had not changed kept their names and survived. Files that had changed were gone.

In the weeks after we moved the site to our own server, we deployed several times a day.

What Google saw

Google fetches the HTML of a page, then renders it later, sometimes much later. If a deploy lands in between, the HTML asks for a script that has been deleted. We tested that on the live site by refusing just the script for one article. The article disappeared and our error page took its place: Something went wrong.

The error page script itself had not changed between builds, so it still existed under its old name, which is why the error page could load when the article could not. And the error page was written with noindex on it.

<SEO title="Something went wrong" noindex={true} />

Put together: Google fetched a healthy page, rendered it after a deploy, saw an error page with noindex, and recorded the page as excluded. This is our reconstruction. We cannot watch Google render, but the timing matched our deploys and the test above shows the same result in a browser.

Fix one: keep the old files

nginx now serves /_app/immutable/ from a folder that collects every build’s files, and passes anything it does not have to the app.

location /_app/immutable/ {
  root /var/www/cairostudio-immutable-root;
  try_files $uri @app_immutable;
  add_header Cache-Control "public, max-age=31536000, immutable";
}

Every deploy copies the new build into that folder without overwriting, then deletes anything older than sixty days.

cp -rn build/client/_app/immutable/. \
  /var/www/cairostudio-immutable/
find /var/www/cairostudio-immutable -type f -mtime +60 -delete

The folder holds 2,787 files today, 21MB on disk, which is a small price for old pages that keep working.

Fix two: the error page does not say noindex

A real missing page returns a 404 status, and that already keeps it out of search. The noindex tag on the error page added nothing for real errors, and for a page whose render failed, it told Google to drop a page that was fine. We removed it.

All eight pages return index, follow when fetched, and we have asked Search Console to check them again.