A site owner wanted a page out of Google's results, so they added it to robots.txt, with a disallow rule for that exact URL. The page stayed in the results. A month later, it was still there, with the same title and the same snippet. The owner added more disallow rules, and the page stayed. The owner had done the one thing that guarantees a page stays in the index: told Google not to look at it.
Robots.txt is a crawling instruction, not an indexing instruction. A disallow rule tells Googlebot not to fetch the page. It does not tell Google to remove the page from the index, and it does not tell Google the page is gone. If the page was already indexed before the disallow was added, it stays indexed, because Google cannot see any change to it. The disallow freezes the result in place.
Google's index is built from what its crawler sees. When a page is disallowed, the crawler stops visiting it, which means it stops seeing the noindex tag, the deletion, or the redirect that would tell it the page should go. The result in the index is the last version Google saw, and it stays there, with the same title and snippet, because nothing has told Google otherwise. The disallow does not remove the page. It removes Google's ability to notice that the page should be removed.
The same logic applies in reverse. A page that was never indexed and is then disallowed will never be indexed, because the crawler never sees it. That is the one case where a disallow works as people expect: for a page that should never appear at all, and never has. For a page that is already in the results, a disallow is the wrong tool, and it is the wrong tool in the exact way that makes the problem worse.
The right tool for removing an indexed page is a noindex tag, and the noindex tag needs the crawler to see it. That means the page must be crawlable: no disallow rule for it, and a server that returns the page with the tag in place. Google visits the page, reads the noindex, and drops the URL from the index, usually within days. The page stays live for visitors, but it stops appearing in search results.
If the page should be gone entirely, delete it and return a 410 status, which is processed faster than a 404. The 410 is an explicit statement that the page is gone deliberately. A 404 works too, but it takes longer, because Google has to decide whether the page is temporarily missing or permanently gone. Either way, the page has to be crawlable for Google to see the status, which means no disallow rule for it.
Robots.txt and noindex are not alternatives. They are different tools for different jobs, and they work together when used correctly. Use robots.txt to keep the crawler out of pages that should never be indexed and never have been: admin pages, internal search results, staging environments. Use noindex for pages that are indexed and should not be, and leave them crawlable until Google has dropped them. Then, once the page is out of the index, add the disallow if you want to keep the crawler away permanently.
Check the order. Remove any disallow for the page first, add the noindex or the 410, wait for Google to drop the URL, and only then add the disallow back. The sequence matters, because the disallow first is what froze the result in place. The fix is to let Google see the change, and then to close the door.
Keep reading: delisting vs Google removal, and which one you need, and how to remove court record aggregator pages cover the nearby cases in this series.
Because a disallow tells Googlebot not to fetch the page, so it never sees the noindex tag, the deletion, or the redirect that would remove it. The result in the index is the last version Google saw, and it stays there. A disallow freezes an indexed page in place.
A noindex tag, and the page must stay crawlable so Google can see it. Google visits the page, reads the noindex, and drops the URL from the index, usually within days. If the page should be gone entirely, delete it and return a 410 status, which is processed faster than a 404.
For a page that was never indexed and should never be: admin pages, internal search results, staging environments. The crawler never sees the page, so it never enters the index. For a page that is already in the results, a disallow is the wrong tool.
Yes, in the right order. Remove any disallow for the page first, add the noindex or the 410, wait for Google to drop the URL, and only then add the disallow back. The sequence matters, because the disallow first is what freezes the result in place.