FREEFirst project with us? Your landing page design is on the house.See the offer

Chapter 2 of 6

Crawl and inventory every URL

You cannot redirect a page you do not know about. Combine six sources into one sheet, save a snapshot of each page, and sort everything by what it earns.

6 min readUpdated 2026-11-07
All chaptersChapter 2 of 6
  1. 01Is moving worth it?
  2. 02Crawl and inventory every URL
  3. 03Build the redirect map
  4. 04Rebuild content, SEO, forms and tracking
  5. 05Launch day order
  6. 06The first 30 days

The short answer

Build one sheet with every URL your site has, from six sources, not just the ones in the menu. Then save what each page says today and what it earns today. This sheet decides what you rebuild, feeds your redirect map, and gives you something to compare against after launch.

Nothing here is hard, but it is the step people skip. The URLs that hurt later are usually the ones nobody remembered: an old campaign page, a PDF that other sites link to, a tag archive that picked up clicks.

Six sources, because each one misses something

No single source sees everything. Use them all, and merge the results.

SourceWhat it findsWhat it misses
Site crawl (Screaming Frog or similar)Everything linked from your pages, with status codes, titles and canonicalsPages with no internal links
XML sitemapWhat the CMS thinks is liveHard-coded pages, and pages the plugin leaves out
Search Console, Performance and PagesURLs Google knows and sends clicks to, including old onesPages with no impressions
Analytics, landing pages for the last 12 monthsPages people arrive on, including ad and email trafficVisits from people who declined tracking
Backlink toolOld URLs other sites link to, including dead onesLinks it has not found yet
Your own listsAd landing pages, QR codes, email links, PDFs in sales decksWhatever nobody wrote down, so ask around

The Search Console screen caps how many rows you can export. On a large site, pull the data through its API, or export by folder.

Also ask your sales and marketing teams one question: "What links do you send people that are not in the menu?" You will usually get an answer that surprises someone.

The URLs people forget

Go through this list on purpose. Each of these has caught someone out.

  • Ad and campaign landing pages kept out of the menu, and sometimes marked noindex.
  • Thank-you pages. Goals and ad conversions are often tied to them.
  • PDFs, images and downloads that other sites link to.
  • Blog tag, category, author and page-two archives.
  • Language and country folders.
  • Old redirects already on the server. They must be carried over, or they break quietly.
  • www and non-www, http and https, with and without a trailing slash.
  • Subdomains such as blog. or help., if they are part of the move.
  • Pages used by other systems, such as a form embedded in your app or a webhook endpoint.

What to record for each URL

One row per URL. These columns are enough, and they become the start of your redirect map in the next chapter.

ColumnWhy
URL and status codeTells you what is live, redirected or already broken
Title, meta description, H1You carry these across, and compare them after launch
Canonical and robotsShows what is meant to be indexed
Schema typesEasy to lose on a rebuild
Internal links inShows which pages your own site treats as important
Clicks and impressions, last 12 monthsFrom Search Console
Sessions and conversionsFrom analytics, if you have them
Links from other sitesFrom your backlink tool
Forms and tools on the pageSo each one gets rebuilt and tested
Tier and ownerFilled in below

Use 12 months if you can, so seasonal pages show up. A page that earns only in December looks dead in March.

Sort everything into tiers

Not every page needs the same care. Tiers tell you where to spend your time, what to test by hand and what to check first after launch.

TierWhat is in itHow you treat it
APages with clicks, links from other sites or leads. Plus money pages, even if quiet: pricing, contact, demo.Rebuild carefully. Compare line by line. Test by hand.
BIndexed pages with a little traffic. Most blog posts.Rebuild or merge. Check by script, and spot check.
CNo clicks, no links, no conversions in 12 months.Decide: keep, merge or retire. Do not move them out of habit.

Decision rule A page with links from other sites or any conversions is never tier C, however little traffic it gets. Retiring a page is fine. Retiring it by accident is not.

A move is a good moment to retire thin pages. Be careful with this on a tier C page with a long tail of small queries. Check its queries before you cut it, and merge it into a stronger page where the topic fits.

Save the "before" picture

Once the move is live, you can never look at the old site again the way it was. Do this before anything changes.

  • Export Search Console data by page and by query, for the last 12 months, and keep the file.
  • Note leads and sales by month, and what normal looks like for the season.
  • Keep the full crawl, with the date in the file name.
  • Take full-page screenshots of your tier A pages.
  • Note current speed and Core Web Vitals for your top templates.
  • Back up the old site and its CMS content where you can get an export.
  • Write down who has access to DNS, hosting, the domain and analytics. You will need all of them on launch day.

Keep the results in one shared folder. If something looks odd two months later, this is what you will reach for. It is also how you can tell a real drop from a normal seasonal one.

Freeze, and agree when

The inventory goes stale the moment someone publishes a new post. Agree a content freeze date with your team, as late as you safely can and well before launch. After the freeze, new content waits, or gets added to both sites and logged in the sheet.

Run one more crawl of the old site at the freeze, and compare it with your sheet. Anything new gets a row.

Done when This chapter is done when one person can answer "what happens to this URL?" for every row, and when you have the before picture saved.

Our redesign checklist has more on what to save before a rebuild.

Takeaways

What to remember from chapter 2.

  • Combine six sources. A crawl alone misses pages with no internal links.
  • Hunt for the forgotten URLs: campaign pages, PDFs, thank-you pages, old redirects.
  • Tier by what a page earns. A page with links or conversions is never tier C.
  • Save the before picture while you still can.
  • Agree a content freeze, and crawl again when it starts.

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.