A desktop website crawler used by SEO professionals to audit the technical health of a site.
SEO professionals read this when auditing a site's crawlability, indexation status, and structural issues compared to how search engines operate.
01What it is and how it works
Screaming Frog SEO Spider runs locally on Windows, macOS, or Linux and crawls a domain starting from a seed URL. It issues HTTP requests, parses HTML, JavaScript-rendered DOM (in JavaScript rendering mode), robots.txt, XML Sitemap, and response headers, then stores each URL's status code, content, and metadata. Results are presented in a tabbed interface and can be exported to CSV or Excel. The free version caps crawls at 500 URLs; the paid licence removes the cap and unlocks features such as scheduled crawls, custom extraction via XPath/CSS path, and Google Search Console and analytics integration. Because the tool mimics a search engine bot, it surfaces the same classes of issues that crawlers encounter in production: redirect loops, soft 404s, canonical conflicts, blocked resources, and thin or duplicate content.
It is a program you install on your computer that visits your web pages the way Google does, then lists problems it finds, such as broken links, missing tags, and redirect chains.
02What to do about it this week
Run a crawl of your own domain and one of a key competitor. In the crawl, filter the Response Codes tab for 4xx and 5xx status codes, then sort the Page Titles and Meta Descriptions tabs for missing, duplicated, or oversized entries. Export the Hreflang tab if you run a multilingual site and verify every URL has a valid hreflang cluster. Use the Custom Extraction feature to pull structured data fields, such as Product price or Review rating, and compare coverage against what is visible on the rendered page. If you have access, run a second crawl in JavaScript rendering mode to compare what the raw HTML returns against what a user sees after script execution. The aim is a short list of concrete fixes, not a 50,000-row export.
03How it is measured or noticed
Coverage is measured by the count of distinct URLs the spider discovers versus the count Google indexes; a persistent gap usually points to crawl budget waste, orphaned pages, or robots.txt disallow rules that are too broad. Structural issues show up in the Redirect Chains and Redirect Loops tabs, where a chain length greater than two hops is a flag. Metadata coverage is read directly from the Page Titles and Meta Descriptions columns, with empty cells indicating missing tags and identical cells indicating duplication. Indexability is read from the combination of Indexability Status (Indexable, Non-Indexable, Blocked by robots) and the Canonical tab, which lists declared canonicals and whether they resolve. Performance is read from the Response Time column, which is useful for spotting slow templates that Googlebot may crawl less often.
04Common mistakes
- Treating the URL cap as a site audit: 500 URLs is enough to spot patterns on a small site but not to represent a 100,000-page property.
- Blocking the user agent in
robots.txtand then wondering why the crawl is empty. - Ignoring the difference between the
InternalandExternaltabs and shipping fixes that are actually outside your control. - Comparing a single desktop crawl against Google's mobile index without enabling the mobile user agent or JavaScript rendering.
- Leaving default crawl settings in place for a large site and exhausting memory, which produces partial results that look authoritative.
05Limits and common confusions
Screaming Frog SEO Spider is not a rank tracker, a backlink index, or a substitute for server logs. It fetches pages from your machine the same way a browser would, which means it does not see what Googlebot sees if your site serves different content by IP, user agent, or geography. It also does not run a real Lighthouse pass; page speed insights must come from a separate tool. Results are a sample: the spider obeys robots.txt and stops where you tell it to stop, so a truncated crawl can hide problems rather than reveal them. Finally, it is frequently confused with a full SEO platform such as Ahrefs, Semrush, or Lumar; in practice, the spider answers technical questions about your own site, while those platforms add third-party data about competitors, backlinks, and keyword demand.
06Worked example
A marketer suspects that the launch of a new product line has introduced duplicate content across faceted navigation. They open Screaming Frog SEO Spider, paste the homepage into the address bar, and set the crawl to follow canonicals rather than every parameter. After the crawl completes, they open the Duplicate tab and find that 312 URLs share the same ` and 89 share a near-identical meta description. A quote from the Content` tab reads:
"Detected 89 duplicate meta descriptions across URLs under /shop?colour= and /shop?size=. Suggested canonical points to the parent category page."
Frequently asked questions
How is Screaming Frog SEO Spider different from a rank tracker?
A rank tracker monitors where your pages appear in search results for specific keywords. Screaming Frog SEO Spider crawls your site like Google does and reports on technical issues such as broken links, missing meta tags, or duplicate content. They measure different things and serve different purposes in an SEO workflow.
Do I need programming skills to use Screaming Frog SEO Spider?
No. The tool has a graphical interface and requires no code. You enter a URL, click crawl, and read the results in a spreadsheet-like table. Basic SEO knowledge helps you understand which issues matter, but the tool itself is designed for SEO practitioners who are not developers.
Why are my crawl results different from what Google indexes?
A crawl often finds more URLs than Google shows in its index because the spider does not obey crawl budget limits the way Google does. Common reasons for the gap include orphaned pages, session ID parameters, noindex directives, or canonical tags pointing elsewhere. The gap itself is the signal to investigate rather than a problem to solve.
What happens if I crawl a site without permission?
Crawling third-party sites without authorization can trigger blocks, consume their server resources, and raise legal concerns. Always crawl your own domains or sites you have explicit permission to audit. The tool lets you set speed limits and robots.txt respect to minimize disruption on your own properties.
Can Screaming Frog SEO Spider replace server log analysis?
No. The spider simulates crawling but does not see what Googlebot actually received or how your server responded. Server logs show real request headers, response codes, and crawl frequency. Use the spider to find issues, and use logs to confirm whether Googlebot encountered those issues in practice.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Usually yes. A small site under a few hundred pages can finish in under a minute. Even larger sites show critical issues like 4xx errors and missing titles within the first few minutes of a crawl. Start the crawl now and sort by status code to see the problems first.
Redirect chains mean a URL bounces through more than one redirect before landing. They slow Googlebot down and waste crawl budget. Fix the chain by updating the redirect to point directly to the final destination. You will see the warning clear once the chain is one hop.
Usually not a crisis. URL parameters often create crawl duplicates that Google ignores. Check whether the page has a canonical tag pointing to the clean URL. If it does, you are fine. If not, add one to tell Google which version is the master.