A desktop website crawler that scans URLs to collect SEO-relevant data such as titles, meta descriptions, headers, links, and response codes.
Web developers or SEO professionals use it when reviewing built-in reports for finding issues like broken links or missing meta descriptions.
01What It Is and How It Works
Screaming Frog is a desktop application that acts as a web spider. You give it a starting URL (the seed), and it follows every link it finds, respecting robots.txt directives and crawl-delay settings. It stores the data it collects in a series of in-memory tables: one for each URL with columns for status code, content type, title, meta description, heading tags, canonical tag, response time, word count, and more. The tool also parses sitemaps and can be configured to use a specific user-agent string (for example, Googlebot) so the crawl mimics how a search engine would see the site. It runs locally on your machine, so no data leaves your network unless you choose to export it. The free version crawls up to 500 URLs; the paid version removes that limit and adds JavaScript rendering, allowing it to see content loaded by client-side scripts.
It is a program you run on your computer that visits every page on a website and reports back SEO problems like broken links, missing meta tags, or duplicate content.
02What to Do About It
Download the free version from the official site and install it. Enter your website URL, adjust the crawl settings (set a crawl speed that won't overload your server, and choose a user-agent that matches your target search engine). Start the crawl and wait for it to finish. Once done, review the built-in reports: find broken links under the Response Codes tab, missing meta descriptions under the Meta Description tab, duplicate content under the Duplicates tab, and redirect chains under the Redirects tab. Export any list of issues as a CSV or Excel file and hand it to your development team. After fixes are deployed, recrawl the same URLs to confirm the problems are resolved. Make this a weekly or monthly routine for ongoing site health.
03How It Is Measured or Noticed
The primary output is a table of URLs with dozens of columns. You notice issues by sorting or filtering those columns. For example, filter the Status Code column to show only 404s to see all broken pages. The tool also provides summary charts: a pie chart of response codes, a bar chart of content types, and a list of the most linked-to pages. You can measure crawl completeness by comparing the number of URLs crawled against the number in your sitemap. The tool logs crawl time and requests per second, which helps you gauge server performance. Over time, you can export reports and compare them to track whether technical SEO issues are increasing or decreasing.
04Common Mistakes
- Crawling too fast without setting a crawl delay can overload your server and get your IP blocked.
- Not changing the user-agent to match Googlebot means the tool may see a different version of the page than Google does.
- Ignoring
robots.txtexclusions: if you do not configure the tool to obeyrobots.txt, it may crawl disallowed paths and waste time. - Using default settings without filters includes every URL parameter, session ID, and pagination link, creating noise that hides real issues.
- Assuming the free version can render JavaScript: it cannot. For JavaScript-heavy sites you need the paid version or a separate rendering service.
- Treating every warning as a critical error: a missing meta description on an admin page is not the same as on a public landing page.
05Limits
Screaming Frog is a snapshot tool, not a continuous monitor. It cannot replace log file analysis or real-user monitoring. The free version is capped at 500 URLs, which is too small for most mid-sized or large sites. The paid version handles larger crawls but still runs on your local machine, so memory and CPU can become bottlenecks for sites with hundreds of thousands of URLs. It does not crawl password-protected pages or pages behind login forms. It is often confused with a full site audit platform, but it is only a crawler: it collects raw data, and you must interpret that data and decide what to fix. JavaScript rendering in the paid version is not perfect; some dynamic content may still be missed.
06Worked Example
A marketer wants to find all pages with missing meta descriptions. They open Screaming Frog, enter the site URL, and start a crawl. After the crawl finishes, they click the 'Meta Description' tab. They see a list of URLs with a column for 'Meta Description 1'. They click the header to sort, then scroll to the blank rows. They select those rows, right-click, and choose 'Export Selected URLs'. The CSV file contains the URL list, which they send to the developer with a note: 'Please add a unique meta description to each of these 47 pages.' After the developer finishes, the marketer recrawls just those 47 URLs (using the 'List' mode) and confirms that every one now has a description.
Frequently asked questions
How is Screaming Frog different from cloud-based SEO tools like Ahrefs or Semrush?
Screaming Frog is a desktop application that crawls your site directly, giving you raw, granular data without aggregation or analysis. Cloud tools often provide built-in insights and historical data, but Screaming Frog offers more control over crawl settings and immediate results.
Should I use Screaming Frog or Google Search Console for crawl data?
It depends on your need. Google Search Console provides aggregated crawl data from Google's perspective, while Screaming Frog gives you a complete, customizable crawl of your own site. Use Screaming Frog for detailed technical audits and Search Console for understanding how Google sees your site.
How do I set up Screaming Frog to crawl my website?
Download the free version from the official site, install it, enter your website's URL in the input bar, configure any filters or settings you need (like excluding certain URL patterns), and click Start. The crawl will begin and populate a table with SEO data.
Is Screaming Frog still relevant for SEO in 2025?
Yes, it remains a standard tool for detailed technical SEO audits. It is widely used for checking redirects, finding broken links, analyzing meta tags, and identifying duplicate content. Its desktop nature means it works offline and gives you full control.
What happens if I crawl more than 500 URLs with the free version?
The free version is limited to crawling 500 URLs. If you exceed that, the crawl will stop or you will need to upgrade to the paid license, which removes the limit and adds features like JavaScript rendering and API integration.
How long does a crawl take for a large site?
It depends on your site size, server speed, and crawl settings. A site with thousands of pages can take from minutes to several hours. Screaming Frog is a snapshot tool, so it captures the state at that moment, not continuous monitoring.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Use Screaming Frog. It crawls your site and lists all 404 errors in minutes, so you can fix them before the meeting.
Run Screaming Frog. It will show you every page missing meta descriptions in a column, and you can export the list.
Screaming Frog can crawl your site and check redirect chains. It's a desktop app you can download and run right now.