Is your SEO audit process getting bogged down by large websites and complex data? With a variety of advanced features, Screaming Frog can be an invaluable asset for technical SEO, yet many users only scratch the surface of what it can accomplish. In fact, only a small percentage of users utilize its deeper functionalities. Screaming Frog has features that can streamline your audits and reveal critical insights efficiently.
This guide covers advanced Screaming Frog tactics that will help you optimize crawl efficiency, refine link structure, validate data integrations, and analyze key performance metrics, allowing you to conduct thorough audits with precision and ease.
10 Advanced Screaming Frog Tactics to Improve SEO Audits
1. Optimizing Screaming Frog for Large-Scale Crawls
When crawling large websites, standard settings might not cut it. Screaming Frog’s advanced configurations can make a significant difference in processing speed, data handling, and overall crawl success.
Switching to Database Storage for Scalability
By default, Screaming Frog uses RAM to store crawl data, which can work well for smaller sites. However, for sites with thousands or even millions of pages, switching to database storage is a game-changer. Database storage allows Screaming Frog to use your machine’s hard drive, particularly if it’s an SSD, to store data without slowing down or crashing.
To switch, go to Configuration > System > Storage Mode and select Database Storage. Ensure you’re using an SSD, as it enhances crawl speed and efficiency on large-scale websites, saving valuable time.
Focused Crawling with Targeted Subdirectories or Subdomains
Not all parts of a large website are equally relevant for every audit. For instance, you may want to crawl only blog pages, product pages, or a specific subdomain. In Screaming Frog, you can use the RegEx feature, which utilizes regular expressions, to focus the crawl on specific sections, making the process quicker and the results more manageable.
Under Configuration > Include, you can set parameters like /blog/.* to limit crawls to URLs within the “/blog” directory.
2. Enhancing Site Analysis with Integrated Data Sources
Connecting Screaming Frog with Google Analytics, Google Search Console, and other tools can enrich your crawl insights. Here’s how you can use these integrations to save time and add data depth.
API Integrations for Comprehensive Data
Integrate Google Analytics and Search Console, as well as tools like Moz or Ahrefs, to import metrics like organic traffic, click-through rate, and keyword rankings directly into Screaming Frog. This streamlines audits by centralizing data and eliminates the need to switch between platforms.
To do this, go to Configuration > API Access and connect your preferred tools.
Requesting Retroactive Data After the Crawl
Sometimes, you may forget to link these tools before beginning a crawl. With Screaming Frog, you can use the Request API Data button after a crawl to pull in data retrospectively. This feature matches the data to the corresponding URLs, saving the trouble of a re-crawl and helping you build a richer report.
3. Improving Crawl Efficiency and Accuracy
An efficient crawl saves time and reduces the risk of missing key issues, which is especially important for larger sites.
HTML-Only Crawling for Efficiency
When performing a metadata audit or page status analysis, crawling only HTML pages is often sufficient. Under Configuration > Spider, uncheck non-HTML resources like images, CSS, and JavaScript. This reduces the processing load, speeds up the crawl, and ensures that only essential data is collected.
Handling Parameter-Heavy URLs
Some sites, like e-commerce stores, have URLs with various parameters that can clutter your crawl data. To exclude these, use RegEx to filter out URLs with specific parameters. For instance, adding \\? to the Exclude field ignores all URLs with parameters, while \\?price will ignore only URLs with the “price” parameter.
4. Auditing Redirects and Link Issues in Migrations
Website migrations can result in broken links and misdirected URLs, harming user experience and SEO. Screaming Frog’s List Mode and redirect analysis tools streamline this process.
Checking 301 Redirects:
- Upload a list of old URLs in List Mode.
- Enable Always Follow Redirects under Configuration to automatically track redirects.
- Filter crawl results by response codes to identify any links not returning a 301 status.
- Perform a full crawl to detect broken and redirected internal links.
- Export the Response Codes > Inlinks report to list all internal links with non-200 status codes.
- Use the report to:
i) Fix 404 errors.
ii) Remove or update unnecessary redirects.
iii) Optimize internal linking for better site structure and SEO performance.
This process ensures all broken links are cleaned up and redirect issues are resolved for a smooth migration and improved user experience.
5. Sitemap Creation and Validation
A well-structured sitemap helps search engines index a website efficiently. Screaming Frog simplifies both creating and auditing sitemaps. To generate a sitemap, crawl the site in Spider Mode, then go to Sitemaps > XML Sitemap and select the pages you want to include.
Screaming Frog includes only 200 response pages by default but allows you to customize which pages are added based on criteria like status code or last-modified date. If there’s an existing sitemap, switch to List Mode and upload the sitemap.xml URL.
It will check for issues such as broken links or redirects within the sitemap. This audit ensures that your sitemap reflects the actual structure of the site, optimizing it for search engine crawlers.
6. Structured Data Validation
Structured data improves visibility by enabling rich results in search. Screaming Frog can validate structured data to ensure it’s implemented correctly. To enable structured data auditing, go to Configuration > Spider > Extraction > Structured Data and select the format (e.g., JSON-LD, Microdata).
This setting will allow Screaming Frog to crawl and list pages containing structured data. After the crawl, go to Reports > Structured Data and export any errors, giving you a clear view of structured data issues.
Once you have the report, prioritize fixing errors over warnings, as these impact the display of rich results. Addressing issues like missing properties or incorrect field types will enhance the chances of your content appearing with rich snippets.
7. Assessing Internal Linking Strategy with Link Score
An effective internal linking strategy ensures that link equity is distributed across important pages, improving SEO performance. Screaming Frog’s Link Score metric helps evaluate and optimize internal linking.
Why Link Score Matters:
- Calculates page value based on internal links.
- Identifies areas where link equity distribution can be improved.
Using Link Score for Optimization:
- Run a crawl and select Crawl Analysis, then click Start to populate the Link Score metric in the Internal tab.
- Analyze pages with:
i) Low scores but high relevance: Add more internal links to improve their visibility and authority.
ii) Overlinked pages: Remove unnecessary links to balance equity distribution.
By leveraging the Link Score metric, you can refine your internal linking strategy to strengthen the overall site structure and improve SEO performance.
8. Page Speed and Performance Analysis
Page speed is crucial for both SEO and user experience. Screaming Frog’s Response Time metric provides insight into potential bottlenecks.
Using Response Time and Page Speed API Integration
The Response Time column in the Response Codes tab highlights pages that load slowly. For more granular insights, integrate Google’s PageSpeed API (API key required). With this setup, Screaming Frog generates a comprehensive PageSpeed Insights tab, showing detailed performance metrics like Time to First Byte (TTFB) and page load times.
9. Crawlability and Rendering Audits
To ensure that all content is accessible to search engines, perform audits that mimic how bots crawl and render your site.
Simulating Crawls with User-Agent Switching
Sometimes, it’s beneficial to crawl different search engines to see potential indexing issues. Change the User-Agent under Configuration > User Agent and select Googlebot Desktop, Mobile, or other bots. This tactic is valuable for auditing mobile-friendliness and ensuring that essential elements load correctly across devices.
JavaScript Rendering and Screenshot Comparison
To audit JavaScript sites, go to Configuration > Spider > Rendering and select JavaScript. Screaming Frog renders pages as a search engine would, allowing you to check that essential links and content aren’t hidden. You can also enable Rendered Page Screenshots to visually compare JavaScript and non-JavaScript versions, identifying issues that could affect crawlability.
10. Ensuring Consistency of Analytics Tracking
Accurate data is critical for measuring SEO success. Screaming Frog can identify pages that are missing analytics tracking, ensuring consistency across the site. Filter for pages that contain or lack your Google Analytics ID.
Use two custom filters: one to find pages containing your unique tracking code and one for those missing it. For efficiency, uncheck all non-HTML resources in Configuration > Spider to focus on the main content pages. Correct any tracking gaps to ensure accurate data collection.
Here is a table summarizing why to choose Screaming Frog and why not:
Why Choose Screaming Frog | Why Not Choose Screaming Frog |
Comprehensive Analysis: In-depth SEO audits for issues like broken links and metadata. | Learning Curve: Interface can be complex for new users. |
Affordable: Reasonably priced with a free version for small websites. | Free Version Limitations: Crawls only up to 500 URLs. |
Fast Crawling: Quick website analysis. | Outdated Interface: UI could be more intuitive. |
Versatile: Supports exporting data in multiple formats. | Technical Complexity: Challenging for non-technical users. |
Best Practices for Using Advanced Screaming Frog Tactics
To make the most of Screaming Frog’s advanced features, it’s essential to approach your audits with precision and adaptability.
Here are some best practices to ensure your audits are efficient, insightful, and aligned with your SEO goals:
- Optimize Crawl Settings for Efficiency:
Use database storage for large sites, and set targeted crawls with RegEx to focus on high-priority site sections without overloading your system. - Integrate Data Sources for Richer Analysis:
Connect Screaming Frog with tools like Google Analytics, Search Console, and PageSpeed Insights to consolidate your data and enrich your reports. - Prioritize Errors Over Warnings in Structured Data:
Structured data validation is essential but start by resolving errors before moving to warnings to ensure an accurate, rich snippet display. - Leverage Link Score to Balance Internal Links:
Use the link score metric to assess which pages need more internal links and which could benefit from link reduction, optimizing your link equity. - Regularly Audit Crawl Settings for Evolving Needs:
Adjust settings for specific project needs, whether for migration checks, speed audits, or JavaScript rendering, to keep your audits relevant and precise.
By following these best practices, you can maximize Screaming Frog’s capabilities and achieve more targeted, actionable insights in every audit.
Upgrade Your SEO Audits with Screaming Frog
Mastering Screaming Frog’s advanced features can significantly enhance the quality and depth of your SEO audits. By pushing beyond the basics, you can streamline complex audits, gain precise insights into site health, and ultimately drive SEO success for even the most demanding websites.
Whether it’s refining internal link structures, auditing large sites efficiently, or validating structured data, these tactics help you confidently cover all the critical bases. Start integrating these advanced Screaming Frog techniques into your process today and see the difference they make.
FAQs
What is the Screaming Frog tool used for?
Screaming Frog is used for technical SEO audits, allowing users to crawl websites, analyze URL structures, find broken links, identify redirects, check metadata, and gather data to optimize site performance.
What are the benefits of Screaming Frog?
Screaming Frog improves SEO by identifying site issues quickly, enhancing crawl efficiency, analyzing page speed, validating structured data, managing sitemaps, and offering insights that optimize content, links, and overall site health.
How to learn Screaming Frog?
Start by exploring Screaming Frog’s official guides and tutorials. Practice using basic features, then delve into advanced functions, like custom extraction, integrations, and filtering, to develop comprehensive auditing skills.
Does Screaming Frog show backlinks?
While Screaming Frog doesn’t natively show external backlinks, it can connect to tools like Ahrefs or Majestic via API to pull backlink data, allowing integration into your audits.
How to find broken backlinks in Screaming Frog?
Import a list of external links into Screaming Frog using List Mode. Crawl the URLs to identify any broken links or 404 errors, helping to locate broken backlinks efficiently.
Ready to elevate your B2B marketing?
We help leading business-to-business brands hit their marketing goals. Get in touch to learn how Altitude Marketing can help you reach your peak performance.
