Screaming Frog SEO Audit. 10 Useful Tricks

Screaming Frog is one of the best freemium tools on the internet to complete an SEO audit. We have worked with this program for years, and below you will find some of our favourite useful tricks to ensure you get the most out of this fantastic tool.

The journey to the first page of Google or Bing is a long one, and SEO is the mechanism to move your website to these prime positions. However, SEO is more than just keywords and fresh content ; it is an ever-developing and changing set of rules and regulations that requires a site to change with the times. This is why SEO audits are an essential part of any website regular operation.

An SEO audit is a look at the technical aspects of a website . This check will look at everything on the site to ensure that the overall health of a website is excellent. These audits ensure that a website continues to maintain and improve its position by keeping it up to date, and in good standing via the crawlers. Failure to audit a site can lead to significant issues, including a dropped ranking in the leading search engines.
So how to use screaming frog? This is what we will demonstrate in this article.

1. How to Find Broken Internal Links on a Page or Site

When managing a website, ensuring that all internal links are functioning correctly is crucial for both user experience and SEO performance. Broken links, redirects, and server errors can frustrate users and diminish search engine rankings. By using Screaming Frog SEO Spider, you can efficiently identify and rectify these issues, maintaining a healthy and optimized website. Below is a detailed guide on the link issues you might encounter and what they mean, helping you understand and prioritize the necessary fixes.

Analyze the Crawl Data

Once the crawl is complete, switch to the "Response Codes" tab.

In the "Response Codes" tab, you can filter to find specific types of issues:

No Response Inlinks

Description: These URLs did not respond to the crawler's request. This could be due to the server being down, having an incorrect URL, or other network-related issues.

Impact : Users and search engines cannot access these pages, leading to a poor user experience and potential loss of SEO value for those links.

Redirection (3xx) Inlinks

Description: These URLs respond with a 3xx status code, indicating that the requested resource has been moved to a different URL. Common 3xx status codes include 301 (Moved Permanently) and 302 (Found).

Impact: While browsers and search engines generally handle redirects, too many redirects can slow down the user experience and dilute SEO value. It's best to minimize the number of redirects where possible.

Redirection (JavaScript) Inlinks

Description: These are URLs that redirect users via JavaScript. This redirection type is not as straightforward as standard HTTP redirects and may not always be handled correctly by search engines.

Impact: JavaScript redirects can be problematic for SEO, as search engines might not always execute the JavaScript code, leading to indexing issues.

Redirection (Meta Refresh) Inlinks

Description: These are URLs that use the <meta http-equiv="refresh"> tag to redirect users. This type of redirection is less common and generally not recommended.

Impact: Meta refresh redirects can be confusing for users and search engines. If not properly handled, they can result in a poor user experience and potential SEO issues.

Client Error (4xx) Inlinks

Description: These URLs return a 4xx status code, indicating a client error. Common 4xx errors include 404 (Not Found) and 403 (Forbidden).

Impact: Links to 4xx pages lead to dead ends for users, resulting in a poor user experience. Additionally, search engines may penalize sites with many broken links, affecting overall SEO performance.

Server Error (5xx) Inlinks

Description: These URLs return a 5xx status code, indicating a server error. Common 5xx errors include 500 (Internal Server Error) and 503 (Service Unavailable).

Impact: Server errors indicate problems with the website's server, preventing users and search engines from accessing the page. Persistent server errors can severely affect the site's performance and search engine rankings.

View Inlinks to Problematic URLs

To see which pages link to these problematic URLs, click on the URL in the list and check the "Inlinks" tab at the bottom of the interface.

This will show you all the internal links pointing to the broken or redirected URLs.

Export the Data

If you need to export this data for reporting or fixing purposes, go to the top menu and select Reports > Crawl Overview > Export.

Alternatively, you can export specific tabs by right-clicking on them and choosing "Export".

Following these steps, you can effectively find and analyze broken internal links on your website using Screaming Frog SEO Spider.

2. Indexable / Non Indexable Pages - Protocol Tab

In the protocol tab, you will find a great tool that will help you with indexing and deindexing. Screaming Frog allows you to go in and check if you have robot's txt robots tag which has made essential pages deindexed. The traditional way to check is to go through the "site:" operator, but with this tool, you can skip this entire step!

Plus, with Screaming Frog, you will be able to not only check if a page is indexed or not, but the program will also showcase the reason why the page is not indexed. This is extremely helpful and will save you hours, especially if it is a large site.

To see indexable pages:

• Once the crawl is complete, switch to the "Overview" tab to get a summary of the crawl results.
• Go to the "Internal" tab.
• In the right-hand pane, look for the column labelled "Indexability". This will show whether a page is "Indexable" or "Non-Indexable".

3. View Images Over 100 KB & Incorrectly Sized Images

To find images over 100 KB , switch to the "Images" tab to view all images on the site.
Use the filter dropdown at the top of the "Images" tab and select "Over 100KB". This will filter the images that exceed the specified size.

In the "Images" tab, you can also find incorrectly sized images :

Use the filter dropdown and select "All Images" to see all images.
Look for the "Dimensions" column to see the actual dimensions of the images.
Compare the displayed dimensions with the required or recommended dimensions for your site. Screaming Frog might highlight images incorrectly sized based on the CSS or HTML attributes.

4. Multiply Canonicals - Canonicals Tab

Another great trick with Screaming Frog to use is to check if the canonical URL is correct and matching up with the URL that you want to be there. Google does treat a canonical as a suggestion or recommendation, but having everything in your site being consistent will help with your ranking.

Even if you have two canonicals on the same page, Screaming Frog will find these and check them. The default in Screaming Frog is only to check one. If you are operating multiple canonicals make sure to change the filter in Screaming Frog to multiply, to ensure that the program checks this.

5.Title / Description Length

The title and description length are essential to ensure that your SEO is as good as possible. Screaming Frog is a great tool to see what is visible in search results, and help you craft the perfect title and description for any search engine. You can use this trick to test our different lengths to ensure that you are encompassing everything your need to without losing words to the ether.

To use this tool, you will check the right tab, and click

Page titles -> Over 60 characters or for descriptions;

Page Description -> Over 155 characters.

This trick will not only improve your CTR, but will make your site just a little cleaner, and that is something we should all be aiming for in our work.

6. The Title is The Same as H1 Tag

One of the more consistent mistakes that we see on websites is that the title is the same as the H1 on a page. This Is not only an issue of bad content, but it can lead to harm to the site and the overall ranking.

Screaming Frog makes it easy to identify these issues, and the program can provide you with a report. To create and access it, you will need to do the following:

Click on the right tab and click Page Titles -> Same as H1

7. Duplicate or Missing H1 Tags

Related to the issues we usually see in useful trick 7, a duplicate or missing H1 tag can be a significant concern. Having no tag, or a copy will impact your page structure for Google, and it can directly negatively influence rankings as soon as the crawlers spot it.

Screaming Frog ensures that you can spot this and will provide you with a report to detect these issues and fix it asap. To access this, you will need to do the following:

Click the right tab, and then click H1 -> Duplicate or Missing and the report will generate.

8. Orphan Pages

When you are building a site, sometimes you add pages and forget to delete the pages. These are what are called orphan pages and are not always in the sitemap. Orphan pages are still part of your website, and depending on the content, especially if it is duplicate content, can negatively affect your ranking.

Screaming Frog can highlight any pages that are not in the sitemap XML so you can either delete them or add them into the sitemap. For this check you will need to:

Click sitemap -> Orphan URLs.

9. Crawl Treemap

One of the biggest timesavers on Screaming Frog is the ability to visualize the structure of the entire website with a single click of the button. SEO and website owners have had to build out these treemaps manually, and it is a bit of a headache.

To showcase these crawl treemap, you will need to:

Click Site structure.

10. Compare Json Rendered and Text Only Version Crawl

Did you know that Google processes HTML, and then only when the Googlebot has resources available to process the content of the site will they then process your JavaScript content? That is right; all of that work that you have put into extra touches on JavaScript could be deferred for months. Thus, Screaming Frog empowers you to compare both the HTML and JavaScript sites to see what the Googlebot sees when it is crawling the site.

To check do this:

For the HTML

Run 1st crawl - Configuration - > spider - > Rendering - > Text Only

For the JavaScript

Run 2nd crawl - Configuration - > spider - > Rendering - > Javascript

Once these reports are done, you can then compare the two and find out what essential links that are on the JavaScript site will need to be moved over the HTML site. It can be a bit of a backbreaking process, but the benefits will be huge once the Googlebot arrives on your website!

Enhance Your Website

Understanding how to use Screaming Frog SEO Spider effectively is essential for maintaining a well-optimized and user-friendly website.

If you need expert assistance with optimizing your website and leveraging the full potential of Screaming Frog SEO Spider, don't hesitate to reach out to the professionals. Our team of SEO specialists is ready to help you achieve outstanding results. Contact Seologist today to take your website's performance to the next level.