Website Log Analysis with Screaming Frog SEO Log File Analyzer

Nowadays, few are able to analyze the site logs, and even more so manually (as was done before). But sometimes it is necessary to do this and even very useful, because when working with logs you are faced with primary information: each visit to the site by a robot and a person is displayed in full in them.

What can they tell? Let’s look at an example. Here is what one line from the log looks like:

• Domain;
• The IP address from which the transition was made;
• the exact time the request was visited;
• the page itself that was visited by the crawler;
• response code that both robots and humans encounter;
• page size in bytes;
• user-agent – the name of the client that accessed the server, with which you can find out which browser and robot accessed the page.

Now imagine how the logs look like, which contain not a couple of visits, but several hundred or even thousands of robot visits.

It is very difficult to analyze such data manually, isn’t it?
I’m not saying it’s impossible. Just spend a lot of time on this analysis.

Fortunately, now there are already a lot of tools for analyzing site logs, both paid and free. After reviewing a few of them, I settled on a tool from Screaming Frog – SEO Log File Analyzer. It is to him that my today’s review is dedicated.

The tool is paid: 1 license – $99 per year. But there is also a free version, which has some limitations:

• 1 project;
• Only 1000 lines in the event log;
• There is no free technical support.

You can find out more information and download the program here.

If you think that the program will do everything for you, then you are mistaken. Yes, she will do most of the work, ie. will collect all the data in a heap and present them in a convenient and understandable form. And you will have the most important task – to analyze this data, draw conclusions and make recommendations.

What does this tool do anyway?

• Determines crawled URLs: Find out exactly which links Googlebot and other crawlers can crawl, when and how often.

• Shows Crawl Rate: Find out which pages are most frequently visited by search bots and how many URLs are crawled each day.
• Finds Broken Links and Errors: Shows all response codes, broken links, and errors that search engine crawlers encountered while crawling a site.

• Checks for redirects: finds temporary and permanent redirects encountered by search robots.
• Helps you optimize your crawl budget by showing you your most and least crawled URLs to identify wastage and improve crawl efficiency.
• Finds uncrawled URLs and orphan pages: you can import a list of URLs and compare them with log data to identify unknown pages or URLs that Googlebot is unable to process.
• Allows you to combine and compare any data.

So let’s get started. I think you won’t have any problems installing the program. Therefore, we proceed to the main stage – loading data.

You can import 2 types of data:

• log files;
• URL list (in excel format).

First, I’ll tell you how to import log files.

First of all, you need to get the log files (access.log) of your site from the server. They are often stored in the /logs/ or /access_logs/ folder, and via FTP you can download these files to your computer.

In order for you to have enough data for analysis, I would recommend using monthly log files.

After you have downloaded them, simply click the “Import” button and select the “Log file” item.

Further, after loading the data, you will need to come up with a name for the project (do it as it is convenient here, the main thing is to make it clear later) and select the time zone.

After that, the main window of the control panel will appear:

Here you can see at a glance which crawlers visited your site and how often, how many pages they visit, etc.

After that, you can move on to a deeper analysis. For example, you can determine which URLs are the slowest to load and which are the most frequently visited.

You can also see which response codes links give. To do this, go to the “Response Codes” tab, where you have the ability to filter URLs by code type. This way you can find bugs.

And I also discovered an interesting thing ☺ When you right-click on a link, the following options appear:

• Copy link;
• Open url in browser;
• Check for the presence of PS in the index (Google, Yahoo, Bing);
• View external links leading specifically to this page;
• Open robots.txt file, etc.

You can see the full list in the following screenshot:

Copying and opening a link is, of course, standard stuff. But to check for the presence in the index or with one button go to robots.txt – this is already interesting. And most importantly, very convenient.

Now I will tell you about the second import method – loading a list of URLs in excel format. Why is this necessary, you think?

The program has the ability to import URLs from various sources and combine them with a log file for more complex analysis. Once you’ve imported them, the program may show URLs that match or don’t match in those datasets.

With this comparison, you can analyze the site for a number of different questions:

• What pages on your site are visited most often? Or which pages were not visited by the robot at all?
• Have the robot visited all the URLs from the XML sitemap? If not, why not?
• How often are sitemap updates checked by the robot?
• When a page is changed, how long does it take from the second crawl until the search index is updated?
• How does new links affect crawl speed?
• How quickly will a newly launched site or site section be crawled?

And many other questions.

The potential for analysis is practically unlimited. By comparing any data, you can get meaningful insights into how your site interacts with search engine crawlers. And this, in turn, allows you to find problems on your site that other SEO tools could not easily find.

So, in general, we can say that log files contain a huge amount of information that can help you analyze the performance of your site and avoid some errors. And Screaming Frog SEO Log File Analyzer will help you with this analysis.