Server log analysis for SEO

This 5-step guide will show you exactly how to conduct such an analysis to make SEO decisions based on hard data, not guesswork.

Step 1. Get access to raw server logs

Server logs are text files that record every single request made to the server. Before you can begin your analysis, you need to get these files.

Where to find them? The location depends on your server configuration, but you will most commonly find them in these paths:
- Apache - /var/log/apache2/access.log or /var/log/httpd/access_log
- Nginx - /var/log/nginx/access.log
How to download them?
- Hosting Panel (cPanel, Plesk, etc.) - most hosting providers have a section called "Raw Access Logs," "Logs," or "Metrics" where you can download your access.log files, often compressed in a .gz format.
- SSH or FTP Access - if you have direct server access, you can connect via an FTP client (like FileZilla) or SSH and download the files directly from the locations mentioned above.

You need the raw access logs (access.log), not the error logs (error.log). These files can be very large (several gigabytes for popular sites), so ensure you have enough disk space.

Choose and configure an analysis tool

Manually reading millions of lines in a text file is impossible. You need specialized software to process this data and present it in a readable format.

Screaming Frog SEO Log File Analyser (Paid, with a free version) - this is the industry standard. It's relatively inexpensive and incredibly powerful. The free version allows you to analyze up to 1,000 log lines, which is enough to get familiar with the tool.
Other options:
- SEOlyzer (Paid, SaaS) - a cloud-based tool that monitors your logs in real-time.
- ELK Stack (Free, Advanced) - the combination of Elasticsearch, Logstash, and Kibana is a powerful, free solution, but its configuration requires significant technical expertise.
- Python/R Scripts (Free, for developers) - you can write your own custom script to parse and analyze the logs.

Example of script in Python

Here's the Python code. You can save this as a .py file e.g., seo_log_analyzer.py

Python

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

import re

import socket

from collections import Counter

import concurrent.futures

import os # Added for path handling and file existence checks

# --- Configuration ---

# Specify the path to your log file

# Example: LOG_FILE_PATH = 'C:/logs/access.log' or '/var/log/apache2/access.log'

LOG_FILE_PATH = 'access.log'

# Number of top URLs to display in the report

TOP_URLS_COUNT = 20

# Regular expression to parse standard Apache/Nginx log format

# Format: IP - - [timestamp] "Method URL Protocol" Status_Code Size "Referer" "User-Agent"

LOG_REGEX = re.compile(

r'(?P<ip>[\d\.]+) - - \[(?P<timestamp>.*?)\] '

r'"(?P<method>\S+) (?P<url>\S+) (?P<protocol>.*?)" '

r'(?P<status_code>\d{3}) (?P<size>\S+) '

r'"(?P<referer>.*?)" "(?P<user_agent>.*?)"'

)

# Cache to store IP verification results to avoid repeated DNS queries

verified_ips = {}

def is_verified_googlebot(ip):

"""

Checks if a given IP address belongs to Googlebot via reverse DNS lookup.

This is critical to avoid analyzing requests from spoofed bots.

"""

if ip in verified_ips:

return verified_ips[ip]

try:

# Perform reverse DNS lookup to get the hostname

hostname = socket.gethostbyaddr(ip)[0]

# Perform a forward DNS lookup on the hostname to get IP addresses

# This double check ensures the IP and hostname are legitimate

if hostname.endswith(('.googlebot.com', '.google.com')):

# Get all IPs for the hostname and check if the original IP is among them

for resolved_ip in socket.gethostbyname_ex(hostname)[2]:

if resolved_ip == ip:

verified_ips[ip] = True

return True

verified_ips[ip] = False

return False

except (socket.herror, socket.gaierror):

# Handle cases where reverse DNS lookup fails (e.g., non-existent domain)

verified_ips[ip] = False

return False

def analyze_log_file(log_path):

"""

Main function to analyze the log file, parse lines, filter for Googlebot,

verify its authenticity, and aggregate key data.

"""

googlebot_requests = []

if not os.path.exists(log_path):

print(f"ERROR: File '{log_path}' not found. Please check the path.")

return None

print(f"Starting analysis of file: {log_path}...")

try:

with open(log_path, 'r', encoding='utf-8', errors='ignore') as f: # 'errors=ignore' handles malformed characters

for line in f:

# Preliminary filter for lines containing "Googlebot" for efficiency

if 'Googlebot' in line:

match = LOG_REGEX.match(line)

if match:

googlebot_requests.append(match.groupdict())

except Exception as e:

print(f"An unexpected error occurred while reading the file: {e}")

return None

if not googlebot_requests:

print("No requests from Googlebot (or lines containing 'Googlebot') found in the file.")

return None

print(f"Found {len(googlebot_requests)} potential Googlebot requests. Starting IP verification (this may take some time for large files)...")

verified_requests = []

# Use multiple threads to speed up DNS queries, which can be slow for many IPs

with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor: # Increased max_workers for potentially faster processing

# Map each IP verification task to an executor, associating it with the original request

future_to_ip = {executor.submit(is_verified_googlebot, req['ip']): req for req in googlebot_requests}

for future in concurrent.futures.as_completed(future_to_ip):

req = future_to_ip[future]

try:

if future.result(): # If verification is successful, add to verified requests

verified_requests.append(req)

except Exception as exc:

print(f"IP verification generated an exception for {req['ip']}: {exc}")

print(f"Successfully verified {len(verified_requests)} requests from authentic Googlebot IPs.")

if not verified_requests:

print("No verified Googlebot requests found after IP verification.")

return None

# Data analysis on verified requests

crawled_urls = Counter(req['url'] for req in verified_requests)

status_codes = Counter(req['status_code'] for req in verified_requests)

return {

"total_googlebot_hits": len(googlebot_requests),

"verified_googlebot_hits": len(verified_requests),

"top_urls": crawled_urls.most_common(TOP_URLS_COUNT),

"status_codes": status_codes

}

def print_report(results):

"""

Prints a formatted, human-readable report from the analysis results.

"""

if not results:

return

print("\n" + "=" * 40)

print(" SEO LOG ANALYSIS REPORT ")

print("=" * 40)

print(f"Total 'Googlebot' User-Agent hits: {results['total_googlebot_hits']}")

print(f"Number of **verified** Googlebot hits: {results['verified_googlebot_hits']}")

print("-" * 40)

print("\n## Googlebot Response Status Codes Breakdown:")

status_summary = Counter()

for code, count in sorted(results['status_codes'].items()):

category = f"{code[0]}xx" # e.g., 2xx (Success), 4xx (Client Error)

status_summary[category] += count

print(f" - Code {code}: {count} times")

print("\nStatus code category summary:")

for category, count in sorted(status_summary.items()):

print(f" - Category {category}: {count} times")

print("-" * 40)

print(f"\n## TOP {TOP_URLS_COUNT} Most Crawled URLs by Googlebot:")

if results['top_urls']:

for i, (url, count) in enumerate(results['top_urls'], 1):

print(f"{i:2d}. {url} ({count} times)")

else:

print("No specific URLs found in verified Googlebot requests.")

print("-" * 40)

print("\nActionable Insights:")

print(" - High 4xx/5xx codes suggest broken internal links or server issues. Investigate these URLs.")

print(" - If Googlebot frequently crawls low-value URLs (e.g., filtered results, old content), consider using robots.txt or canonical tags to manage crawl budget.")

print(" - Pages with low crawl frequency but high importance might need improved internal linking or sitemap updates.")

print("=" * 40)

# --- Script Execution ---

if __name__ == "__main__":

# The following block is for testing purposes.

# When using with your actual log file, ensure this block is removed or commented out.

# It creates a dummy access.log file to demonstrate functionality.

dummy_log_data = """

66.249.75.132 - - [26/Sep/2025:12:05:11 +0200] "GET /home-page HTTP/1.1" 200 15214 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

123.123.123.123 - - [26/Sep/2025:12:05:12 +0200] "GET /secret-data HTTP/1.1" 200 800 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.135 - - [26/Sep/2025:12:05:13 +0200] "GET /products/new-model HTTP/1.1" 200 2345 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.132 - - [26/Sep/2025:12:05:14 +0200] "GET /old-article HTTP/1.1" 404 300 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

192.168.1.1 - - [26/Sep/2025:12:05:15 +0200] "GET /home-page HTTP/1.1" 200 15214 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

66.249.75.135 - - [26/Sep/2025:12:05:16 +0200] "GET /products/new-model HTTP/1.1" 200 2345 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.132 - - [26/Sep/2025:12:05:17 +0200] "GET /home-page HTTP/1.1" 200 15214 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.132 - - [26/Sep/2025:12:05:18 +0200] "GET /assets/style.css HTTP/1.1" 200 5000 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

66.249.75.132 - - [26/Sep/2025:12:05:19 +0200] "GET /api/data HTTP/1.1" 503 100 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

"""

# Write the dummy data to the specified log file path

with open(LOG_FILE_PATH, 'w') as f:

f.write(dummy_log_data)

analysis_results = analyze_log_file(LOG_FILE_PATH)

print_report(analysis_results)

# Clean up the dummy log file after analysis

try:

os.remove(LOG_FILE_PATH)

print(f"\nCleaned up dummy log file: {LOG_FILE_PATH}")

except OSError as e:

print(f"Error removing dummy log file: {e}")

How to use the script

Prerequisites:

Python - ensure Python 3.x is installed on your system.
Log file - obtain your access.log file(s) from your web server (Apache, Nginx). You can usually find these in /var/log/apache2/ or /var/log/nginx/ via SSH/FTP, or download them from your hosting panel's "Raw Access Logs" section.
Placement - place your access.log file(s) in the same directory as your Python script, or update the LOG_FILE_PATH variable to point to its exact location.

Save the Script - save the code above as a Python file (e.g., seo_log_analyzer.py).

Prepare your log file:

If your log file is compressed (e.g., access.log.gz), you'll need to decompress it first. You can use tools like gunzip on Linux/macOS or 7-Zip on Windows.
If you have multiple log files (e.g., access.log.1, access.log.2), you can concatenate them into one large file for a comprehensive analysis using cat access.log.* > combined_access.log (Linux/macOS) or manually combine them. Then, update LOG_FILE_PATH to point to this combined file.

Configure LOG_FILE_PATH:

Open seo_log_analyzer.py in a text editor.
Locate the LOG_FILE_PATH variable.
Change 'access.log' to the actual name/path of your log file. For example: LOG_FILE_PATH = 'my_website_access_logs.log' or LOG_FILE_PATH = '/path/to/your/logs/access.log'.

Remove dummy data block:

The if __name__ == "__main__" - block at the bottom of the script contains dummy data creation for testing. For analyzing your real logs, you must remove or comment out the lines that create the dummy_log_data and write it to LOG_FILE_PATH. Keep only the analysis_results = analyze_log_file(LOG_FILE_PATH) and print_report(analysis_results) lines.
The cleanup os.remove(LOG_FILE_PATH) at the end should also be removed if you're working with your actual log files, as you don't want to delete them.

Run the script:

Open your terminal or command prompt.
Navigate to the directory where you saved seo_log_analyzer.py and your log file.
Execute the script using: python seo_log_analyzer.py

Interpreting the report & taking action

The script will output a detailed report directly to your console, similar to this:

Starting analysis of file: access.log...
Found 8 potential Googlebot requests. Starting IP verification (this may take some time for large files)...
Successfully verified 7 requests from authentic Googlebot IPs.

========================================
        SEO LOG ANALYSIS REPORT        
========================================
Total 'Googlebot' User-Agent hits: 8
Number of **verified** Googlebot hits: 7
----------------------------------------

## Googlebot Response Status Codes Breakdown:
  - Code 200: 5 times
  - Code 404: 1 times
  - Code 503: 1 times

Status code category summary:
  - Category 2xx: 5 times
  - Category 4xx: 1 times
  - Category 5xx: 1 times
----------------------------------------

## TOP 20 Most Crawled URLs by Googlebot:
 1. /home-page (2 times)
 2. /products/new-model (2 times)
 3. /old-article (1 times)
 4. /assets/style.css (1 times)
 5. /api/data (1 times)
----------------------------------------

Actionable Insights:
  - High 4xx/5xx codes suggest broken internal links or server issues. Investigate these URLs.
  - If Googlebot frequently crawls low-value URLs (e.g., filtered results, old content), consider using robots.txt or canonical tags to manage crawl budget.
  - Pages with low crawl frequency but high importance might need improved internal linking or sitemap updates.
========================================

Configuration in Screaming Frog SEO Log File Analyser:

Launch the program and create a new project.
Drag and drop your .log or .gz file(s) into the program window.
The tool will automatically start processing the data. In the "User Agents" tab, you'll see a list of all bots that have visited your site.

Identify and Verify Googlebot's Activity

Not every request with a "Googlebot" User-Agent actually comes from Google. Malicious bots often spoof their user agent to bypass security measures. That's why verification is critical.

In the Screaming Frog Log File Analyser, navigate to the "Bots" tab. The tool automatically performs a reverse DNS lookup to verify if the request's IP address truly belongs to Google. You will see a breakdown of "Verified Bots" and "Spoofed Bots." For your analysis, only consider the verified bots.

Analyze key data – What is Googlebot actually doing?

This is the core of the entire process. You must now interpret the data to understand the bot's behavior. Focus on the following reports and metrics:

Most Crawled URLs (URLs -> All URLs):
- What to check? Sort the list by "Total Bot Hits." Are your key pages (products, categories, articles) at the top, or is it low-value pages (e.g., URLs with sorting parameters, old versions, login pages)?
- Implication - if Googlebot is spending time on unimportant resources, it's wasting the crawl budget that could have been spent on indexing your valuable content.
Server Response Codes (Response Codes):
- What to check? Pay close attention to response codes other than 200 OK. How many 404 Not Found or 5xx Server Error codes are there? Is the bot frequently hitting redirects (301 or 302)?
- Implication - a high number of 404s indicates problems with internal linking or improperly removed pages. 5xx errors are a critical issue that can lead to de-indexing.
Crawl Waste:
- What to check? Review the list of pages that shouldn't be indexed but are being crawled regularly by Googlebot. These could be URLs with parameters (?color=red), internal search results pages, or non-canonical versions of pages.
- Implication - this is direct evidence of a wasted crawl budget. These URLs should be blocked or have their canonical versions specified.
Crawl Frequency (Events):
- What to check? See how often Googlebot visits your site over a day/week. Is there any correlation between publishing new content and a spike in bot activity? How often are your most important pages re-crawled?
- Implication - if key pages that you update frequently are rarely visited, it could signal issues with their accessibility or poor internal linking.

Take concrete actions based on your analysis

The analysis itself is worthless without implementation. Here is a table of common problems and their solutions:

Problem Identified in Logs	Specific Action to Take
Googlebot frequently hits pages that return a 404 error.	1. Identify these URLs. 2. If they have valuable replacements, set up 301 redirects. 3. Fix the internal links that point to these broken pages.
The bot wastes time on URLs with parameters (e.g., `?sort=price`).	1. Block these parameters in your `robots.txt` file using the `Disallow` directive. 2. Use the `rel="canonical"` tag to point to the "clean" version of the URL.
The most important business pages are rarely crawled.	1. Increase the number of internal links pointing to these pages. 2. Ensure they are in your `sitemap.xml` with a high priority.
5xx server errors appear in the logs.	1.Immediately contact your server administrator or hosting company. 2. Analyze the `error.log` files to diagnose the cause.
The bot is crawling non-canonical versions of pages (e.g., with and without `www`).	1. Implement server-level 301 redirects to force a single, preferred version of your domain.

Server log analysis is one of the most powerful techniques in a technical SEO's toolkit. It allows you to stop guessing and start acting based on hard, undeniable evidence. It shows you where you're losing money, where Google is encountering problems, and which elements of your site require immediate attention. Dedicate one day to this analysis, and you'll get a concrete to-do list that will yield far better results than months of "creative" marketing.

Step 1. Get access to raw server logs

Choose and configure an analysis tool

Example of script in Python

How to use the script

Interpreting the report & taking action

Configuration in Screaming Frog SEO Log File Analyser:

Identify and Verify Googlebot's Activity

Analyze key data – What is Googlebot actually doing?

Take concrete actions based on your analysis

Keep exploring

How to use Reddit for SEO and measure the results

Why Search Console’s sitemap API can show zero indexed pages

Why removing noindex with JavaScript can leave a page out of Google