Log File Analysis

Server logs show what bots actually do, not what we hope they do. The user supplies the log file; this skill turns it into crawl-behaviour findings and concrete fixes.

Inputs

• Required: path to an access log file (Apache/Nginx combined format; a week or more of traffic is ideal)
• Optional: a single bot to focus on, top-N depth for path reports

Data pulls

python scripts/log_analyzer.py --file access.log --pretty
python scripts/log_analyzer.py --file access.log --top 25
python scripts/log_analyzer.py --file access.log --bot Googlebot --pretty