[one-liner]: Pulling Usage Data out of Apache’s access_log

Posted on July 22, 2009 by News

The other day a website I maintain started experiencing what appeared to be a DoS attack. When this occurs I usually take a peek at Apache’s access_log to see if there is an “unusual” amount of traffic coming from a set of IP addresses. A DoS can be classified as one of 2 situations:

  • a lot of page hits coming from the same IPs
  • a lot of IPs hitting the same URL

The first situation is easy to diagnose with a one-liner like this:

1
2
3
4
5
6
7
8
9
10
11
12

# displays the top 10 IP addresses along with there frequency counts
% cut -d" " -f1 access_log | sort -n | uniq -c | sort -nr | head
31123 216.246.75.191
20922 204.2.196.164
20746 204.2.196.177
17723 216.246.75.202
14762 165.254.127.134
13967 165.254.127.127
13718 206.57.29.168
11670 206.57.29.174
8099 96.17.161.211
7264 96.17.161.207

The second situation requires a little bit more complex command but it’s kinda sorta doable. At least to the point that gives you a warm fuzzy.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# display the top 20 URLs by IP requests
# columns in the output are: (frequency, URL, IP)
 
% cat access_log | awk '{print $7, $1}' | sort | uniq -c | sort -rn | head -20
 729 /globe/mul/webAnalytics/cj_metrics.js 96.17.161.211
 714 /globe/mul/webAnalytics/cj_metrics.js 216.246.75.191
 604 /globe/mul/webAnalytics/cj_metrics.js 96.17.161.207
 526 /globe/mul/webAnalytics/cj_metrics.js 216.246.75.202
 312 /globe/mul/webAnalytics/cj_metrics.js 12.182.252.217
 259 /ejnac/webAnalytics/metrics.js 216.246.75.191
 251 /ejnac/webAnalytics/metrics.js 96.17.161.211
 234 /ejnac/PageQuery.jhtml?pq-path=3316/13061/15193&pq-locale=en_US 216.246.75.202
 227 /ejnac/webAnalytics/metrics.js 96.17.161.207
 202 /global/mul/webAnalytics/cj_metrics.js 12.182.252.212
 192 /ejnac/webAnalytics/metrics.js 216.246.75.202
 189 /ejnac/PageQuery.jhtml?pq-path=1234/43067/45141&pq-locale=en_US 12.182.252.212
 175 /eknec/PageQuery.jhtml?pq-path=1234/43064/14194&pq-locale=en_US 216.246.75.191
 161 /akamai-sure-toast-object.html 216.246.75.191
 151 /globe/mul/metrics/metrics.js 12.182.252.217
 148 /globe/mul/metrics/metrics.js 216.246.75.191
 140 /globe/mul/webAnalytics/cj_metrics.js 209.170.118.220
 140 /globe/mul/metrics/metrics.js 12.182.252.212
 137 /ejnac/PageQuery.jhtml?pq-path=1234/15066/16197&pq-locale=en_US 12.182.252.217
 120 /globe/mul/metrics/metrics.js 216.246.75.202

After a while of running these types of commands against your access_logs, you’ll start to develop a feel for what’s normal, and what’s not.

NOTE: For further details regarding my one-liner blog posts, check out my one-liner style guide primer.

This entry was posted in apache, monitoring, one-liner, server, Syndicated, tip, tips & tricks. Bookmark the permalink.

Comments are closed.