Splunk Log Analysis with SSH logs
I accidentally captured a pile of server logs into analytics during the attacks on linode infrastructure over the holidays. All my logs, including the SSH logs, are dumping into Splunk Light. I covered installing Splunk on CentOS in a recent post. When log files are too big to eyeball from the terminal, Splunk is great.
Normally, I block port 22 (sshd) in the server except for whitelisted IPs. For fun, I opened 22 back up for a while. With a splunk trap to email me anytime a user logs into ssh, and a hardened sshd config, I felt pretty safe about this.
I'll allow splunk to visualize what happened next:
It's easy to see my blog underwent a large, indiscriminate dictionary attack. This peaked on Dec 21 with 33,772 log entries. I'd estimate that would amount to \~10,000 login attempts that day, with similar volume through Dec 26th. The rest of the timeline graph looks dead by comparison.
In my selected search, there were \~54,000 root login attempts. Earlier, many usernames were attempted. Whoever wants into this box right now just cares about root.
However, if I zoom on to Just the last week, a similar pattern appears. Today, the logs show only \~1500 login attempts.
For more fun, the patterns tab gives us helpful analysis of the hundreds of thousands of ssh log events:
That "reverse mapping checking getaddrinfo for..." gives us something interesting to google... where i find a reminder about a tool called fail2ban. Would be useful if you didn't firewall out ssh traffic to the world.
My first instinct was to write a perl script to get some stats out of the logs that I wanted. I did this first and played with it for a few hours. Careful, I could write sysadmin scripts all night and miss sleep. But could I get the same info out of Splunk? It took less than five minutes to get a list of usernames used in failed login attempts, along with a count of how many times each username was attempted. This data cleanly exported directly to csv. What a neat trick!
To pull the data:
- I Enter the search term: host="mitchjacksontech.com" sshd authentication failure. This lists attempts against valid usernames.
- Or enter the search term: host="mitchjacksontech.com" sshd invalid user. This displays attempts against nonexistent usernames.
- Click on the field "user" under "interesting fields" on the left sidebar.
- This brings up a count of top usernames used to attempt a login.
- Click on "top values" above this list, and a full report is generated.
- This report can then be exported to CSV
- Any report you create in the browser can be saved. The visualizations can be added to your splunk dashboard. Saved reports can be automatically emailed to you periodically or with set triggers. Choose your report format.
The 4th most common attempted username is 'linode.' No surprise. This data was generated preceeding the holiday linode attack, where compromised linode servers were used to attack the linode network.
The complete list of usernames is listed in this post.
At this point, I'm sold on Splunk. I didn't expect to have an opportunity to evaluate it during a large-scale network attack on the data center hosting this server, but I'm enjoying it a bit. I feel for those servers that fall on the sword to such simplistic attach methods. I'm excited to see what other kinds of reports I can have splunk generate.