Log file analysis
 

I have a variety of log files from disparate sources that I need to analyse for spurious activity.
For these various log files I need to look back a few months but only between a certain time period on any given day. I've had a look at Splunk and while I can use date ranges for start/end criteria, I can't figure out how to do what I need.

So, using Splunk, is there away to do what I need? Or is there a better tool?

TIA

Mark

Would some simple scripts in Python do the job? That's what I usually reach for for this kind of job.

Isn't standard practise these days to outsource this kind of thing to a team of low-paid Indians or Chinese? ;)

Seriously though, if Ian's suggestion of Python doesn't really register with you, post up whether you're using Linux or Windows along with a quick sample of the format that the time is displayed in in your logs, and I can send you the 'grep' or 'find' commands you'd need to extract the lines you want out of the logs.

If using BASH or KSH in a UNIX/LINUX setup, then a simple shell script can be used to interogate a flat file you describe.
Have a look into Grep, and even cut and awk commands.

If this is windows then a Command Line Batch file maybe able to be deployed.

Have heard of Splunk, I know of a certain Banking institution who are using this/were as the people who researched this product claimed it was a good way to keep log files in tune.

Thanks, all

Some of the logs have been CSV files so have used Excel/Access. For some of the others I have used WinGrep. Found a handy utility for viewing IAS logs too.

Still have to look at IronPort email and web logs plus a whole bunch of Unix stuff.

Haven't clue about Python.

Think you are going around the houses here. Unless you have multiple systems with various types of logs.
If the files are in a comma seperated format thyen why not port them all onto a single LINUX/UNIX server and then run a 1 off script for the analysis? A basic shell script can do this for you. You could alway automate with an SFTP/FTP to bring the files over and then call the script to do what you want.

Python is a type of programming language, there are others like PERL. But for what you want to do then a simple shell script in LINUX/UNIX will suffice. No need to go to complicated unless you need to accomplish complicated things :)

Yep, multiple systems; Windows event logs, IIS, RAS, NPS, SharePoint, IronPort, Unix and some Cisco stuff and probably a few others.

I don't know Linux, born and bred Windoze :)

Fair enough. Maybe worth getting them all into 1 place and looking at what can be done with MS Excel. Maybe the reporting tool on excel could be of help.

Good luck.

If we're talking about basic shell scripts and 'grep', pretty much everything that can be done in Linux can also be done in Windows with the right batch file(s) and 'find'. It would just be a little less straightforward and a bit more ugly.

At the end of the day you're still looking for a string or pattern of strings, which is far from rocket-science.