Log file analysis
#1
Scooby Regular
Thread Starter
Join Date: Jun 2002
Location: Perth, Western Australia
Posts: 1,866
Likes: 0
Received 0 Likes
on
0 Posts
Log file analysis
I have a variety of log files from disparate sources that I need to analyse for spurious activity.
For these various log files I need to look back a few months but only between a certain time period on any given day. I've had a look at Splunk and while I can use date ranges for start/end criteria, I can't figure out how to do what I need.
So, using Splunk, is there away to do what I need? Or is there a better tool?
TIA
Mark
For these various log files I need to look back a few months but only between a certain time period on any given day. I've had a look at Splunk and while I can use date ranges for start/end criteria, I can't figure out how to do what I need.
So, using Splunk, is there away to do what I need? Or is there a better tool?
TIA
Mark
#3
Scooby Regular
iTrader: (11)
Isn't standard practise these days to outsource this kind of thing to a team of low-paid Indians or Chinese?
Seriously though, if Ian's suggestion of Python doesn't really register with you, post up whether you're using Linux or Windows along with a quick sample of the format that the time is displayed in in your logs, and I can send you the 'grep' or 'find' commands you'd need to extract the lines you want out of the logs.
Seriously though, if Ian's suggestion of Python doesn't really register with you, post up whether you're using Linux or Windows along with a quick sample of the format that the time is displayed in in your logs, and I can send you the 'grep' or 'find' commands you'd need to extract the lines you want out of the logs.
#4
If using BASH or KSH in a UNIX/LINUX setup, then a simple shell script can be used to interogate a flat file you describe.
Have a look into Grep, and even cut and awk commands.
If this is windows then a Command Line Batch file maybe able to be deployed.
Have heard of Splunk, I know of a certain Banking institution who are using this/were as the people who researched this product claimed it was a good way to keep log files in tune.
Have a look into Grep, and even cut and awk commands.
If this is windows then a Command Line Batch file maybe able to be deployed.
Have heard of Splunk, I know of a certain Banking institution who are using this/were as the people who researched this product claimed it was a good way to keep log files in tune.
#5
Scooby Regular
Thread Starter
Join Date: Jun 2002
Location: Perth, Western Australia
Posts: 1,866
Likes: 0
Received 0 Likes
on
0 Posts
Thanks, all
Some of the logs have been CSV files so have used Excel/Access. For some of the others I have used WinGrep. Found a handy utility for viewing IAS logs too.
Still have to look at IronPort email and web logs plus a whole bunch of Unix stuff.
Haven't clue about Python.
Some of the logs have been CSV files so have used Excel/Access. For some of the others I have used WinGrep. Found a handy utility for viewing IAS logs too.
Still have to look at IronPort email and web logs plus a whole bunch of Unix stuff.
Haven't clue about Python.
#6
Thanks, all
Some of the logs have been CSV files so have used Excel/Access. For some of the others I have used WinGrep. Found a handy utility for viewing IAS logs too.
Still have to look at IronPort email and web logs plus a whole bunch of Unix stuff.
Haven't clue about Python.
Some of the logs have been CSV files so have used Excel/Access. For some of the others I have used WinGrep. Found a handy utility for viewing IAS logs too.
Still have to look at IronPort email and web logs plus a whole bunch of Unix stuff.
Haven't clue about Python.
If the files are in a comma seperated format thyen why not port them all onto a single LINUX/UNIX server and then run a 1 off script for the analysis? A basic shell script can do this for you. You could alway automate with an SFTP/FTP to bring the files over and then call the script to do what you want.
Python is a type of programming language, there are others like PERL. But for what you want to do then a simple shell script in LINUX/UNIX will suffice. No need to go to complicated unless you need to accomplish complicated things
#7
Scooby Regular
Thread Starter
Join Date: Jun 2002
Location: Perth, Western Australia
Posts: 1,866
Likes: 0
Received 0 Likes
on
0 Posts
Yep, multiple systems; Windows event logs, IIS, RAS, NPS, SharePoint, IronPort, Unix and some Cisco stuff and probably a few others.
I don't know Linux, born and bred Windoze
I don't know Linux, born and bred Windoze
Trending Topics
#8
Good luck.
#9
Scooby Regular
iTrader: (11)
At the end of the day you're still looking for a string or pattern of strings, which is far from rocket-science.
Thread
Thread Starter
Forum
Replies
Last Post
Sam Witwicky
Engine Management and ECU Remapping
17
13 November 2015 10:49 AM