Parse IP addresses out of a web server log file

9 comments

1. This is horribly inefficient for large files or large amount of files.
   It would be better to do this on a line by line basis.

2. @Ayende,
   Perhaps, but it works great for 10-20 files. 🙂 Just enough to find out what IP addresses have accessed the server in the past few days.

3. You asked for it. 😉

   require ‘set’
   require ‘pp’

   results = SortedSet.new

   Dir[‘*.log’].each do |path|
   File.read(path).scan(
   /b(d{1,3}.d{1,3}.d{1,3}.d{1,3})b/
   ).flatten.each { |ip| results << ip }
   end

   pp results

   It took me 2 minutes to write this (I'm slow 😉 ). Not that you took any longer. I've just been thinking lately how it feels like I'm at the point where I can constantly throw away little solutions like this and not get wrapped up in reusability for a utility because they're one of the few things I can (almost) write faster in my head than I can type.

   That level of proficiency or approach to a problem doesn't get much press in programming circles it seems. Then again, not much more to say, so that's probably just because it's not a very interesting topic. Just thinking out loud here. 🙂

4. Walk, don’t run, to http://www.logparser.com, and query your logs like you would if they were SQL tables … 😉

5. Log parser is the way to go… especially when you are talking about large log files.

   http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en

6. 1 word: PowerShell! 🙂

7. Jeffrey — Ayende’s right here. The performance implications of the following lines are really terrible:

   string text = “”;
   …
   foreach (var filePath in filePaths) {
   text += File.ReadAllText(filePath);
   }

   You’re allocating:
   1.) A string to hold the entire contents of 1 text file
   2.) Another string that holds the entire contents read up to this point PLUS the data from step #1
   3.) And you’re repeating this for every log file

   Like he said, you could solve this by processing a file at a time, line-by-line, but if you don’t want to do that, you could improve things tremendously by using a StringBuild that you append to.

8. I can’t believe you haven’t heard of “x”. There are what? 600 different things that could have worked? You did one that you know well and wrote quickly, that’s what works well. There a couple things I don’t agree with.

   1. loading all of the files into a single combined text string. I would have opened one file, processed the matches then moved on to the next file.

   2. Writing and running this as a unit test. The test framework brings nothing to your solution. You aren’t using a setup or any assertions. You have a console application that you use NUnit to run. I’d have just written it as a console application.

   I probably would have also output the results to a file instead of just the console window, but that’s a very minor change.

9. I’m just left wondering what your intention with this post was…
   You knew there were better tools and method out there to do the job, and you knew people would call them out. So that rules out the “providing a solution for others to follow” motive.
   But then you seem a bit defensive when people suggest alternatives, so that rules out the “information gathering” motive.

Comments are closed.