Parse and mail log files

[magikman]

hello,

I am trying to parse, format and mail some log files on a server. I was using shell scripts to parse the different log files and mail them out. However, the network has grown and as a result the log files are massive. And as a result of the larger log files, the shell scripts aren't cutting it. This brings me to my current issue.

I have never used Perl to write anything. I just started playing around with this idea this afternoon and have hit a wall and need some help. What i have so far:

#!/usr/bin/perl
#
use warnings;

open(LOG,"cmtslogs") or die "Unable to open logfile:$!\n";

        while(<LOG>) {


                if(($_  =~ /up/) || ($_ =~ /down/))
                {

                        print ($_);
                        $last_line = $_;
                }
}


        close(LOG);

This works well but isn't anywhere near what i want. I would like to set it up so it ignores multiple lines of the same log and instead prints out the number of times the line repeated. In addition, it should ignore certain parts of the log line. Here is an example of one of the log lines:

Dec 14 17:39:34 <RFC1918> 1864244: SLOT 8/1: Dec 14 17:40:33: %UBR10000-5-UNREGSIDTIMEOUT: CMTS deleted unregistered Cable Modem <MAC000.0000.0000>

I would like to print out the dates, times, device name and the actual log message.

[KevinADC]

There are a lot of logfile parsers alreay written on CPAN. I suggest you search there and see if there is one for the particular log file you are working with. If not, you need to tackle your parsing issues one problem at a time.

The code you posted will only print lines containing the substrings "up" or "down".

[magikman]

There are a lot of logfile parsers alreay written on CPAN. I suggest you search there and see if there is one for the particular log file you are working with. If not, you need to tackle your parsing issues one problem at a time.

The code you posted will only print lines containing the substrings "up" or "down".

I have already searched CPAN for something that would work, i found nothing. I know that the code above will only print out lines containing up or down, that's what i wanted it to do.

I have gotten a little bit further in what i want to do. However, the section where i am trying to print out the number of times a line repeated still isn't working.

#!/usr/bin/perl
#
use warnings;

open(LOG,"cpslogs") or die "Unable to open logfile:$!\n";

        while(<LOG>) {


                if(($_ =~ /Deny/) ||
                ($_ =~ /URL/))
                {
                        next;
                }


#               if($lastline = $_)
#               {
#                       $matched = 1;
#                       $count++;
#               }
#               else
##              {
#                       if($matched == 1)
#                       {
#                               print "repeated $count times";
#                               $count = 0;
##                              $matched = 0;
#                       }
#               }
#
#               if(($_  =~ /up/) ||
#               ($_ =~ /down/))
#               {
#
#                       print ($_);
#                       $lastline = $_;
#               }
                print;
}


        close(LOG);

[KevinADC]

first thing you must do, is start using the strict pragma:

use strict;

you already have "use warnings;" which would have been my next suggestion.

This line in your code is wrong:

if($lastline = $_)

You assign the value of $_ to $lastline, what you mean to do is check to see if they are the same, for that you would use a regexp (to check for patterns) or a string operator, probably "eq", to check for equivalence:

if($lastline eq $_)

[magikman]

Thank you for your help! I really appreciate it.

I have pretty much finished what i am going to do with this. Here's my code:

#!/usr/bin/perl
#
#
## Declare packages being used ##
use warnings;
use strict;
use Net::SMTP;
use MIME::Lite;

## Declare variables ##

my $logFile="/var/log/cpslogs";
my $tmpFile="/tmp/cpslogs.tmp";
my $Subject="CPS logs";

## Open input and output files ##
open(LOG,"$logFile") or die "Unable to open logfile:$!\n";
open(OUT,">$tmpFile") or die "Unable to open tmp file:$!\n";


## Call functions ##
&logFilter();
&sendMail();


## Function to send out mail ##
sub sendMail
{

         $msg = new MIME::Lite;

         $msg->build(
                Type    =>'text',
                Path    =>"cat $tmpFile |",
                ReadNow => 1
);
        $msg->add(From  =>"<email>");
        $msg->add(To    =>"$ARGV[0];");
        $msg->add(Subject => "$Subject");
        $msg->send('smtp','localhost',Debig=>1);

}

## Function to filter logs ##
sub logFilter
{
        while(<LOG>)
        {
                if(($_ =~ /Deny/) ||($_ =~ /URL/) ||($_ =~ /Denied/i) ||($_ =~ /Authentication failure for SNMP/) ||
                ($_ =~ /No matching connection for ICMP error message/))
                {
                        next;
                }

                print OUT;
        }

        close(LOG);
        close(OUT);
}

When using strict i know that you must either declare everything with the package::var or use my. However, with $msg, i can't seem to get it right. Any help would be greatly appreciated!

[KevinADC]

Because $msg is scoped to just the one block of code (sendMail), declare the object with 'my' when you create it:

my $msg = new MIME::Lite;

[KevinADC]

To make your code a bit more readable, these constructs:

($_ =~ /Deny/)

can be simplified to:

(/Deny/)

perl automatically binds regexps to $_. You have to actually bind regexps to any other scalar variables using the binding operators, like =~, !~, etc:

if ($foo =~ /foo/)

many perl programmers consider it bad to use $_ too much, named variables can make code easier to understand, but it your code it would be OK since the usage is rather obvious.

If you think using named variabls is not so important, that only indicates you have not written enough perl programs yet. ;)

[magikman]

Thank you for your help! If there are any other suggestions that you can think of, please post.