Filtering a text file and counting the occurance

[kavithanb]

Hi All,

I am trying to write a shell script that filter java exceptions along with stack trace from a log file and also it has to count how many times each exception has occurred in the log file. Please help me in this.

The resulting file should be like

Number of times(Count) Exception1 the exception has occurred Stack trace( Log like NullPointerException)

Number of times(Count) Exception2 the exception has occurred Stack trace(Log like NameNotFoundException)

Please help me on this

Thanks in advance

Kavitha

[JeoSaurus]

This sounds interesting Kavitha!

What have you tried so far?
Do you have any example text so that we can see what the log looks like?

I worked on something similar recently, but I didn't go as far as getting the entire stack trace (that part might be a little trickier). Finding and counting the exceptions should be fairly simple to do.

[kavithanb]

Hi,
My log file looks similar like
2006-01-07 18:39:18,212 host123 WARN com.host123 .elf.UserQuest - Quest/option {o.q.more.paper.osc#0} references unknown dependent {t.what.form.file.more.action} in application {src-code}. Please revise.
2006-01-07 18:39:18,212 host123 WARN com.host123 .elf.UserQuest - Quest/option {o.q.more.paper.osc#1} references unknown dependent {t.what.form.file.more.action} in application {src-code}. Please revise.
2006-01-07 18:40:34,281 cessor32 ERROR com.host123 .email.DirectMailer - Unable to connect to server {1.1.1.1}:
javax.mail.MessagingException: Could not connect to SMTP host: 1.1.1.1, port: 25, response: 451
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:996)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:197)
at javax.mail.Service.connect(Service.java:233)
at javax.mail.Service.connect(Service.java:134)
at com.host123.email.DirectMailer.deliverMessage(DirectMailer.java:191)
at com.host123.email.DirectMailer.send(DirectMailer.java:153)
at com.host123.webface.util.Notifications.sendEmailX(Notifications.java:126)
at com.host123.webface.util.Notifications.sendEmail(Notifications.java:91)
at com.host123.webface.util.Notifications.sendEmail(Notifications.java:145)
at com.host123.edp.webface.action.DocRecoveryActionProcessor.perform(DocRecoveryActionProcessor.java:81 )
2006-01-07 18:43:45,811 host123 WARN com.host123.webface.RequestProcessorImpl - beanId = 7061279 beanTag= Docket , bean is null or unaccessible
2006-01-07 18:43:46,029 host123 WARN com.host123.webface.RequestProcessorImpl - beanId = 7061279 beanTag= Docket , bean is null or unaccessible

My results should be like

For eg If the log file has below exception thrice , result should be like

3-2006-01-07 18:40:34,281 cessor32 ERROR com.host123 .email.DirectMailer - Unable to connect to server {1.1.1.1}:
javax.mail.MessagingException: Could not connect to SMTP host: 1.1.1.1, port: 25, response: 451
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:996)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:197)
at javax.mail.Service.connect(Service.java:233)
at javax.mail.Service.connect(Service.java:134)
at com.host123.email.DirectMailer.deliverMessage(DirectMailer.java:191)
at com.host123.email.DirectMailer.send(DirectMailer.java:153)
at com.host123.webface.util.Notifications.sendEmailX(Notifications.java:126)
at com.host123.webface.util.Notifications.sendEmail(Notifications.java:91)
at com.host123.webface.util.Notifications.sendEmail(Notifications.java:145)
at com.host123.edp.webface.action.DocRecoveryActionProcessor.perform(DocRecoveryActionProcessor.java:81 )

Till now i am able to filter error exceptions along with stack trace using egrep command
egrep "ERROR|^[ ]*java|^[ ]*at" log file

Counting the exceptions along with stack trace is bit difficult.
I have done finding only exceptions and counting them but i want to display stack trace also. This is scratching my head.

This sounds interesting Kavitha!

What have you tried so far?
Do you have any example text so that we can see what the log looks like?

I worked on something similar recently, but I didn't go as far as getting the entire stack trace (that part might be a little trickier). Finding and counting the exceptions should be fairly simple to do.

[JeoSaurus]

Thanks for the additional information! Do you need the count printed before each line, or do you just want a total at the end?

What kind of output do you need? Just print to STDOUT, or write to a log file?

[kavithanb]

Hi ,

I want the count of each exception either at the beginning or at the end of exception.

Output i am writing it to a new file.

Thanks for the additional information! Do you need the count printed before each line, or do you just want a total at the end?

What kind of output do you need? Just print to STDOUT, or write to a log file?

[JeoSaurus]

Okay, this script is not the prettiest (and I'm sure there's a pure sed way to do this...), but it might at least give you a good starting point... Here goes :)

logfile="test.txt"
i=1

grep ERROR $logfile | while read line
    do echo -n "$i - "
    sed -e :b -e '/'"$line"'/!d;n;:a' -e '/^[0-9]/bb' -e 'n;ba' $logfile
    i=$((i+1))
done

There's a good explanation for that 'sed' command here: http://www.catonmat.net/c/4323

I hope this helps!
-G