help parsing log

[phynias]

i have a modsecurity log that i want to pull a couple things from. here is an example line

[05/Mar/2008:12:50:07 --0800] [www.mydomain.com/sid#1ef44178][rid#2042fa58][//member/index.php][2] Warning. Match of "rx ^OPTIONS$" against "REQUEST_METHOD" required. [file "/usr/local/apache2/modsec/modsecurity_crs_21_protocol_anomalies.conf"] [line "41"] [id "960015"] [msg "Request Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"]

so i want to print out the id, domain,path, and tag from lines that only have CRITICAL in them.
so from the above i want
960015 | www.mydomain.com | /member/index.php | PROTOCOL_VIOLATION/MISSING_HEADER

i am really bad at regular expressions and can only do simple things. if anyone can help and whip something up, i would appreciate it.

[eggi]

Hey There,

Here's a cheap hack:

awk -F"[" '{print $3 " " $5 " " $9 " " $12}' LOGFILE|sed 's/^\([^\/]*\)\/[^\/]*\(\/[^]]*\)][^\"]*\"\([^\"]*\)\"[^\"]*\"\([^\"]*\)\".*/\3\|\1\|\2\|\4/'

If you always get the double /'s on your directory path, just make this slight change:

awk -F"[" '{print $3 " " $5 " " $9 " " $12}' LOGFILE|sed 's/^\([^\/]*\)\/[^\/]*\/\(\/[^]]*\)][^\"]*\"\([^\"]*\)\"[^\"]*\"\([^\"]*\)\".*/\3\|\1\|\2\|\4/'

Hope that helps. It's kind of confusing, but makes sense if you just step through it :)

, Mike

[ghostdog74]

awk  'BEGIN{FS="[]].[[]|[[]|[]]"}
{
 gsub(/id|\"/,"",$13)
 split($3,a,"/")
 gsub("tag","",$16)
 print $13,a[1],$7,$16
}' file

[manik007]

I am trying to create a shell script which should parse the logs from SystemOut.log file for a certain pattern for eg. Timeout.

i am creating this shell in my home directory which is /home/manik/
and the Systemout.log file location is /opt/---/---/NodeA/logs/Server_name/SystemOut.log

can anyone help me create this shell script

Thank you in advance..!! :)

[JeoSaurus]

Hi Manik, you might want to start a new thread, since this was was solved back in March :)

If this isn't what you're looking for, please start a new thread with more details. From what you've given us so far, it sounds like you just need to do something like: grep Timeout /opt/---/---/NodeA/logs/Server_name/SystemOut.log Not very exciting, but should do the job unless you need it to do more.

Hope this helps!
-G

[manik007]

i tried to do that but it will display me the details of the SystemOut.log file on the page but i want to capture all line which matches the pattern (TimeOut), there are more then one SystemOut.log file and i want to print out the details of all the files and save it in a folder in my home directly with the file name they are coming from.

[manik007]

#!/bin/sh
T1=`grep -i SocketTimeout /opt/---/---/NodeA/logs/Server_Name/SystemOut.log`
T2=`cat /opt/---/---/NodeA/logs/Server_Name/SystemOut.log | wc -l`
T3=$(($T2 - $T1))
tail $T3 /opt/---/---/NodeA/logs/Server_Name/SystemOut.log > 123

I tried to create somthing like this but it gave me some errors , dosen't seem to work

[eggi]

Hey There,

Maybe something along the lines of:

find /opt/---/---/NodeA/logs/Server_name/ -name "*SystemOut*"|xargs -I var grep -i Timeout "var" /dev/null >DUMPFILE 2>&1

you need the extra /dev/null in the xargs grep, so that the filename will show up in the grep output. This code isn't exact, but should give you the general idea.

Hope that helps :)

, Mike

[manik007]

Can you please explain me the DUMPFILE i could not understand that......i am really new to this and i want to understand the whole thing...

Thank you