Ad:

Python Discussion Thread
Unsolved
Views: 667

Jul 28th, 2009

Query regarding Data mining and plotting graph...

Expand Post »

I have log file which gives CPU utilization and timestamp value at regular interval.

Logs snippet below:
==========LOGS==============

a:CPU [ 85%]: asdf asd 123 xyz
A: Ts 23086, Netvalue 3286, someothervalues 3456
abc abc
xyz xyz
a:CPU [ 75%]: asdf asd 123 xyz
A: Ts 24088, Netvalue 3266, someothervalues 6576
======End of Logs ===========

What I want to do:
I want to make a graph of CPU load vs timestamp(Ts) value
So I thought if I can create a Dictionary like Dict = {"Timestamp":"CPU Load"} (eg: Dict = {24088:85} for above logs)
then plot a graph somehow using entries in dictionary.

My Queries:
1. How to get a dictionary in above format? I'm weak at data mining...
2. For plotting graph - suggest if this can be a correct approach using dictionary? ..or some other approach would work ...(something like exporting values to MS Excel and creating plot cpu/time.)

Any help appreciated....

Similar Threads

in search of patterns in Computer Science
Data-mining & Security in Networking
problem plotting graph in Python
data-grabbing & mining - need script-help in PHP
Opening another software application in runtime in C#

you2

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

11 posts
since May 2009

Permalink

Jul 30th, 2009

Re: Query regarding Data mining and plotting graph...

still hoping for some hint on this...

you2

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

11 posts
since May 2009

Permalink

Jul 30th, 2009

Re: Query regarding Data mining and plotting graph...

Click to Expand / Collapse Quote originally posted by you2 ...

Logs snippet below:
==========LOGS==============

a:CPU [ 85%]: asdf asd 123 xyz
A: Ts 23086, Netvalue 3286, someothervalues 3456
abc abc
xyz xyz
a:CPU [ 75%]: asdf asd 123 xyz
A: Ts 24088, Netvalue 3266, someothervalues 6576
======End of Logs ===========

Will your logs always have a: A: on the lines that your relevant information is on?

If so you could do something like this:

python Syntax (Toggle Plain Text)
fh = open('my_log.inf')
flines = fh.readlines()
fh.close()
 
for line in flines:
    if line[:2\] == 'a:'
        #get CPU
    # etc...
fh = open('my_log.inf')
flines = fh.readlines()
fh.close()

for line in flines:
    if line[:2\] == 'a:'
        #get CPU
    # etc...

Do you understand what I'm saying? But the important question is, is this the consisent pattern in your source data? Once you hammer that out this will be a simple task.

jlm699

Reputation Points: 355

Solved Threads: 292

Veteran Poster

Offline

1,102 posts
since Jul 2008

Permalink

Jul 31st, 2009

Re: Query regarding Data mining and plotting graph...

Thanks for the input... I'm able to extract keywords ...but Im stuck at next level:
Now I want to check a condition that
if
"CPU" keyword occurs in line 1 (#I'm able to do this)
then
check for next line for keyword "Ts" (How do I do this ???)
and then only
create an entry in dictionary like D{CPU : timestamp} (#I'm able to do this)

Code Snippet below:
===========
a:CPU [ 75%]: asdf asd 123 xyz
A: Ts 24088,
=========

you2

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

11 posts
since May 2009

Permalink

Jul 31st, 2009

Re: Query regarding Data mining and plotting graph...

If you read your lines into a list via readlines() you can simply use indexing... something like this:

python Syntax (Toggle Plain Text)
my_index = 0
while my_index < len(my_lines):
    current_line = my_lines[index]
    if 'CPU' in current_line:
        #get CPU data
        index += 1
        next_line = my_lines[index]
        # Get Ts data
        # Save data
    index += 1
my_index = 0
while my_index < len(my_lines):
    current_line = my_lines[index]
    if 'CPU' in current_line:
        #get CPU data
        index += 1
        next_line = my_lines[index]
        # Get Ts data
        # Save data
    index += 1

That's an extremely simple solution that you could use. There's many ways to do it, but I hope this at least gets you on a path to answering your query

jlm699

Reputation Points: 355

Solved Threads: 292

Veteran Poster

Offline

1,102 posts
since Jul 2008

Permalink

Aug 4th, 2009

Re: Query regarding Data mining and plotting graph...

Thanks a lot for useful tip... it worked for me...
Python really rocks...

you2

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

11 posts
since May 2009

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Python Forum Timeline: Calling grep from within python

Next Thread in Python Forum Timeline: wxpython - repositioning help

DaniWeb Message

Cancel Changes

User Name
Password