Ad:

Python Discussion Thread
Marked Solved
Views: 1571

Oct 30th, 2008

File I/O Problem: Incomplete or Missing Content During Writeline

Expand Post »

I am new to python and posting to this forum and my searches turned up nothing. I am writing a component for a log parser to combine events from matching dates into one file. For some reason the log files split between 19:59:00 and 20:00:00 hours daily. To resolve this the code sample below reads the previous event count from a file based on the date being processed. Next the event number is incremented and added to each of the events and the merged lines are written to a file.

The problem I am having is some of the last events between 10 and 50 are either not written to the file or partially written. When using the interactive mode, while printing each line they are displayed correctly. I have tried using fileinput and read both produce the same results for each date. Using write instead of writelines also did not change the results.

Any suggestions are greatly appreciated.

Thanks in advance

Mece

Event Sample
FIREWALL-1 2008-10-02 20:00:04 Deny proto=17 192.168.10.90 123 172.15.6.5 123

last_count sample
2008-10-01 628756

Python Syntax (Toggle Plain Text)
count_file = open("/home/user/logs/last_count","r")
 
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
count_file = open("/home/user/logs/last_count","r")

other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)

mece

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

10 posts
since Oct 2008

Permalink

Oct 30th, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Where is data_file defined?

Last edited by tyincali; Oct 30th, 2008 at 11:10 pm.

tyincali

Reputation Points: 31

Solved Threads: 7

Light Poster

Offline

45 posts
since Oct 2008

Permalink

Oct 30th, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Sorry about that, I updated the code

Python Syntax (Toggle Plain Text)
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
                more_logs.write('\n')
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
                more_logs.write('\n')

mece

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

10 posts
since Oct 2008

Permalink

Oct 30th, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

You must close the file object to flush the output buffers. You can manually flush the buffers with file method flush().

solsteel

Reputation Points: 86

Solved Threads: 40

Junior Poster

Offline

141 posts
since Mar 2007

Permalink

Oct 31st, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Thanks for the suggestion, but flush did not work. Here is the updated code, is my syntax correct?

Python Syntax (Toggle Plain Text)
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
		        more_logs.flush()
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
		        more_logs.flush()

I also tried moving the flush statement to the beginning and end of the loop, in addition to trying when the loop is completed. I tried this both in read and fileinput.

mece

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

10 posts
since Oct 2008

Permalink

Oct 31st, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Try this

python Syntax (Toggle Plain Text)
for line ...
    if other_date ...
        ...
        more_logs = open(..., "w")
        try:
            new_log_data ...
            for line_data ...
                ...
                more_logs.write(new_data)
        finally:
            more_log.close()
for line ...
    if other_date ...
        ...
        more_logs = open(..., "w")
        try:
            new_log_data ...
            for line_data ...
                ...
                more_logs.write(new_data)
        finally:
            more_log.close()

Last edited by Gribouillis; Oct 31st, 2008 at 1:44 pm.

Gribouillis

Reputation Points: 930

Solved Threads: 668

Posting Maven

Offline

2,655 posts
since Jul 2008

Permalink

Oct 31st, 2008

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Thanks I ended up using flush and close to solve the problem by adding it before the code was executed immediatlely following the first section of code that first processes the logs.

mece

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

10 posts
since Oct 2008

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Python Forum Timeline: Beginner Code

Next Thread in Python Forum Timeline: Python (Django) vs PHP

DaniWeb Message

Cancel Changes

User Name
Password