File I/O Problem: Incomplete or Missing Content During Writeline

[mece]

I am new to python and posting to this forum and my searches turned up nothing. I am writing a component for a log parser to combine events from matching dates into one file. For some reason the log files split between 19:59:00 and 20:00:00 hours daily. To resolve this the code sample below reads the previous event count from a file based on the date being processed. Next the event number is incremented and added to each of the events and the merged lines are written to a file.

The problem I am having is some of the last events between 10 and 50 are either not written to the file or partially written. When using the interactive mode, while printing each line they are displayed correctly. I have tried using fileinput and read both produce the same results for each date. Using write instead of writelines also did not change the results.

Any suggestions are greatly appreciated.

Thanks in advance

Mece

Event Sample
FIREWALL-1 2008-10-02 20:00:04 Deny proto=17 192.168.10.90 123 172.15.6.5 123

last_count sample
2008-10-01 628756

count_file = open("/home/user/logs/last_count","r")

other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)

[tyincali]

Where is data_file defined?

[mece]

Sorry about that, I updated the code

count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
                more_logs.write('\n')

[bvdet]

You must close the file object to flush the output buffers. You can manually flush the buffers with file method flush().

[mece]

Thanks for the suggestion, but flush did not work. Here is the updated code, is my syntax correct?

count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
		        more_logs.flush()

I also tried moving the flush statement to the beginning and end of the loop, in addition to trying when the loop is completed. I tried this both in read and fileinput.

[Gribouillis]

Try this

for line ...
    if other_date ...
        ...
        more_logs = open(..., "w")
        try:
            new_log_data ...
            for line_data ...
                ...
                more_logs.write(new_data)
        finally:
            more_log.close()

[mece]

Thanks I ended up using flush and close to solve the problem by adding it before the code was executed immediatlely following the first section of code that first processes the logs.