Python

File I/O Problem: Incomplete or Missing Content During Writeline

Please support our Python advertiser: Programming Forums - DaniWeb Sister Site

Thread Solved

•

Join Date: Oct 2008

Posts: 6

Reputation: mece is an unknown quantity at this point

Solved Threads: 0

mece

Offline

Newbie Poster

File I/O Problem: Incomplete or Missing Content During Writeline

Oct 30th, 2008

I am new to python and posting to this forum and my searches turned up nothing. I am writing a component for a log parser to combine events from matching dates into one file. For some reason the log files split between 19:59:00 and 20:00:00 hours daily. To resolve this the code sample below reads the previous event count from a file based on the date being processed. Next the event number is incremented and added to each of the events and the merged lines are written to a file.

The problem I am having is some of the last events between 10 and 50 are either not written to the file or partially written. When using the interactive mode, while printing each line they are displayed correctly. I have tried using fileinput and read both produce the same results for each date. Using write instead of writelines also did not change the results.

Any suggestions are greatly appreciated.

Thanks in advance

Mece

Event Sample
FIREWALL-1 2008-10-02 20:00:04 Deny proto=17 192.168.10.90 123 172.15.6.5 123

last_count sample
2008-10-01 628756

 Help with Code Tags 
 Python Syntax (Toggle Plain Text) 
count_file = open("/home/user/logs/last_count","r")
 
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
count_file = open("/home/user/logs/last_count","r")

other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)

•

Join Date: Oct 2008

Posts: 45

Reputation: tyincali is an unknown quantity at this point

Solved Threads: 6

tyincali

Offline

Light Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 30th, 2008

Where is data_file defined?

Last edited by tyincali; Oct 30th, 2008 at 11:10 pm.

I wish I had something cool to put here...

•

Join Date: Oct 2008

Posts: 6

Reputation: mece is an unknown quantity at this point

Solved Threads: 0

mece

Offline

Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 30th, 2008

Sorry about that, I updated the code

 Help with Code Tags 
 Python Syntax (Toggle Plain Text) 
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
                more_logs.write('\n')
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
                more_logs.write('\n')

•

Join Date: Mar 2007

Posts: 110

Reputation: solsteel is on a distinguished road

Solved Threads: 31

solsteel

Offline

Junior Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 30th, 2008

You must close the file object to flush the output buffers. You can manually flush the buffers with file method flush().

•

Join Date: Oct 2008

Posts: 6

Reputation: mece is an unknown quantity at this point

Solved Threads: 0

mece

Offline

Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 31st, 2008

Thanks for the suggestion, but flush did not work. Here is the updated code, is my syntax correct?

 Help with Code Tags 
 Python Syntax (Toggle Plain Text) 
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"
 
for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
		        more_logs.flush()
count_file = open("/home/user/logs/last_count","r")
data_file = "/home/user/home/prev-date-logs"
other_date = "2008-10-01"

for line in count_file.readlines():
        if other_date in line:
                line = line.split()
                prev_event_count = line[1]
                more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
                new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
                for line_data in open(data_file,"r"):
                        prev_event_count = int(prev_event_count) + 1
                        nudata = str(prev_event_count),str(line_data)
                        new_data = ' '.join(nudata)
                        more_logs.write(new_data)
		        more_logs.flush()

I also tried moving the flush statement to the beginning and end of the loop, in addition to trying when the loop is completed. I tried this both in read and fileinput.

•

Join Date: Jul 2008

Posts: 984

Reputation: Gribouillis is a jewel in the rough

Solved Threads: 223

Gribouillis Gribouillis is offline

Offline

Posting Shark

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 31st, 2008

Try this

 Help with Code Tags 
 python Syntax (Toggle Plain Text) 
for line ...
    if other_date ...
        ...
        more_logs = open(..., "w")
        try:
            new_log_data ...
            for line_data ...
                ...
                more_logs.write(new_data)
        finally:
            more_log.close()
for line ...
    if other_date ...
        ...
        more_logs = open(..., "w")
        try:
            new_log_data ...
            for line_data ...
                ...
                more_logs.write(new_data)
        finally:
            more_log.close()

Last edited by Gribouillis; Oct 31st, 2008 at 1:44 pm.

•

Join Date: Oct 2008

Posts: 6

Reputation: mece is an unknown quantity at this point

Solved Threads: 0

mece

Offline

Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

Oct 31st, 2008

Thanks I ended up using flush and close to solve the problem by adding it before the code was executed immediatlely following the first section of code that first processes the logs.