Python RSS Python RSS

File I/O Problem: Incomplete or Missing Content During Writeline

Thread Solved

•
•
Join Date: Oct 2008
Posts: 6
Reputation: mece is an unknown quantity at this point 
Solved Threads: 0
mece mece is offline Offline
Newbie Poster

File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #1
Oct 30th, 2008
I am new to python and posting to this forum and my searches turned up nothing. I am writing a component for a log parser to combine events from matching dates into one file. For some reason the log files split between 19:59:00 and 20:00:00 hours daily. To resolve this the code sample below reads the previous event count from a file based on the date being processed. Next the event number is incremented and added to each of the events and the merged lines are written to a file.

The problem I am having is some of the last events between 10 and 50 are either not written to the file or partially written. When using the interactive mode, while printing each line they are displayed correctly. I have tried using fileinput and read both produce the same results for each date. Using write instead of writelines also did not change the results.

Any suggestions are greatly appreciated.

Thanks in advance

Mece

Event Sample
FIREWALL-1 2008-10-02 20:00:04 Deny proto=17 192.168.10.90 123 172.15.6.5 123

last_count sample
2008-10-01 628756

Python Syntax (Toggle Plain Text) 
  1. count_file = open("/home/user/logs/last_count","r")
  2.  
  3. other_date = "2008-10-01"
  4.  
  5. for line in count_file.readlines():
  6.         if other_date in line:
  7.                 line = line.split()
  8.                 prev_event_count = line[1]
  9.                 more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
  10.                 new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
  11.                 for line_data in open(data_file,"r"):
  12.                         prev_event_count = int(prev_event_count) + 1
  13.                         nudata = str(prev_event_count),str(line_data)
  14.                         new_data = ' '.join(nudata)
  15.                         more_logs.write(new_data)
Reply With Quote Quick reply to this message  
•
•
Join Date: Oct 2008
Posts: 45
Reputation: tyincali is an unknown quantity at this point 
Solved Threads: 6
tyincali tyincali is offline Offline
Light Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #2
Oct 30th, 2008
Where is data_file defined?
Last edited by tyincali; Oct 30th, 2008 at 11:10 pm.
I wish I had something cool to put here...
Reply With Quote Quick reply to this message  
•
•
Join Date: Oct 2008
Posts: 6
Reputation: mece is an unknown quantity at this point 
Solved Threads: 0
mece mece is offline Offline
Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #3
Oct 30th, 2008
Sorry about that, I updated the code

Python Syntax (Toggle Plain Text) 
  1. count_file = open("/home/user/logs/last_count","r")
  2. data_file = "/home/user/home/prev-date-logs"
  3. other_date = "2008-10-01"
  4.  
  5. for line in count_file.readlines():
  6.         if other_date in line:
  7.                 line = line.split()
  8.                 prev_event_count = line[1]
  9.                 more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
  10.                 new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
  11.                 for line_data in open(data_file,"r"):
  12.                         prev_event_count = int(prev_event_count) + 1
  13.                         nudata = str(prev_event_count),str(line_data)
  14.                         new_data = ' '.join(nudata)
  15.                         more_logs.write(new_data)
  16.                 more_logs.write('\n')
Reply With Quote Quick reply to this message  
•
•
Join Date: Mar 2007
Posts: 110
Reputation: solsteel is on a distinguished road 
Solved Threads: 31
solsteel solsteel is offline Offline
Junior Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #4
Oct 30th, 2008
You must close the file object to flush the output buffers. You can manually flush the buffers with file method flush().
Reply With Quote Quick reply to this message  
•
•
Join Date: Oct 2008
Posts: 6
Reputation: mece is an unknown quantity at this point 
Solved Threads: 0
mece mece is offline Offline
Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #5
Oct 31st, 2008
Thanks for the suggestion, but flush did not work. Here is the updated code, is my syntax correct?

Python Syntax (Toggle Plain Text) 
  1. count_file = open("/home/user/logs/last_count","r")
  2. data_file = "/home/user/home/prev-date-logs"
  3. other_date = "2008-10-01"
  4.  
  5. for line in count_file.readlines():
  6.         if other_date in line:
  7.                 line = line.split()
  8.                 prev_event_count = line[1]
  9.                 more_logs = file("/home/user/logs"+other_date+"-DENY.log-cont","w")
  10.                 new_log_data = "/home/user/logs"+other_date+"-DENY.log-cont"
  11.                 for line_data in open(data_file,"r"):
  12.                         prev_event_count = int(prev_event_count) + 1
  13.                         nudata = str(prev_event_count),str(line_data)
  14.                         new_data = ' '.join(nudata)
  15.                         more_logs.write(new_data)
  16. 		        more_logs.flush()

I also tried moving the flush statement to the beginning and end of the loop, in addition to trying when the loop is completed. I tried this both in read and fileinput.
Reply With Quote Quick reply to this message  
•
•
Join Date: Jul 2008
Posts: 918
Reputation: Gribouillis is a jewel in the rough Gribouillis is a jewel in the rough Gribouillis is a jewel in the rough 
Solved Threads: 215
Gribouillis's Avatar
Gribouillis Gribouillis is offline Offline
Posting Shark

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #6
Oct 31st, 2008
Try this
python Syntax (Toggle Plain Text) 
  1. for line ...
  2.     if other_date ...
  3.         ...
  4.         more_logs = open(..., "w")
  5.         try:
  6.             new_log_data ...
  7.             for line_data ...
  8.                 ...
  9.                 more_logs.write(new_data)
  10.         finally:
  11.             more_log.close()
Last edited by Gribouillis; Oct 31st, 2008 at 1:44 pm.
Reply With Quote Quick reply to this message  
•
•
Join Date: Oct 2008
Posts: 6
Reputation: mece is an unknown quantity at this point 
Solved Threads: 0
mece mece is offline Offline
Newbie Poster

Re: File I/O Problem: Incomplete or Missing Content During Writeline

 
0
  #7
Oct 31st, 2008
Thanks I ended up using flush and close to solve the problem by adding it before the code was executed immediatlely following the first section of code that first processes the logs.
Reply With Quote Quick reply to this message  
Reply

Quick Reply to Thread
This thread has been marked solved.
Perhaps start a new thread instead?
•
•
•
•
•
•
•
•
Message:
•
•
•
•



Other Threads in the Python Forum
Thread Tools Search this Thread



abrupt ansi anti apache approximation array assignment avogadro backend beginner binary book builtin calculator character code converter countpasswordentry curved customdialog dan08 dictionaries dictionary dynamic examples exe file float format function gnu graphics gui heads homework import inches input java launcher library line lines linux list lists loop mouse mysql mysqlquery number numbers numeric output parsing path phonebook plugin pointer port prime programming progressbar projects py2exe pygame pysimplewizard python random recursion redirect scrolledtext software statictext statistics string strings sum table terminal text textarea thread threading time tlapse trick tricks tuple tutorial twoup ubuntu unicode urllib urllib2 variable wordgame write wxpython xlib
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC