943,729 Members | Top Members by Rank

Ad:
  • Python Discussion Thread
  • Unsolved
  • Views: 819
  • Python RSS
Mar 16th, 2009
0

Process text file with parameters

Expand Post »
Hi

I am trying to process a large text file. I would like to include parameters at some places in the file. Run it through python and write out the "processed" file with the values of the parameters in the new file.

Example: original file:
a =1
b=2
c= param1
d=4

Run python with param1=3: Gives new file with
a=1
b=2
c=3
d=4

---
Is this possible? The original text file will be around 100Mb and have maybe 100 parameters.

Any ideas how to do this in python?

Best regards
Joakim
Reputation Points: 10
Solved Threads: 0
Newbie Poster
ask2 is offline Offline
2 posts
since Mar 2009
Mar 16th, 2009
1

Re: Process text file with parameters

If the input is a "text" file and the replacement values are encoded the same way and has "reasonably" short lines, then the following will suffice.

Python Syntax (Toggle Plain Text)
  1. params={'param1':1, 'param2':2}
  2. fi=open("a.txt")
  3. fo=open("a_processed.txt","w")
  4. for line in fi:
  5. for k,v in params.iteritems():
  6. fo.write(line.replace(k,str(v)))
  7. fo.close()
  8. fi.close()


You can also store the params in a separate file and load it before processing.
Reputation Points: 56
Solved Threads: 65
Posting Whiz in Training
slate is offline Offline
242 posts
since Jun 2008
Mar 16th, 2009
0

Re: Process text file with parameters

For a more general approach see the KMP algo:
http://code.activestate.com/recipes/117214/
Reputation Points: 56
Solved Threads: 65
Posting Whiz in Training
slate is offline Offline
242 posts
since Jun 2008
Mar 16th, 2009
0

Re: Process text file with parameters

The general idea is to store the replacement values in a container like a dictionary, isolate the string that you want to test, and replace it if it is in the replacement dictionary. As stated above you could store them in a separate text file and read into a dictionary. The following code uses a list for input and output to simulate files, but gives the general idea.
Python Syntax (Toggle Plain Text)
  1. class Param(object):
  2. def __init__(self):
  3. ## parameter dictionary with the key=string to search for,
  4. ## and value=replacement value
  5. self.params_dic={"param1":"3",
  6. "param2":"2",
  7. "param3":"1" }
  8.  
  9.  
  10. def process_file(self, data_in):
  11. output_list = []
  12. for rec in data_in:
  13. r_list = rec.split("=")
  14. test_s = r_list[1].strip()
  15. if test_s in self.params_dic: ## found
  16. output_list.append("%s=%s\n" % \
  17. (r_list[0], self.params_dic[test_s]))
  18. else: ## not found
  19. output_list.append(rec)
  20.  
  21. print "final output ="
  22. for rec in output_list:
  23. print rec,
  24.  
  25. P = Param()
  26. input_file_list = ['a =1\n', 'b=2\n', 'c= param1\n', 'd=4\n', 'e=param3\n']
  27. P.process_file(input_file_list)
Reputation Points: 741
Solved Threads: 692
Nearly a Posting Maven
woooee is offline Offline
2,305 posts
since Dec 2006
Mar 18th, 2009
0

Re: Process text file with parameters

Quote ...
Is this possible? The original text file will be around 100Mb and have maybe 100 parameters.
Were you able to do this with the suggestions posted?
Reputation Points: 741
Solved Threads: 692
Nearly a Posting Maven
woooee is offline Offline
2,305 posts
since Dec 2006
Mar 18th, 2009
0

Re: Process text file with parameters

Click to Expand / Collapse  Quote originally posted by woooee ...
Were you able to do this with the suggestions posted?
Hello

The suggested method works fine. I haven't been able to try it on a "production" size file yet. So I am not sure how long it will take. But on smaller files it works perfectly.

Thanks to all of you
Reputation Points: 10
Solved Threads: 0
Newbie Poster
ask2 is offline Offline
2 posts
since Mar 2009

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in Python Forum Timeline: Image averaging using Python
Next Thread in Python Forum Timeline: help with web crawler





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC