954,549 Members — Technology Publication meets Social Media
Username:
Password:
Lost login information?
Have something to say? Contribute New Article Reply to this Article

Fixed width to csv conversion

Greetings, I'm a python newbie struggling with the obvious. I have a txt file with fixed width whitespace between the elements. Excel can't import directly because the size of the whitespace varies between columns. I have stripped the header out leaving only data.

This is the code I've tried to date, which up to a point gives me a beautiful list 'out' that is comma separated. Trying to write that to a clean .csv is proving fruitless. Any suggestions?

import sys
import csv

input = open(sys.argv[1])

lines = 0
hits = 0
out = list()

while 1:
    record = input.readline()
    if record == "": break
    lines += 1
    out.extend(record.split())
    #print record,
    hits += 1

print "Processed",hits,"out of",lines

#writer = csv.writer(open("f:/CalcOut.csv", "wb"))
#writer.writerows(out)
#writer.close()

sample of 'out' list:

['1.1507e+003', '1.238', '4.950', '1.250000', '790102', '0600', '1.858', '99.6', '111.0', '0.1767', '0.1442', '5.371', '6.934', '28.0', '0.3636', '7.451', '111.8', '6.920', '-2.765', '9.2239e+003', '97.0', '9.1555e+003', '-1.1213e+003', '9.0847e+003', '1.5966e+003', '9.4082e+003', '9.3009e+003', '1.4167e+003', '1.821', '5.316', '9.5286e+003', '9.4218e+003', '1.4225e+003', '1.740', '5.286']

How to get the list above into csv is the real question?
What command to use to close the output file?

input style:

0.125000 790101 0300 1.036 77.7 69.0 0.2100 0.1919 4.611 5.210 22.3 0.2533 4.201 80.5 4.143 0.692 2.4472e+003 77.5 2.3896e+003 5.2775e+002 2.1319e+003 1.2015e+003 2.4524e+003 2.1446e+003 1.1897e+003 1.031 4.603 2.5005e+003 2.2698e+003 1.0489e+003 0.974 4.537

That's one line of input data, consisting of 32 elements. My sample file has 10 lines, my real work has 59,000 lines. It will just fit into excel for further analysis.

Cheers
Caraka

Caraka
Newbie Poster
10 posts since Mar 2007
Reputation Points: 25
Solved Threads: 0
 
import csv

rf = open('sample.txt')     #input file handle
wf = open('writer.csv','w') #output file handle

writer = csv.writer(wf)

for row in rf.readlines():
    writer.writerow(row.split())

rf.close()  # close input file handle
wf.close()  # close output file hanle


kath.

katharnakh
Posting Whiz in Training
237 posts since Jan 2006
Reputation Points: 19
Solved Threads: 34
 

Brilliant katharnakh! I knew it had to be simple, I've just been looking at it too long. Many thanks, I can now get onto the task of analising the data itself. Process time for 16Mb file = 43 seconds.

Caraka
Newbie Poster
10 posts since Mar 2007
Reputation Points: 25
Solved Threads: 0
 

Hi Katharnak,

is the code u gave is a xl macro? just to know.

Thanks

ysrmkp
Newbie Poster
1 post since Jan 2009
Reputation Points: 10
Solved Threads: 0
 

This code only works for space separated files. It will not work for fixed width positional files.

dave8
Newbie Poster
1 post since Sep 2011
Reputation Points: 10
Solved Threads: 0
 
This code only works for space separated files. It will not work for fixed width positional files.


The input given is not positional, but variable width. You can see, if you look given input by clicking reply for the original post (as it has no code tags).

Fixed width input would be anyway even simpler as you only need to do simple slices from input line.

pyTony
pyMod
Moderator
5,359 posts since Apr 2010
Reputation Points: 782
Solved Threads: 852
 

This question has already been solved

Post: Markdown Syntax: Formatting Help
You