CSV changed from 2.6 to 3.1??

[Tommymac501]

This code worked fine under 2.6:

f = open("v20100515.csv",'rt')
r = csv.reader(f)

try:
    r.next()  # skip the header record
    for row in r:
        n = 1
        for col in row:
            print str(n) + ": " + col
            n += 1

......

But since I upgraded to 3.1, I modified it to say:

f = open("v20100515.csv",'r')
r = csv.reader(f)

try:
    r.next()  # skip the header record
    for row in r:
        n = 1
        for col in row:
            print (str(n) + ": " + col)  #new print method in 3.1
            n += 1

......

When I try and read a row ( r.next() ), I get:

Message File Name Line Position
Traceback
<module> C:\Users\Dads_Laptop\Documents\Downloads\Toms_PySort.py 90
TypeError: must be bytes or buffer, not str

Also, when I tried to run the code unchanged (except for adjusting the print routines), I was getting Error: 'row' undefined. ????
What changed??

[jcao219]

Try changing 'r' to 'rb'.

[Tommymac501]

Same thing:

TypeError: must be bytes or buffer, not str

[jcao219]

Try this:

r = csv.reader(open("v20100515.csv", newline=''))

[Beat_Slayer]

Have you tried like this?

r = csv.reader(open('v20100515.csv', 'rb'))

EDIT: Sorry i began the reply and went eat some sandwich, finnished when I came, and there were already all this ones posted, but I didn't saw them.

[Tommymac501]

Have you tried like this?

r = csv.reader(open('v20100515.csv', 'rb'))

EDIT: Sorry i began the reply and went eat some sandwich, finnished when I came, and there were already all this ones posted, but I didn't saw them.

I tried the most simple;

r = csv.reader(open('v20100515.csv', 'rb'))
for row in r:
    print(row)

Fails with:

Message File Name Line Position
Traceback
<module> <module1> 3
Error: iterator should return strings, not bytes (did you open the file in text mode?)

Next, I tried a simple 'r' instead, and it read. Finally, I tried skiping a record with r.next() and got:

Message File Name Line Position
Traceback
<module> <module1> 3
AttributeError: '_csv.reader' object has no attribute 'next'

Yet the spec clearly states it does:
Reader objects (DictReader instances and objects returned by the reader() function) have the following public methods:

csvreader.next()¶
Return the next row of the reader’s iterable object as a list, parsed according to the current dialect.

I guess that changed for 3.1 also, I need to find the new spec....

[jcao219]

This is the new spec:
http://docs.python.org/py3k/library/csv.html

Anyways, in Python 3 you don't do (object).next(), you do next(object).

From the docs you see:

The simplest example of reading a CSV file:

import csv
reader = csv.reader(open("some.csv", newline=''))
for row in reader:
    print(row)

[vegaseat]

Python2 seems to take care of empty lines
Python3 needs newline='\n', for instance ...
reader = csv.reader(open("some.csv", newline='\n'))

[jcao219]

Python2 seems to take care of empty lines
Python3 needs newline='\n', for instance ...
reader = csv.reader(open("some.csv", newline='\n'))

I think the newline='' is required for csv in python 3.

[vegaseat]

Let's put it to the test ...

# explore module csv with Python 3.1.2

import csv

fout = open('eggs.csv', 'w')
writer = csv.writer(fout, delimiter=';')
writer.writerow(['Spam'] * 5 + ['Baked Beans'])
writer.writerow(['Spam'] * 5 + ['Green Eggs'])
# needed for fin to work (or use with ...)
fout.close()

"""resulting file read with an editor (has 2 empty lines) -->
Spam;Spam;Spam;Spam;Spam;Baked Beans

Spam;Spam;Spam;Spam;Spam;Green Eggs

"""

# read the file back in
# Python2 seems to take care of empty lines
# Python3 needs newline='\n'
fin = open('eggs.csv', newline='\n')
reader = csv.reader(fin, delimiter=';')
# note that reader is a generator
for row in reader:
    print(row)
fin.close()

"""my result -->
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Baked Beans']
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Green Eggs']
"""

print('-'*60)

# test Python3 and newline=''
fin = open('eggs.csv', newline='')
reader = csv.reader(fin, delimiter=';')
# note that reader is a generator
for row in reader:
    print(row)
fin.close()

"""my result (empty line turns into empty list) -->
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Baked Beans']
[]
['Spam', 'Spam', 'Spam', 'Spam', 'Spam', 'Green Eggs']
[]
"""

[Tommymac501]

Thanks for all of the input guys, boy, was this frustrating. I understand that languages need to evolve, but the concept behind Python (and Ruby) was to let the developer spend more time developing and less time worring about syntax and structure. I guess I'm just a crumudgen.. lol

[Tommymac501]

ok, maybe it's me...

f = open('v20100515.csv', 'r', newline='\n')
Try:
    r = csv.reader(f, quoting = csv.QUOTE_ALL)
        for row in r:
        print (row[76][0:5])
        .......
except:
    print ("Unexpected error:", sys.exc_info()[0])

Produces the error: error: <class 'IndexError'> Implying that the data wasn't separated into columns?..

If I try:

f = open('v20100515.csv', 'rb', newline='\n')

It throws: ValueError: binary mode doesn't take a newline argument... I think I need to start over..

error: <class 'IndexError'>

[Tommymac501]

This is making me insane...

Input data (in hex, end of line, start of new line):

'5F 43 4F 44 45 0A 22 52 ......' The record clearly ends with a newline.

Reading with 'r = csv.reader(open('TEST.csv', newline=''))' and writing with 'w = csv.writer(open('temp1.tmp', 'w'))' produces: '5F 43 4F 44 45 0D 0D 0A 22 52 ......' (2) Carraige returns and one line feed.

Reading with ''r = csv.reader(open('TEST.csv', newline='\n'))' produces the same result. Opening with 'rb' and skipping the 'newline =' (it's not allowed) causes an 'iterator should return strings, not bytes (did you open the file in text mode?)' Error.

All I want to do is open a file, read a row, and save it to a temp file.. ugh!!

[Tommymac501]

It appears that my major issue is that the data comes from a Unix box, and has newlines as line ends. Opening in binary mode works, but, you cannot use the

for x in Y

because the data types dont work, and, you can't use the 'newline' in binary mode. If you use the default newline = '', or, newline = '\n', the existing newline is converted to a carraige return, and a CRLF is appended at write time, making the line end \n\r\n and causing space between rows.

Answer?, go back to 2.6, it all works just fine.. LOLOL

(There is a hot issue going on in the python bug reports on this issue actually, and I thought it was just me. http://bugs.python.org/issue5455)

Thanks everyone for you're help!!

[Beat_Slayer]

Is it solved???

[Tommymac501]

Is it solved???

Near as can tell, they agreed to change the documentation to reflect the limitaiton, they didn't fix the problem.