Parsing a CSV File and inserting into a dictionary

[shaun.b]

Hi everyone,

Ive just got a bit of a problem with a program i am currently writing.

Basically i have got a csv file with a load of flight info in it, i need to parse that and add it to a dictionary so thati can search through it easily.

I ave used the CSV module to open the csv file and i have added the data to a dictionary, but when i print the dictionary, it just shows a load of {}.

This is the first few lines of the csv file:

Origin,Destination,Flight No.,Aircraft ,Days of ,Departure,Arrival ,Begin Date,End Date,
Agartala,Guwahati,CD-7756,ATR,1357,1755,1845,02-Jun-09,----,
Agartala,Guwahati,CD-7756,ATR,246,1850,1945,02-Jun-09,----,
Agartala,Guwahati,CD-7771,ATR,3,0745,0940,03-Jun-09,09-Jun-09,
Agartala,Guwahati,CD-7771,ATR,1,0740,0935,03-Jun-09,09-Jun-09,
Agartala,Guwahati,CD-7771,ATR,3,0740,0935,10-Jun-09,10-Jun-09,

and this is my code so far:

import csv

dicts = []

inputFile = open("ia-schedule.csv", "rb")
parser = csv.reader(inputFile)

firstRec = True
for fields in parser:
    if firstRec:
        fieldNames = fields
        firstRec = False
    else:
        dicts.append({})
        
for i,f in enumerate(fields):
    dicts[-1][fieldNames[i]] = f

print dicts

any help is very much appreciated.

Thanks

Shaun

[jlm699]

it just shows a load of {}.

firstRec = True
for fields in parser:
    if firstRec:
        fieldNames = fields
        firstRec = False
    else:
        dicts.append({})
        
for i,f in enumerate(fields):
    dicts[-1][fieldNames[i]] = f

In your code, you are iterating over the contents of parser. You ignore the first line, then set firstRec to False. The remainder of the lines are also ignored and you are appending an empty dictionary {} to your list called dicts.

Where's the code that's supposed to read the contents of the csv file? I don't see any attempt at doing so...

[shaun.b]

Hi,

Thanks very much for the reply. Just realised how much of an idiot i am expecting that code too work lol, havnt dont much programming in python.

Anyway, i have now got the program reading the csv file, and adding the data to a list of dictionaries, is this the best way to do it, the next job is to create a search function which would allow someone to search for flights which fly on a certain day, and then it would bring up all the corresponding flights and thier details (e.g flight times etc).

Here is my code so far:

import csv

dicts = []

inputFile = open("ia-schedule.csv", "rb")
csvParser = csv.reader(inputFile)

for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
    #print Origin,Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate

    dicts.append({'Origin': Origin, 'Destination': Destination, 'Flight No': FlightNo, 'Aircraft': Aircraft, 'Days Of': DaysOf,
                  'Departure': Departure, 'Arrival': Arrival, 'Begin Date': BeginDate, 'End Date': EndDate})

One problem i am having though, the program runs fine, but ends with an error, as it gets to the end of the csv file and there is nothing else to read, so there is a valueError, anyone know how to prevent this?

Thanks again

Shaun

[jlm699]

for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
    dicts.append({'Origin': Origin, 'Destination': Destination, 'Flight No': FlightNo, 'Aircraft': Aircraft, 'Days Of': DaysOf,
                  'Departure': Departure, 'Arrival': Arrival, 'Begin Date': BeginDate, 'End Date': EndDate})

as it gets to the end of the csv file and there is nothing else to read, so there is a valueError, anyone know how to prevent this?

It would help if you pasted the traceback to know which bit is causing the problem, but here's my suggestion: basically you need to make sure that each line is valid for the number of fields you're unpacking...

Go back to using the for fields in parser: method, and then do a check like this:

if fields and len(fields) == 9:
    Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate = fields
else:
    print 'Invalid line format: %s' % str(fields)
    break

Like I said, it's hard to say that will fix it without the traceback, as I'm not sure what exactly is failing... but this is my suspicion.

I never use the csv module so I don't really know how that iteration process works although I'm surprised it wouldn't raise a StopIteration... Maybe there's extra lines with less fields than you're expecting at the end there... And the unpacking into all those fields is crapping out when there aren't enough values to unpack.

[shaun.b]

Thanks for the quick reply.

It was because there were extra returns after the last line in the csv file, ive deleted them and it works a treat, thank you very much!!

One other thing, the original csv file that i got had commas at the end of each line, and when i first tried this it came up with a value error as it though there were 10 entries but i was only providing 9 field names, but it worked fine when i deleted the comma at the end of each line.

Obviously thats ok with small files, but would be impossible to delete the commas of the end of every line with a larger file. Any ideas why it is doing this, here is the error:

Traceback (most recent call last):
File "C:/Documents and Settings/Shaun/Desktop/APE/parser2", line 8, in <module>
for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
ValueError: too many values to unpack

Say if i remove the commas from the first 3 lines, and get my code to print each line it parses, it will print the first three and then show the above error when it gets to the next line with a comma at the end. Any ideas on that?

Thanks for your replies, they have really helped!

Shaun

[woooee]

the next job is to create a search function which would allow someone to search for flights which fly on a certain day, and then it would bring up all the corresponding flights and thier details (e.g flight times etc)

How are you going to do this? You have a list of dictionaries so you will have to parse the list and check each dictionary, in which case the list is redundant, since you could achieve the same result just parsing a dictionary with one entry per flight. Perhaps you want two dictionaries, the first has the flight number as key, since that is unique, pointing to a tuple with all of the oher info. A second dictionay would be indexed on date and point to a list of filght numbers for that date, so you could then print all of the infro from the first dictionary for all the the filght numbers found. In a large scale database, you would use SQL, but I am assuming that you want to do it with dictionaries.

Edit: I don't use a csv parser either, but you should be able to do this:

for data_tuple in csvParser:
   print len(data_tuple), type(data_tuple)
   if len(data_tuple) > fields_used:
      ## now Origin=data_tuple[0], Destination=data_tuple[1], etc
      ## and you can just ignore the field created by the last comma