Python RSS Python RSS

Parsing a CSV File and inserting into a dictionary

Reply

•
•
Join Date: Jan 2009
Posts: 5
Reputation: shaun.b is an unknown quantity at this point 
Solved Threads: 0
shaun.b shaun.b is offline Offline
Newbie Poster

Parsing a CSV File and inserting into a dictionary

 
0
  #1
Aug 6th, 2009
Hi everyone,

Ive just got a bit of a problem with a program i am currently writing.

Basically i have got a csv file with a load of flight info in it, i need to parse that and add it to a dictionary so thati can search through it easily.

I ave used the CSV module to open the csv file and i have added the data to a dictionary, but when i print the dictionary, it just shows a load of {}.

This is the first few lines of the csv file:

Origin,Destination,Flight No.,Aircraft ,Days of ,Departure,Arrival ,Begin Date,End Date,
Agartala,Guwahati,CD-7756,ATR,1357,1755,1845,02-Jun-09,----,
Agartala,Guwahati,CD-7756,ATR,246,1850,1945,02-Jun-09,----,
Agartala,Guwahati,CD-7771,ATR,3,0745,0940,03-Jun-09,09-Jun-09,
Agartala,Guwahati,CD-7771,ATR,1,0740,0935,03-Jun-09,09-Jun-09,
Agartala,Guwahati,CD-7771,ATR,3,0740,0935,10-Jun-09,10-Jun-09,

and this is my code so far:

Python Syntax (Toggle Plain Text) 
  1. import csv
  2.  
  3. dicts = []
  4.  
  5. inputFile = open("ia-schedule.csv", "rb")
  6. parser = csv.reader(inputFile)
  7.  
  8. firstRec = True
  9. for fields in parser:
  10.     if firstRec:
  11.         fieldNames = fields
  12.         firstRec = False
  13.     else:
  14.         dicts.append({})
  15.  
  16. for i,f in enumerate(fields):
  17.     dicts[-1][fieldNames[i]] = f
  18.  
  19. print dicts

any help is very much appreciated.

Thanks

Shaun
Reply With Quote Quick reply to this message  
•
•
Join Date: Jul 2008
Posts: 1,046
Reputation: jlm699 is a jewel in the rough jlm699 is a jewel in the rough jlm699 is a jewel in the rough jlm699 is a jewel in the rough 
Solved Threads: 263
Sponsor
jlm699's Avatar
jlm699 jlm699 is offline Offline
Knows where his Towel is

Re: Parsing a CSV File and inserting into a dictionary

 
0
  #2
Aug 6th, 2009
•
•
•
•
Originally Posted by shaun.b View Post
it just shows a load of {}.
Python Syntax (Toggle Plain Text) 
  1. firstRec = True
  2. for fields in parser:
  3.     if firstRec:
  4.         fieldNames = fields
  5.         firstRec = False
  6.     else:
  7.         dicts.append({})
  8.  
  9. for i,f in enumerate(fields):
  10.     dicts[-1][fieldNames[i]] = f
In your code, you are iterating over the contents of parser. You ignore the first line, then set firstRec to False. The remainder of the lines are also ignored and you are appending an empty dictionary {} to your list called dicts.

Where's the code that's supposed to read the contents of the csv file? I don't see any attempt at doing so...
1. Use Code Tags.
2. Homework? Show Effort.
3. Keep discussions on the forum: no PMs
Reply With Quote Quick reply to this message  
•
•
Join Date: Jan 2009
Posts: 5
Reputation: shaun.b is an unknown quantity at this point 
Solved Threads: 0
shaun.b shaun.b is offline Offline
Newbie Poster

Re: Parsing a CSV File and inserting into a dictionary

 
0
  #3
Aug 6th, 2009
Hi,

Thanks very much for the reply. Just realised how much of an idiot i am expecting that code too work lol, havnt dont much programming in python.

Anyway, i have now got the program reading the csv file, and adding the data to a list of dictionaries, is this the best way to do it, the next job is to create a search function which would allow someone to search for flights which fly on a certain day, and then it would bring up all the corresponding flights and thier details (e.g flight times etc).

Here is my code so far:

Python Syntax (Toggle Plain Text) 
  1. import csv
  2.  
  3. dicts = []
  4.  
  5. inputFile = open("ia-schedule.csv", "rb")
  6. csvParser = csv.reader(inputFile)
  7.  
  8. for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
  9.     #print Origin,Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate
  10.  
  11.     dicts.append({'Origin': Origin, 'Destination': Destination, 'Flight No': FlightNo, 'Aircraft': Aircraft, 'Days Of': DaysOf,
  12.                   'Departure': Departure, 'Arrival': Arrival, 'Begin Date': BeginDate, 'End Date': EndDate})

One problem i am having though, the program runs fine, but ends with an error, as it gets to the end of the csv file and there is nothing else to read, so there is a valueError, anyone know how to prevent this?

Thanks again

Shaun
Reply With Quote Quick reply to this message  
•
•
Join Date: Jul 2008
Posts: 1,046
Reputation: jlm699 is a jewel in the rough jlm699 is a jewel in the rough jlm699 is a jewel in the rough jlm699 is a jewel in the rough 
Solved Threads: 263
Sponsor
jlm699's Avatar
jlm699 jlm699 is offline Offline
Knows where his Towel is

Re: Parsing a CSV File and inserting into a dictionary

 
0
  #4
Aug 6th, 2009
•
•
•
•
Originally Posted by shaun.b View Post
Python Syntax (Toggle Plain Text) 
  1. for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
  2.     dicts.append({'Origin': Origin, 'Destination': Destination, 'Flight No': FlightNo, 'Aircraft': Aircraft, 'Days Of': DaysOf,
  3.                   'Departure': Departure, 'Arrival': Arrival, 'Begin Date': BeginDate, 'End Date': EndDate})
as it gets to the end of the csv file and there is nothing else to read, so there is a valueError, anyone know how to prevent this?
It would help if you pasted the traceback to know which bit is causing the problem, but here's my suggestion: basically you need to make sure that each line is valid for the number of fields you're unpacking...

Go back to using the for fields in parser: method, and then do a check like this:
python Syntax (Toggle Plain Text) 
  1. if fields and len(fields) == 9:
  2.     Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate = fields
  3. else:
  4.     print 'Invalid line format: %s' % str(fields)
  5.     break
Like I said, it's hard to say that will fix it without the traceback, as I'm not sure what exactly is failing... but this is my suspicion.

I never use the csv module so I don't really know how that iteration process works although I'm surprised it wouldn't raise a StopIteration... Maybe there's extra lines with less fields than you're expecting at the end there... And the unpacking into all those fields is crapping out when there aren't enough values to unpack.
1. Use Code Tags.
2. Homework? Show Effort.
3. Keep discussions on the forum: no PMs
Reply With Quote Quick reply to this message  
•
•
Join Date: Jan 2009
Posts: 5
Reputation: shaun.b is an unknown quantity at this point 
Solved Threads: 0
shaun.b shaun.b is offline Offline
Newbie Poster

Re: Parsing a CSV File and inserting into a dictionary

 
0
  #5
Aug 6th, 2009
Thanks for the quick reply.

It was because there were extra returns after the last line in the csv file, ive deleted them and it works a treat, thank you very much!!

One other thing, the original csv file that i got had commas at the end of each line, and when i first tried this it came up with a value error as it though there were 10 entries but i was only providing 9 field names, but it worked fine when i deleted the comma at the end of each line.

Obviously thats ok with small files, but would be impossible to delete the commas of the end of every line with a larger file. Any ideas why it is doing this, here is the error:

•
•
•
•
Traceback (most recent call last):
File "C:/Documents and Settings/Shaun/Desktop/APE/parser2", line 8, in <module>
for Origin, Destination, FlightNo, Aircraft, DaysOf, Departure, Arrival, BeginDate, EndDate in csvParser:
ValueError: too many values to unpack
Say if i remove the commas from the first 3 lines, and get my code to print each line it parses, it will print the first three and then show the above error when it gets to the next line with a comma at the end. Any ideas on that?

Thanks for your replies, they have really helped!

Shaun
Reply With Quote Quick reply to this message  
•
•
Join Date: Dec 2006
Posts: 1,008
Reputation: woooee is a jewel in the rough woooee is a jewel in the rough woooee is a jewel in the rough 
Solved Threads: 285
woooee woooee is offline Offline
Veteran Poster

Re: Parsing a CSV File and inserting into a dictionary

 
0
  #6
Aug 6th, 2009
•
•
•
•
the next job is to create a search function which would allow someone to search for flights which fly on a certain day, and then it would bring up all the corresponding flights and thier details (e.g flight times etc)
How are you going to do this? You have a list of dictionaries so you will have to parse the list and check each dictionary, in which case the list is redundant, since you could achieve the same result just parsing a dictionary with one entry per flight. Perhaps you want two dictionaries, the first has the flight number as key, since that is unique, pointing to a tuple with all of the oher info. A second dictionay would be indexed on date and point to a list of filght numbers for that date, so you could then print all of the infro from the first dictionary for all the the filght numbers found. In a large scale database, you would use SQL, but I am assuming that you want to do it with dictionaries.

Edit: I don't use a csv parser either, but you should be able to do this:
Python Syntax (Toggle Plain Text) 
  1. for data_tuple in csvParser:
  2.    print len(data_tuple), type(data_tuple)
  3.    if len(data_tuple) > fields_used:
  4.       ## now Origin=data_tuple[0], Destination=data_tuple[1], etc
  5.       ## and you can just ignore the field created by the last comma
Last edited by woooee; Aug 6th, 2009 at 1:09 pm.
Linux counter #99383
Reply With Quote Quick reply to this message  
Reply

Quick Reply to Thread
This thread is more than three months old.
Perhaps start a new thread instead?
•
•
•
•
•
•
•
•
Message:
•
•
•
•



Similar Threads
Other Threads in the Python Forum
Thread Tools Search this Thread



abrupt ansi anti approximation array assignment avogadro backend beginner binary bluetooth builtin calculator character cmd code converter countpasswordentry curved customdialog dan08 decimals dictionaries dictionary dynamic error examples exe file float format function gnu graphics gui heads homework http ideas import input java launcher leftmouse line linux list lists loop module mouse mysqlquery number numbers output parsing path plugin pointer port prime programming progressbar projects push py2exe pygame pyqt python random recursion schedule scrolledtext sqlite statistics string strings sudokusolver sum table terminal text textarea thread threading time tlapse tricks tuple tutorial twoup ubuntu unicode urllib urllib2 variable wikipedia write wxpython xlib
About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC