Ad:

Python Discussion Thread
Marked Solved
Views: 2273

Nov 10th, 2008

Combining txt from 2 files

Expand Post »

Alright this is my first program in python, so be gentle. I have 2 files I need to basically combine to create a record in mysql. Features.txt has around 1300 records and property.txt has about 25,000 records. I can read in property.txt fine and get my insert statements. My problem is trying to read in features.txt. I only need the lines that have field 1 = RES. I was thinking a dictionary ('1701' : 'Central Air', '1702' : 'Window Unit", etc) but can seem to quite get it.

python Syntax (Toggle Plain Text)
for featureline in ffeaturefile.readlines():
        featureline = featureline.strip().split("|")
        addlist += "'%s', '%s'," % (featureline[0], featureline[3])
        resfeaturelist = dict(addlist)
for featureline in ffeaturefile.readlines():
        featureline = featureline.strip().split("|")
        addlist += "'%s', '%s'," % (featureline[0], featureline[3])
        resfeaturelist = dict(addlist)

Python Syntax (Toggle Plain Text)
features.txt
 
1701|RES|HVAC|Central Air
1702|RES|HVAC|Window Unit
1703|RES|HVAC|Heat Pump - AC|
1704|RES|HVAC|No A/C
1705|RES|HVAC|Wall Unit - AC
1706|RES|HVAC|Multizone A/C
1708|RES|HVAC|Gas Hot Air
1701|CND|HVAC|Central Air
...
 
property.txt (there are a 105 fields so I shortened it here)
0.293|53548|47459||4|101|QUART|2|4||Matthews||1505,1510,1514||1701,1708,1722|
...
features.txt

1701|RES|HVAC|Central Air
1702|RES|HVAC|Window Unit
1703|RES|HVAC|Heat Pump - AC|
1704|RES|HVAC|No A/C
1705|RES|HVAC|Wall Unit - AC
1706|RES|HVAC|Multizone A/C
1708|RES|HVAC|Gas Hot Air
1701|CND|HVAC|Central Air
...

property.txt (there are a 105 fields so I shortened it here)
0.293|53548|47459||4|101|QUART|2|4||Matthews||1505,1510,1514||1701,1708,1722|
...

I'm inserting into a mysql database here is what I have right now:

Python Syntax (Toggle Plain Text)
Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('HVAC','1701,1708','2','0');
Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('HVAC','1701,1708','2','0');

but I need

Python Syntax (Toggle Plain Text)
Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('HVAC','Central Air,Gas Hot Air','2','0');
Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('HVAC','Central Air,Gas Hot Air','2','0');

python Syntax (Toggle Plain Text)
for number, resline in enumerate(fresfile):
        resline = resline.strip().split("|")
 
        for count in [0, 21]:
                if (count == 21):
                       #change values to text in features.txt
                        resfields = resline[count].split(",")
                        for i, item in enumerate(resfields):
                                if (featureline[0] == item):
                                        print resfields[3]
                else:
                        fsqlfile.write ("Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('" + resheader[count] + "','" + resline[count] + "','" + str(number) + "','0');\n")
for number, resline in enumerate(fresfile):
        resline = resline.strip().split("|")

        for count in [0, 21]:
                if (count == 21):
                       #change values to text in features.txt
                        resfields = resline[count].split(",")
                        for i, item in enumerate(resfields):
                                if (featureline[0] == item):
                                        print resfields[3]
                else:
                        fsqlfile.write ("Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('" + resheader[count] + "','" + resline[count] + "','" + str(number) + "','0');\n")

Also, once I get the features.txt into a dictionary how do I do the substitution?

Similar Threads

Plz, helping with writing a bit of code. Thanks in C++

grambo

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

4 posts
since Nov 2008

Permalink

Nov 10th, 2008

Re: Combining txt from 2 files

Click to Expand / Collapse Quote originally posted by grambo ...

python Syntax (Toggle Plain Text)
for featureline in ffeaturefile.readlines():
        featureline = featureline.strip().split("|")
        addlist += "'%s', '%s'," % (featureline[0], featureline[3])
        resfeaturelist = dict(addlist)
for featureline in ffeaturefile.readlines(): featureline = featureline.strip().split("|") addlist += "'%s', '%s'," % (featureline[0], featureline[3]) resfeaturelist = dict(addlist)
python Syntax (Toggle Plain Text)
for number, resline in enumerate(fresfile):
        resline = resline.strip().split("|")
 
        for count in [0, 21]:
                if (count == 21):
                       #change values to text in features.txt
                        resfields = resline[count].split(",")
                        for i, item in enumerate(resfields):
                                if (featureline[0] == item):
                                        print resfields[3]
                else:
                        fsqlfile.write ("Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('" + resheader[count] + "','" + resline[count] + "','" + str(number) + "','0');\n")
for number, resline in enumerate(fresfile): resline = resline.strip().split("|") for count in [0, 21]: if (count == 21): #change values to text in features.txt resfields = resline[count].split(",") for i, item in enumerate(resfields): if (featureline[0] == item): print resfields[3] else: fsqlfile.write ("Insert into openrealty_en_listingsdbelements (listingdbelements_field_name,listingdbelements_field_value,listingsdb_id,userdb_id) values ('" + resheader[count] + "','" + resline[count] + "','" + str(number) + "','0');\n")
Also, once I get the features.txt into a dictionary how do I do the substitution?

I'm not quite sure what you mean by substituion.. do you mean use the value instead of the key? Also, I'm wary of how you are creating your dictionary.

I propose the following modification to the first bit of code:

python Syntax (Toggle Plain Text)
resfeaturelist = {}
for featureline in ffeaturefile.readlines():
        featureline = featureline.strip().split("|")
        addlist += "'%s', '%s'," % (featureline[0], featureline[3])
        resfeaturelist[ featureline[0] ] = featureline[3]
resfeaturelist = {}
for featureline in ffeaturefile.readlines():
        featureline = featureline.strip().split("|")
        addlist += "'%s', '%s'," % (featureline[0], featureline[3])
        resfeaturelist[ featureline[0] ] = featureline[3]

Now take a look at this example of working with a dictionary to see if it answers your question about "substitution":

python Syntax (Toggle Plain Text)
>>> dd = {} # Initialize the dictionary
>>> dd[ '7801' ] = 'Heat Pump'
>>> dd[ '7902' ] = 'Central A/C'
>>> dd[ '8901' ] = 'Box Fan'
>>> print dd[ '7801' ]
Heat Pump
>>> "INSERT x into y VALUES( '%s', '%s', '0' );" % ( dd['7801'], dd['8901'] )
"INSERT x into y VALUES( 'Heat Pump', 'Box Fan', '0' );"
>>> dd.get( '7901' )
>>> dd.get( '7902' )
'Central A/C'
>>>
>>> dd = {} # Initialize the dictionary
>>> dd[ '7801' ] = 'Heat Pump'
>>> dd[ '7902' ] = 'Central A/C'
>>> dd[ '8901' ] = 'Box Fan'
>>> print dd[ '7801' ]
Heat Pump
>>> "INSERT x into y VALUES( '%s', '%s', '0' );" % ( dd['7801'], dd['8901'] )
"INSERT x into y VALUES( 'Heat Pump', 'Box Fan', '0' );"
>>> dd.get( '7901' )
>>> dd.get( '7902' )
'Central A/C'
>>>

As you can see there's two different methods to retrieving values from a dictionary. Using get() is safer since you can check if <value> == None: so that you don't get any code-stopping exceptions; however in your case you may want the code to stop.

Let me know whether this clears up your questions, and whether that first bit helped to alleviate your problems..

Last edited by jlm699; Nov 10th, 2008 at 10:47 am.

jlm699

Reputation Points: 355

Solved Threads: 292

Veteran Poster

Offline

1,102 posts
since Jul 2008

Permalink

Nov 10th, 2008

Re: Combining txt from 2 files

That's perfect. I think I was originally just creating a list. I could find anything good for docs on dictionarys (or python for that matter) online. The dictionary is great, and I can see how to use it on one item. How can I do it for multiple items? Also, 1 record might have 1 and another might have 10.

my code right now is just

python Syntax (Toggle Plain Text)
if (count == 21):
 resfields = resline[count].split(",")
 print resfields
if (count == 21):
 resfields = resline[count].split(",")
 print resfields

Which prints out:
['1701', '1708', '1722']
['1701', '1708']
['1701', '1708']
['1701', '1708', '1706']

which I need to transform into:
['Central Air', 'Gas Hot Air', 'Woodstove']
['Central Air', 'Gas Hot Air']
['Central Air', 'Gas Hot Air']
['Central Air', 'Gas Hot Air', 'Multizone A/C']

I'm sure it simple. I tried just

python Syntax (Toggle Plain Text)
if (count == 21):
  resfields = resfeaturelist [resline[count].split(",")]
  print resfields
if (count == 21):
  resfields = resfeaturelist [resline[count].split(",")]
  print resfields

Which just gives me an error.

Do you know of a good syntax reference?

Thanks!
Chris

grambo

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

4 posts
since Nov 2008

Permalink

Nov 10th, 2008

Re: Combining txt from 2 files

If I understand you correctly, you want to substitute the "VALUE" of a residence feature which would be a "KEY" in a dictionary. For each different value, there will be a different key. A dictionary is perfect for this. Example:

Python Syntax (Toggle Plain Text)
# features = open('features.txt').read()
 
features = '''1701|RES|HVAC|Central Air
1702|RES|HVAC|Window Unit
1703|RES|HVAC|Heat Pump - AC|
1704|RES|HVAC|No A/C
1705|RES|HVAC|Wall Unit - AC
1706|RES|HVAC|Multizone A/C
1708|RES|HVAC|Gas Hot Air
1722|RES|HVAC|Woodstove
1701|CND|HVAC|Central Air'''
 
featuresDict = {}
for item in features.split('\n'):
    if 'RES' in item:
        items = item.split('|')
        featuresDict[items[0]] = items[3]
 
resfield = ['1701','1706','1722']
 
# as a function, retain original list
def sub_features(s):
    return [featuresDict[item] for item in s[:]]
 
print sub_features(resfield)
 
# OR
# modify original list
for i, item in enumerate(resfield):
    resfield[i] = featuresDict[item]
 
print resfield
# features = open('features.txt').read()

features = '''1701|RES|HVAC|Central Air
1702|RES|HVAC|Window Unit
1703|RES|HVAC|Heat Pump - AC|
1704|RES|HVAC|No A/C
1705|RES|HVAC|Wall Unit - AC
1706|RES|HVAC|Multizone A/C
1708|RES|HVAC|Gas Hot Air
1722|RES|HVAC|Woodstove
1701|CND|HVAC|Central Air'''

featuresDict = {}
for item in features.split('\n'):
    if 'RES' in item:
        items = item.split('|')
        featuresDict[items[0]] = items[3]

resfield = ['1701','1706','1722']

# as a function, retain original list
def sub_features(s):
    return [featuresDict[item] for item in s[:]]

print sub_features(resfield)

# OR
# modify original list
for i, item in enumerate(resfield):
    resfield[i] = featuresDict[item]
    
print resfield

Both print:
>>> ['Central Air', 'Multizone A/C', 'Woodstove']

solsteel

Reputation Points: 86

Solved Threads: 40

Junior Poster

Offline

141 posts
since Mar 2007

Permalink

Nov 10th, 2008

Re: Combining txt from 2 files

That's what I needed. Thanks!!!!

grambo

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

4 posts
since Nov 2008

Permalink

Nov 11th, 2008

Re: Combining txt from 2 files

Click to Expand / Collapse Quote originally posted by grambo ...

Do you know of a good syntax reference?

Here's a good one: Python docs

Also, in the future it helps if you post the traceback of your error message instead of just saying "this gives me an error". It'll help the community debug your problem !

jlm699

Reputation Points: 355

Solved Threads: 292

Veteran Poster

Offline

1,102 posts
since Jul 2008

This thread is solved

Either the thread starter or a moderator has marked this thread as solved. You can most likely trust the responses and answers given. There is most likely no reason for any further responses to be posted here. If you have a related question, please start a new thread in this forum instead.

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in Python Forum Timeline: sending information with a wsdl file

Next Thread in Python Forum Timeline: New to pytong and need help with a database connection

DaniWeb Message

Cancel Changes

User Name
Password