Subprocess Module Confusion

[bgk111]

I am trying to write a python program that will enter another program using a shell command, then issue commands to that program, then exit the program and continue running the rest of a python program. To be more precise, if I was simply running in a shell, I would type the following commands to get what I wanted:

-grass -text /water6/NO_BACKUP/Brandon/exposure/grassdata/WGS84_latlon/PERMANENT
-v.patch -e input=countytest,countytest2 output=countytesttotal
-exit

So to do this, I use the python code below:

#!/usr/bin/env python

import sys
import os
import subprocess
import time
os.system('clear')

trythis1   = 'grass -text /water6/NO_BACKUP/Brandon/exposure/grassdata/WGS84_latlon/PERMANENT'
trythis2   = 'v.patch -e input=countytest,countytest2 output=countytesttotal\n'
trythis3   = 'exit\n'

print "Updating zip code vector data using newest shape file"
proc = subprocess.Popen(trythis1, shell=True, stdin=subprocess.PIPE)
proc.stdin.write(trythis2)
proc.stdin.write(trythis3)
print('completed')

and for output I get this:

brandon@ca1flood:/water6/NO_BACKUP/Brandon/exposure/Scripts$ python ZipVectorUpdate.py 
Updating zip code vector data using newest shape file
completed
brandon@ca1flood:/water6/NO_BACKUP/Brandon/exposure/Scripts$ Cleaning up temporary files.....
Starting GRASS ...
GRASS GUI should be text
          __________  ___   __________    _______________
         / ____/ __ \/   | / ___/ ___/   / ____/  _/ ___/
        / / __/ /_/ / /| | \__ \\_  \   / / __ / / \__ \ 
       / /_/ / _, _/ ___ |___/ /__/ /  / /_/ // / ___/ / 
       \____/_/ |_/_/  |_/____/____/   \____/___//____/  

Welcome to GRASS 6.3.0 (2008) 
GRASS homepage:                          http://grass.osgeo.org/
This version running thru:               Bash Shell (/bin/bash)
Help is available with the command:      g.manual -i
See the licence terms with:              g.version -c
Start the GUI with:                      g.gui tcltk
When ready to quit enter:                exit

Patching vector map <countytest@PERMANENT>...
Patching vector map <countytest2@PERMANENT>...
Building topology ...
866 primitives registered      
Building areas:  100%
294 areas built      
189 isles built
Attaching islands:  100%
Attaching centroids:  100%
Topology was built.
Number of nodes     :   761
Number of primitives:   866
Number of points    :   0
Number of lines     :   0
Number of boundaries:   572
Number of centroids :   294
Number of areas     :   294
Number of isles     :   189
Intersections at borders will have to be snapped
Lines common between files will have to be edited
The header information also may have to be edited
v.patch complete. 2 vector maps patched
Closing monitors.....
Cleaning up temporary files.....
done



Goodbye from GRASS GIS

The odd thing is that, even if I dont put in a exit statement, the child process still closes out the grass program as if I did. Is there a way to make my primary program wait for the subprocess to complete so that things are done in a certain order? This program is part of a much larger set of programs that will need to run in a certain order, needing to wait until all the previous steps are completed before going on. Also, I had to add \n to get my statements to not run together, if I hadnt used them, my output file would have been named countytesttotalexit. There must be a better way to go about this. Any ideas?

[zachabesh]

Okay, I quickly glanced over your code. It doesn't look like it works at all. Am I right?

Note that subprocess.Popen only creates the instance, it doesn't run the command line. You need to use the methods communicate() or poll().

Once you get your command line to run, the python script should wait for it to finish and close before moving on. You don't need -exit

Test some of your code, and if you're still having problems let us see the error codes.

[bgk111]

The code as written above works and does exactly what i want it to do save that it does not wait for my commands to finish...hmm, ill take a look at the communicate option, what is the syntax for issuing a communicate option, i have looked at the documentation and want to make sure i understand it right, something like proc.communicate('command') ?

[harryhaaren]

Hey,

As far as i remember (and its been a while!) using subprocess.Popen() makes a process object, and to actually communicate with that process, you use the "communicate" method.

processName = subprocess.Popen("ls -l" , shell=True etc etc)
processName.communicate( Pipes ,Pipes , Pipes)

where Pipes are pipe objects like stdin,stdout,stderr.

There's probably somebody here who will tell you the more up-to-date method, but i've done that one before.
-Harry

[bgk111]

Ahh. Ok, so i experimented around a bit with the communicate and learned that it is not what I really wanted as it requires that I put all of my statements in one line separated by \n markers. Not a big deal, but I like the readability of individual statements. What I DID learn however, is that if you add a process.communicate() after all of your commands to the child process, then it will wait. Below is my code, working exactly as I wanted it to. Hope this is useful to others:

#!/usr/bin/env python

import sys
import os
import subprocess

os.system('clear')

#Location of shape files to create vectors, and the name to use for the new vectors
newzipshapelocation    = '/xxxx/NO_BACKUP/Brandon/exposure/grassdata/WGS84_latlon/PERMANENT/NewVect/ZIP/usa_zip.shp'
newcountyshapelocation = '/xxxx/NO_BACKUP/Brandon/exposure/grassdata/WGS84_latlon/PERMANENT/NewVect/County/'
grassopenlocation      = '/xxxx/NO_BACKUP/Brandon/exposure/grassdata/WGS84_latlon/PERMANENT'
zipout                 = 'v_us_zip_2008d'
countyout              = 'v_us_county_2008d'
store                  = 'TempMap'

#Code block to form command to be passed into grass
OPENGRASSCOMMAND       = 'grass -text '+grassopenlocation

#Code block to create and open a text file containg the filenames for all county shape files
os.system('find '+newcountyshapelocation+' -name "*.shp" > temp')
fa       = open('temp','r')
counter  = 0


#Code block to open a pipe to a child process in grass with statements to clean out old variables
proc    = subprocess.Popen(OPENGRASSCOMMAND, shell=True, stdin=subprocess.PIPE)
proc.stdin.write('g.remove vect='+countyout+' \n')
proc.stdin.write('g.remove vect='+zipout+'\n')

#Code block to step through the file using grass to create the new county vector map
for line in fa:
    outname  = line.replace(newcountyshapelocation, '').replace('.shp','').strip('\n') 
    proc.stdin.write('v.in.ogr -o dsn='+line.strip('\n')+' output='+outname+'\n')

    if counter == 0:
         outnameprev = outname
    
    elif counter == 1:
         proc.stdin.write('v.patch -e input='+outname+','+outnameprev+' output='+store1+'\n')
         proc.stdin.write('v.clean input='+store1+' output='+countyout+' tool=snap,break,rmdupl\n')
         proc.stdin.write('g.remove vect='+store1+'\n')        
         
    else:
         proc.stdin.write('v.patch -e input='+countyout+','+outname+' output='+store1+'\n')
         proc.stdin.write('g.remove vect='+countyout+'\n')
         proc.stdin.write('v.clean input='+store1+' output='+countyout+' tool=snap,break,rmdupl\n')
         proc.stdin.write('g.remove vect='+store1+'\n') 
    counter  = counter + 1


#Code to create new zip vector map
proc.stdin.write('v.in.ogr -o dsn='+newzipshapelocation+' output='+zipout+'\n')

#Code to reorder county categories in the new county vector map 


#Statement which forces the parent program to wait for the completion of the child
proc.communicate() 


print('Operation Complete')

fa.close()

I wont go into the output as it is very long, but I will say that the last thing that prints is the Operation Complete, which means that the program is now waiting for the child processes to complete, Huzzah!