Puzzled by subprocess

[fordprefect]

Hi all!

I'm a complete pyhthon n00b and I'm trying to execute an external command from within my python script. For this I use the subprocess module. Here is all the relevant code:

class PipeThread(threading.Thread):
    def __init__(self, fin):
        self.fin = fin
        self.sout = ""
        threading.Thread.__init__(self)
    def run(self):
        self.sout = self.fin.read()
    def getOutput(self):
        return self.sout

class ExecutionFailed(Exception):
    def __init__(self, value):
        self.parameter = value
    def __str__(self):
        return str(self.parameter)

def execute(command, name = None):
    try:
        if name == None:
            name = "'" + command + "'"
        cmd = subprocess.Popen(command, 0, None, subprocess.PIPE, subprocess.PIPE, subprocess.PIPE)
        cmd.stdin.close()
        outPipe, errPipe, retcode = PipeThread(cmd.stdout), PipeThread(cmd.stderr), cmd.wait()
        outPipe.start(), errPipe.start()
        out, err = outPipe.getOutput(), errPipe.getOutput()
        if retcode != 0 and err != '':
            raise ExecutionFailed, name + " exited with error code " + str(retcode) + " and errors:\n" + err
        elif retcode != 0:
            raise ExecutionFailed, name + " exited with error code " + str(retcode) + " and output:\n" + out
        elif err != '':
            return out + "\n" + name + " gave warnings:\n" + err
        else:
            return out
    except Exception, e:
        if isinstance(e, ExecutionFailed):
            raise
        else:
            raise ExecutionFailed, "Error while executing " + name + ":\n" + str(e)

def javac(out, root):
    return execute("javac -d" +
                   " " + os.path.normpath(out) +
                   " " + os.path.normpath(root),
                   "Java Compiler")

and finally:

javac("classes", "src/package/main.java")

So far I have only tested this in Windows. javac is a console application and it seems that like all console applications it has a limited stdout (and stderr?) buffer size. Thus, if you don't empty its stdout stream fast enough it will appear to have hung. This is why I use the PipeThread class which is supposed to read data from the subprocess stdout and stderr streams and store them in a string, thus preventing a stall.
My problem is that it still won't work. If large amouts of data are sent to either stdout or stderr my script will hang. In my case this happens while javac is running. If I close it with Ctrl+C I see that my script has indeed read a certain amount of output from the subprocess (always the same) but no more. So it seems to me that while self.fin.read() does read data from self.fin it doesn't actually empty it!
Can anyone explain this and tell me what the correct appoach is or if it's possible to change the default buffering behavior for windows applications?

Any help is greatly appreciated!!!

[fordprefect]

Never mind...
The problem was embarassingly simple:

outPipe, errPipe, retcode = PipeThread(cmd.stdout), PipeThread(cmd.stderr), cmd.wait()
outPipe.start(), errPipe.start()

ought to be:

outPipe, errPipe = PipeThread(cmd.stdout), PipeThread(cmd.stderr)
outPipe.start(), errPipe.start()
retcode = cmd.wait()

[bumsfeld]

Thanks for answer, for complete pyhthon n00b , this is pretty complex coding!!!

[rajatdudeja]

Hi all!

I'm a complete pyhthon n00b and I'm trying to execute an external command from within my python script. For this I use the subprocess module. Here is all the relevant code:

class PipeThread(threading.Thread):
    def __init__(self, fin):
        self.fin = fin
        self.sout = ""
        threading.Thread.__init__(self)
    def run(self):
        self.sout = self.fin.read()
    def getOutput(self):
        return self.sout

class ExecutionFailed(Exception):
    def __init__(self, value):
        self.parameter = value
    def __str__(self):
        return str(self.parameter)

def execute(command, name = None):
    try:
        if name == None:
            name = "'" + command + "'"
        cmd = subprocess.Popen(command, 0, None, subprocess.PIPE, subprocess.PIPE, subprocess.PIPE)
        cmd.stdin.close()
        outPipe, errPipe, retcode = PipeThread(cmd.stdout), PipeThread(cmd.stderr), cmd.wait()
        outPipe.start(), errPipe.start()
        out, err = outPipe.getOutput(), errPipe.getOutput()
        if retcode != 0 and err != '':
            raise ExecutionFailed, name + " exited with error code " + str(retcode) + " and errors:\n" + err
        elif retcode != 0:
            raise ExecutionFailed, name + " exited with error code " + str(retcode) + " and output:\n" + out
        elif err != '':
            return out + "\n" + name + " gave warnings:\n" + err
        else:
            return out
    except Exception, e:
        if isinstance(e, ExecutionFailed):
            raise
        else:
            raise ExecutionFailed, "Error while executing " + name + ":\n" + str(e)

def javac(out, root):
    return execute("javac -d" +
                   " " + os.path.normpath(out) +
                   " " + os.path.normpath(root),
                   "Java Compiler")

and finally:

javac("classes", "src/package/main.java")

So far I have only tested this in Windows. javac is a console application and it seems that like all console applications it has a limited stdout (and stderr?) buffer size. Thus, if you don't empty its stdout stream fast enough it will appear to have hung. This is why I use the PipeThread class which is supposed to read data from the subprocess stdout and stderr streams and store them in a string, thus preventing a stall.
My problem is that it still won't work. If large amouts of data are sent to either stdout or stderr my script will hang. In my case this happens while javac is running. If I close it with Ctrl+C I see that my script has indeed read a certain amount of output from the subprocess (always the same) but no more. So it seems to me that while self.fin.read() does read data from self.fin it doesn't actually empty it!
Can anyone explain this and tell me what the correct appoach is or if it's possible to change the default buffering behavior for windows applications?

Any help is greatly appreciated!!!

Hi,

I found your code helped solved a similar problem that I've been trying to solve for the lasta of couple of days.

I actually started getting errors my subprocess is giving me. Earlier I was getting no response from the child process.

Since you haven't posted the complete code details e.g. you did not posted the definition of def start() function.

I'll be grateful to you if you can post me the complete code.

Thanks and regards,
Rajat