execve problem: How to construct argument lists

[QuantumMechanic]

I am hoping someone here can help me with a little bit of trouble using the execve() call. I have some code that needs to call an external script under CNL (Compute Node Linux, i.e. - Cray XT4/XT5). CNL is extremely lean and is missing enough libraries to make using something as common as /usr/bin/env impossible. I'm trying to use execve() to pass the arguments to the script and a custom (read stripped-down) set of environmental variables as we want very tight control over these. (Many of the paths in $PATH are not available to the file system on the compute nodes.)

Here's what I have:

char *cmdstr; // This will hold a string with the name of the script
char **my_args = NULL; // these next two should be self-explanatory
char **my_env = NULL;

I have a simple little function to append a string to the array of strings that looks something like this:

void add_strtoarray(char ***str_array, int *num_elements,
                    const char *my_string)
{
    (*num_elements)++;
    *str_array = (char **) realloc(*str_array,
                                     (*num_elements)*sizeof(char *));
    (*str_array)[(*num_elements) - 1] = strdup(my_string);

So ... in my code I do the following:

// Set up the info for execve
char *cmdstr;
char **my_env = NULL;
char **my_args = NULL;
int num_my_env = 0;
int num_my_args = 0;
pid_t pid;

// Populate my environment
add_strtoarray(&my_env,&num_my_env,env_ROOT);
add_strtoarray(&my_env,&num_my_env,env_EXTRALIBS);
add_strtoarray(&my_env,&num_my_env,env_EXTRAINCLUDE);
.
.
.
add_strtoarray(&my_env,&num_my_env,env_PYTHONHOME);
add_strtoarray(&my_env,&num_my_env,env_PYTHONPATH);
add_strtoarray(&my_env,&num_my_env,env_PATH);

// Generate the argument list
add_strtoarray(&my_args,&num_my_args,"-i");
add_strtoarray(&my_args,&num_my_args,inputdeck);
.
.
.
add_strtoarray(&my_args,&num_my_args,"-o");
add_strtoarray(&my_args,&num_my_args,outputdeck);

// UTILPATH is in a site-specific header file
// script_name is supplied on the command line
sprintf(cmdstr,"%s/%s",UTILPATH,script_name);

pid = vfork();
if (pid == -1) {
    // fork error code here
}
if (pid == 0) {
    exeve(cmdstr,my_args,my_env);
}
else {
    wait(NULL);
}

According the the man pages, the native environment variable const char **environ is of the same type as my declarations. However, if I modify the script referred to in cmdstr to print out its argv[] list I see that only the first of my strings in my_args is passed in. That is ... for the example above the script thinks that it has a single element in argv (ignoring the name of the executable), namely "-i". Similarly, the environment doesn't seem to be getting passed properly to the script.

I'm completely baffled by this! Any suggestions would be helpful.

[nezachem]

Few things are obvious right away:
1. cmdstr is not initialized.
2. my_env and my_args are not terminated by NULL.
3. A first element of my_args is a program name. The way you set the arguments up is not fatal, but highly unconventional.
4. You must test the value returned by realloc.
5. You are working too hard. I'd just go with

char * my_env[] = {
    env_ROOT,
    env_EXTRALIBS,
    ....
    env_PATH,
    0
};

and a similar setup for my_args.

[QuantumMechanic]

My apologies for snipping out too much of my code and not providing details.

1) cmdstr is indeed initialized, just not in the bits of code I provided

2) So ... should I do a realloc on my_args (and my_env ) to add a NULL pointer after the last string?

3) from the man page: execve() executes the program pointed to by filename. filename must be either a binary executable, or a script starting with a line of the form "#! interpreter [arg]". In the latter case, the interpreter must be a valid pathname for an executable which is not itself a script, which will be invoked as interpreter [arg] filename. What I have is a script with the first line as:

#!/path_to_python/python2.5

So this should effectively make cmdstr the first argument to the Python interpreter. At least that's my understanding. Still, I'll try prepending cmdstr to my_args and see if anything improves.

4) The return value from realloc is tested, just not shown in the code snippet I provided for clarity's sake. (Nor are my corresponding free() calls.)

5) Constant declarations lead to portability problems. I cannot afford portability problems. Besides, I may not know the values of these environment varibles (or how many I might need to supply) prior to run time.

Few things are obvious right away:
1. cmdstr is not initialized.
2. my_env and my_args are not terminated by NULL.
3. A first element of my_args is a program name. The way you set the arguments up is not fatal, but highly unconventional.
4. You must test the value returned by realloc.
5. You are working too hard. I'd just go with

char * my_env[] = {
    env_ROOT,
    env_EXTRALIBS,
    ....
    env_PATH,
    0
};

and a similar setup for my_args.

[nezachem]

My apologies for snipping out too much of my code and not providing details.

Accepted.

For us to be on a same page, can you provide a minimal compilable snippet which exhibits the behaviour you observe?

2) So ... should I do a realloc on my_args (and my_env ) to add a NULL pointer after the last string?

Absolutely yes.

3) from the man page: execve() executes the program pointed to by filename. filename must be either a binary executable, or a script starting with a line of the form "#! interpreter [arg]". In the latter case, the interpreter must be a valid pathname for an executable which is not itself a script, which will be invoked as interpreter [arg] filename.

<An attempt to explain deleted. My wording rather confuses than clarifies matters. I'll return to it later>

5) Constant declarations lead to portability problems. I cannot afford portability problems. Besides, I may not know the values of these environment varibles (or how many I might need to supply) prior to run time.

Frankly, I don't see any portability problems here. The second point is very valid, so please disregard my advice.

[QuantumMechanic]

Thanks for your help! I've been a numerical guy most of my life and I've managed to avoid calling external programs in my code ... until now. This was my first foray into argument lists and such. (Yes, I've been writing numerical code for 15 years an never needed to use the exec() family before. Amazing, isn't it!) That's probably why I was being so unconventional with it.

So ... what I discovered is that just tacking the (char *) NULL on the end didn't work, but when I did that and prepended the name of the script being called everything worked like a charm. I can now get the Python script to print the list of arguments and environmental variables and everything's exactly as it should be. I have a test run queued up on a supercomputer right now and hopefully I'll get the behavior I need. Your suggestions were spot on!

As far as the portability thing goes, I try never to make static declarations of things. I try to always make code that goes and looks for things or gets variables at runtime so that quirky configurations don't come back to bite me.

If you ever need any help crunching numbers, let me know. I'm much better at that sort of thing. Strings and such make my flesh creep, but I can calculate eigenvectors with the best of them.

Accepted.

For us to be on a same page, can you provide a minimal compilable snippet which exhibits the behaviour you observe?

Absolutely yes.

<An attempt to explain deleted. My wording rather confuses than clarifies matters. I'll return to it later>

Frankly, I don't see any portability problems here. The second point is very valid, so please disregard my advice.