Want to execute a command on all files recursively

[ppraj]

Hi,

I have folder which is deep rooted (i.e many folders and files under it)
My requirement is to execute a command on each file on the above folder recursively and it should be fast.

My approach was to use to the Directory Get Files method to get the files in an array and loop though each file . While Looping through , I use the Process class something like this

Process P =new Process();
p.StartInfo.command="someexe";
p.StartInfo.arguments="<dynamically set by the filevalue in the loop everytime>";
p.Start();
p.WaitForExit();

The above method of execution takes an hour or so. Can I make the process faster using threads if so how ?

[sknake]

This sounds too much like something to do with spyware, viruses, or other generally annoying things. Can you explain what you are doing first?

[ppraj]

Hehe..
It is for labelling the source control (i.e Clearcase view that has the files loaded in it ). So trying to automate the same with a C# script which will call "cleartool.exe with a parameter of the path of the file to do the job. Hence the requirement to loop over all the files.

[sknake]

Well for starters the p.WaitForExit() does just that... do you need to wait for the process to exit or are you just trying to stop 9999 processes from spawning at once? If so you could do something to only have N processes started by the application at once.

Threading is a possible solution but there is also overhead in creating and tearing down a thread, so you need to give us a little more information on this.

[ppraj]

Let us not spawn 9999 processes of the cleartool.exe at once . That will make it really fast, bur atleast create a thread or two to the job run faster than the current one .

[LizR]

Best answer would be to recursively get a list of files, and pop the filename in a list

Then have a class which spawns say 5 threads max, and processes the file, so you dont thrash the heck out of everything but also achieve a realistic effect

[LizR]

ignore - dup

[ppraj]

Anybody ?

[LizR]

ppraj, your last post has no point. You were the last to answer before me, I said something your response is clearly not in relation to that, so what was the point?

[ppraj]

Yeah, I didn not see that..thanks
I'm aready loopinh though the files list, so the point it to spawn the prcoess across 5 threads. How can I do that ?
I also have a query like , when you spawn the process, how are you going to keep track of the files that are already prcessed by some other thread. To that I mean , we should not process a file in the list if the other thread has taken care of it.

[LizR]

Thats the joy of programming. *YOU* decide how it works.

me? I would use a counter, and when a thread starts, it adds one, when it finishes it decreases by 1, if its less than 5 and there are still filles to process, then, start new thread, if not, keep waiting.

How you decide which you have processed, again, up to you, you could use a stack and pop off each item you wanted, or a list, or mark an array, or use a counter..

Its up to *YOU*

[sknake]

Build a list of files, divide the list 5 times and pass the thread their responsible list. Or you could pass each thread a number of characters for file prefixs, ex. thread#1 does a-f, thread#2 does g-z. Or you could request the next file name from the main thread in each child thread after it finishes executions.

Obviously some of these don't make sense but it gives you a starting point.

[sknake]

Thats the joy of programming. *YOU* decide how it works.

me? I would use a counter, and when a thread starts, it adds one, when it finishes it decreases by 1, if its less than 5 and there are still filles to process, then, start new thread, if not, keep waiting.

How you decide which you have processed, again, up to you, you could use a stack and pop off each item you wanted, or a list, or mark an array, or use a counter..

Its up to *YOU*

I didn't realize this thread spilled on to the second page, I think i just reposted what you said.

Oops...

[ppraj]

What I would like to do is create 2 -3 threads and in the parent loop (i.e the loop of the filelist determine which thread is free and pass on the processing information to the same)

[LizR]

Thats ok, read what I wrote, and you can a way to achieve that.
Try and code it, and when you struggle *then* we will help with code.

[ppraj]

Let us not execute the process. i will do it later.But help me to print the filenames from the loop , using the thread concept.

Here you go:

string[] filePaths = Directory.GetFiles(@"c:\MyDir\");
Thread t=new Thread(); //What method should I instantiate, sincei have the below loop only, and for each f it should use any of the thread create dabove.
foreach (string f in filepaths)
{
}

[LizR]

This is why you need to design what you're going to do before you code it.

The thread needs something to run, so the thing you want to run has to be separate from the method you're in right now.

You might consider sending the thing to be run by the thread a parameter - take a look at the helpfile for threads for examples

[ppraj]

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.IO;
using System.Collections;

namespace Threading
{
    class Program
    {


        public static string[] secArray = new string[10];

        static void doSecLoop()
        {
            foreach (string t in secArray)
                Console.WriteLine("second" + t);

        }     


        static void Main(string[] args)
        {
            string[] filePaths = Directory.GetFiles(@"I:\Software\", "*.*", SearchOption.AllDirectories);
            int maXLen = filePaths.GetLength(0);
            int halveLen = maXLen / 2;
            string[] secArray = new string[filePaths.Length - halveLen];
            Array.Copy(filePaths, halveLen, secArray, 0, filePaths.Length - halveLen);  

            Thread t = new Thread(doSecLoop);
            t.Start();
             //Console.ReadLine();
            //filePaths.
             for (int i = 0; i <= halveLen - 1;i++)
                Console.WriteLine("first" + filePaths[i]);

             Console.ReadLine();

        }

    }
}

In the above program , uses 2 thread one main thread and other a created thread. On executing the program, the main thread tends to execute . The issue I found is that in the doSecLoop method I 'm using a array "secArray", but it's value is not pouplated in the method from the main method .It has 10 elements only.
What is the issue

[LizR]

You've not bothered with code tags which makes reading it difficult
you redeclare secArray as a local array, so its out of scope.

[ppraj]

using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.IO;
using System.Collections;

namespace Threading
{
    class Program
    {
       
       
        public static string[] secArray = new string[10];

        static void doSecLoop()
        {
            foreach (string t in secArray)
                Console.WriteLine("second" + t);

        }     
           
 
        static void Main(string[] args)
        {
            string[] filePaths = Directory.GetFiles(@"I:\Software\", "*.*", SearchOption.AllDirectories);
            int maXLen = filePaths.GetLength(0);
            int halveLen = maXLen / 2;
            secArray = new string[filePaths.Length - halveLen];
            Array.Copy(filePaths, halveLen, secArray, 0, filePaths.Length - halveLen);  
            
            Thread t = new Thread(doSecLoop);
            t.Start();
             //Console.ReadLine();
            //filePaths.
             for (int i = 0; i <= halveLen - 1;i++)
                Console.WriteLine("first" + filePaths[i]);
             
             Console.ReadLine();
           
        }
        
    }
}

Thanks man, I did as u advised and it printed bot the arrays perfectly.
Now, I would like to invoke "cleartool.exe " on each of the filename .What should be the best way to do it. Shoild I use Process class , but it takes a lot of overhead , anything easier so that I can reuse two process objects , without destorying the sam euntil all the array is finished.

[LizR]

OK, so whatever "cleartool.exe" does, you have a filename, do you know how to run cleartool.exe on a specific file from a command line for example?

[ppraj]

Yes it is something like this

cleartool.exe mklabel <file>

[LizR]

So you have some how to run a process code, and send it parameters..
When you have some code to show, come back with any problems - be sure to include the relevant code, and whats happening/not happening.