Need your advice here

[renzlo]

Hi all,

I am making a file monitoring tool, I've asked google what's the best approach in doing that and it returned that I should use filesystemwatcher, others suggested a polling program. Now I've decide to ask here for my final decision, hoping for some advices.

Here's my the current scenario:

My server path (local): C:\FilServer

List of folder to monitor for created event:
C:\FilServer\AB_Incoming - zip files
C:\FilServer\AC_Incoming - xml and tif files

List of folder to monito for deleted event:
C:\FilServer\AB_Outgoing - zip files
C:\FilServer\AC_Outgoing - xml and tif files

When the folder raises event, the detected file will be following this routine:

Move the file to d:\backup and copy to e:\source[date][time] and write sumary log.

Thanks in advance.

[Reverend Jim]

Definitely use the filesystemwatcher. One thing to consider is how files are created in the folder. You may have a problem with timing. For example, if you are watching a folder for *.txt files, the event will be triggered when the file is created, but that doesn't necessarily mean that the file is available to be moved. The file will likely still be in use by the process that created it. If you have control over the process that creates the file I suggest you use the following sequence:

1. create the file with the extension ".tmp" or something similar
2. write all data to the file
3. close the file
4. rename the file to *.txt

That way you can be sure that when a *.txt file appears it is available for processing. You could always just build in a delay between file detection and file moving but no matter how much of a delay you build in, there will always be a situation where that delay will never be long enough and when you build in delays you run the risk of slowing down your process when multiple files come in.

I did this for ten years in a real-time system where data absolutely had to be processed on time. Because our requirements were so complex we decided not to write a custom tool for the file/folder watch. Instead we used a terrific program called adTempus from ArcanaDev. I can't say enough good things about the product or the extremely high quality of their techniical support. Yes, this is a blatant plug.

[renzlo]

HI Jim thanks for the inputs. Regarding on how files are created in the folder, actually that's the first problem that I have encountered andI have come up with this function that reads the files if it's ready to be written and it solved my problem.

What should be my approach in monitoring files? Do I have to monitor only the root folder (c:\FilServer) and turn on the include subdirectories of FS or do I have to create instance in each folder?

[Reverend Jim]

There is a property that you can enable to watch subfolders. I'm curious as to the function that you wrote to determine availability. In one of our work systems, a process on our EMS (AGC/SCADA) network sent data files (8000+ analog and status points) every five minutes by FTP. The process was

1. open the FTP connection
2. send datafile.txt
3. close the connection
4. reopen the connection
5. compare file sizes (sent & received)
6. if different sizes then resend

The problem was that the processing at the receiving end (my code) had no way of knowing whether the complete process was done. If there was a delay between steps 3 & 4 then there was a good chance that I would move the file before the size check was done. In which case the sender would resend the file causing general mayhem. If the sender had used temp files then there would have been no problem. So depending on how the files are created you may need to rethink your file availability check. Just a heads up that may not be necessary.

[renzlo]

Hi Jim,

This is the snippet that I am talking about:

'Is file ready?
    Private Function FileReadyToRead(ByVal filePath As String, ByVal maxDuration As Integer) As Boolean
        Dim readAttempt As Integer = 0
        While readAttempt < maxDuration
            readAttempt += 1
            Try
                Using stream As New StreamReader(filePath)
                    Return True
                End Using
            Catch
                System.Threading.Thread.Sleep(1000)
            End Try
        End While
        Return False
    End Function

Luckily, we are not using FTP connection, our client is just connecting to our shared folder and there's a folder inside those paths that I want to monitor named _filserver_state wherein the client creates a temporary file that gives me the information that the client is currently sending files. If this the case, what should be the best way?