Extreme Memory Leak

[HookDevil]

Hi to all,
I have a severe problem with memory when executing my application. At start it merely uses about 14.000 K without doing anything. Then, when the main code starts in a seperate thread, it spikes up to 700.000 K (observed in TaskManager). Program in question is an implementation of Run-Length Encoding compression technique (part of my diploma paper). The code itself is rather complex but it works as intended (besides the huge memory leak). The memory leak occured while I was testing it on a 4 MB file so if you guys and gals know of any general advice for working around memory leaks in general in vb.net please share it here... Thx in advance...

P.S.: If needed I can post parts of my code here (I think at least) just say the word...

[Momerath]

Are you sure it's a memory leak? Does the usage go down once your code is done running? Are you using unmanaged resources without Disposing of them?

Feel free to post code since there is no other way for us to tell you why you are leaking memory (if you actually are).

[HookDevil]

Thx for reply... I'll check memory again immediately and post the results...

[HookDevil]

Tested again... this time on 1 MB file... Memory leak definitely (spiked up to 200 MB and stayed at that)...

[HookDevil]

Not sure if I may actually do this (given that it's a part of my diploma paper) but I don't really need help with the compression code itself I need only help with memory management. So here comes the code (beware it's kinda confusing and large):

Private Sub btnStart_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnStart.Click

        Dim cPar As New cParameter

        cPar.Path = sPath
        cPar.fileList = tmpFileList
        cPar.fileNames = tmpFileNames
        cPar.fileLength = tmpFileLength

        tStart = New Thread(AddressOf Compress)
        tStart.Priority = ThreadPriority.BelowNormal

        tStart.Start(cPar)

        btnStart.Enabled = False

    End Sub

    Public Sub Compress(ByVal cParam As Object)

        Dim cPar As cParameter
        cPar = CType(cParam, cParameter)

        Console.WriteLine("Compress Thread: " & Threading.Thread.CurrentThread.ManagedThreadId.ToString)

        'coder ------------------------------------------
        For i As Integer = 0 To cPar.fileList.Count - 1

            Dim tPath As String = cPar.Path
            Dim flag As Byte = 0
            Dim writeLast As Boolean = False
            Dim lastByte As Byte = 0
            Dim tmpByte As Byte = 0
            Dim literalData As New List(Of Byte)
            Dim encodedData As New List(Of Byte)
            Dim fileReader As New FileStream(cPar.fileList(i), _
                                             FileMode.Open, FileAccess.Read)

            setMaxValues(cPar.fileLength(i), "TP")

            'read first symbol
            fileReader.Seek(0, SeekOrigin.Begin)
            lastByte = fileReader.ReadByte()
            showProgress()

            For j As Integer = 1 To cPar.fileLength(i) - 1

                tmpByte = fileReader.ReadByte

                If lastByte = tmpByte Then 'ENCODED
                    'literal buffer empty?
                    If literalData.Count = 0 Then
                        'encoded buffer full?
                        If encodedData.Count = 128 Then
                            'write encoded buffer and clear it
                            flag = getFlag(RLEscheme.Encoded, encodedData.Count)
                            writeEncoded(encodedData(0), flag, _
                                         cPar.fileNames(i), tPath)
                            encodedData.Clear()
                        End If
                    Else
                        'write literal buffer and clear it
                        flag = getFlag(RLEscheme.Literal, literalData.Count)
                        writeLiteral(literalData, flag, cPar.fileNames(i), tPath)
                        literalData.Clear()
                    End If
                    'add lastByte to encoded Buffer
                    encodedData.Add(lastByte)
                    lastByte = tmpByte

                    'tmpByte last symbol?
                    If j = cPar.fileLength(i) - 1 Then
                        'encoded buffer full?
                        If encodedData.Count = 128 Then
                            'write encoded buffer and clear it
                            flag = getFlag(RLEscheme.Encoded, encodedData.Count)
                            writeEncoded(encodedData(0), flag, cPar.fileNames(i), tPath)
                            encodedData.Clear()
                            writeLast = True
                        Else
                            'add tmpByte to encoded buffer, write it and clear it
                            encodedData.Add(tmpByte)
                            flag = getFlag(RLEscheme.Encoded, encodedData.Count)
                            writeEncoded(encodedData(0), flag, cPar.fileNames(i), tPath)
                            encodedData.Clear()
                            showProgress()
                            Continue For
                        End If
                    End If
                Else 'LITERAL
                    'encoded buffer empty?
                    If encodedData.Count = 0 Then
literal:
                        'literal buffer full?
                        If literalData.Count = 128 Then
                            'write literal buffer and clear it
                            flag = getFlag(RLEscheme.Literal, literalData.Count)
                            writeLiteral(literalData, flag, cPar.fileNames(i), tPath)
                            literalData.Clear()
                        End If
                        literalData.Add(lastByte)
                    Else
                        'encoded buffer full?
                        If encodedData.Count = 128 Then
                            'write encoded buffer
                            'and clear it
                            flag = getFlag(RLEscheme.Encoded, encodedData.Count)
                            writeEncoded(encodedData(0), flag, _
                                         cPar.fileNames(i), tPath)
                            encodedData.Clear()
                            GoTo literal
                        End If
                        'add lastByte to encoded buffer
                        encodedData.Add(lastByte)
                        'write encoded buffer and clear it
                        flag = getFlag(RLEscheme.Encoded, encodedData.Count)
                        writeEncoded(encodedData(0), flag, cPar.fileNames(i), tPath)
                        encodedData.Clear()

                    End If

                    lastByte = tmpByte

                    'tmpByte last symbol?
                    If j = cPar.fileLength(i) - 1 Then
                        'literal buffer full?
                        If literalData.Count = 128 Then
                            'write literal buffer and clear it
                            flag = getFlag(RLEscheme.Literal, literalData.Count)
                            writeLiteral(literalData, flag, cPar.fileNames(i), tPath)
                            literalData.Clear()
                            writeLast = True
                        Else
                            'add tmpByte to literal buffer , write and clear it
                            literalData.Add(tmpByte)
                            flag = getFlag(RLEscheme.Literal, literalData.Count)
                            writeLiteral(literalData, flag, cPar.fileNames(i), tPath)
                            literalData.Clear()
                            showProgress()
                            Continue For
                        End If
                    End If
                End If
                showProgress()
            Next

            If writeLast Then
                'add tmpByte to literal buffer, write and clear it
                literalData.Add(tmpByte)
                flag = getFlag(RLEscheme.Literal, literalData.Count)
                writeLiteral(literalData, flag, cPar.fileNames(i), tPath)
                literalData.Clear()
                writeLast = False
                showProgress()
            End If

        Next

    End Sub

    Private Sub writeLiteral(ByVal DATA As List(Of Byte), ByVal flag As Byte, ByVal fileName As String, ByVal tPath As String)

        Dim sPath As String = tPath & fileName & ".RLE"

        Dim Buffer(DATA.Count) As Byte

        Buffer(0) = flag

        For i As Integer = 1 To DATA.Count
            Buffer(i) = DATA(i - 1)
        Next

        'write to file

        Using fileWriter As New FileStream(sPath, FileMode.Append, FileAccess.Write)
            Dim tPos As Long = FileLen(sPath)
            fileWriter.Seek(tPos, SeekOrigin.Begin)
            fileWriter.Write(Buffer, 0, Buffer.Length)
            fileWriter.Close()
        End Using

    End Sub

    Private Sub writeEncoded(ByVal DATA As Byte, ByVal flag As Byte, ByVal fileName As String, ByVal tPath As String)

        Dim sPath As String = tPath & fileName & ".RLE"

        Dim Buffer(1) As Byte

        Buffer(0) = flag
        Buffer(1) = DATA

        'write to file
        Using fileWriter As New FileStream(sPath, FileMode.Append, FileAccess.Write)
            Dim tPos As Long = FileLen(sPath)
            fileWriter.Seek(tPos, SeekOrigin.Begin)
            fileWriter.Write(Buffer, 0, 2)
            fileWriter.Close()
        End Using

    End Sub

    Private Function getFlag(ByVal scheme As RLEscheme, ByVal runCount As Integer) As Byte

        'convert runCount - 1 to binary system
        Dim tmpBit As String = ByteToBit(runCount - 1)

        Select Case scheme

            Case RLEscheme.Literal
                'add "0"
                tmpBit = "0" & tmpBit
            Case RLEscheme.Encoded
                'add "1
                tmpBit = "1" & tmpBit
        End Select

        'convert to decimal system and return value
        Dim tmpByte As Byte = BitToByte(tmpBit)

        Return tmpByte

    End Function

    Private Function BitToByte(ByVal Input As String) As Byte

        Dim bit(7) As Integer
        Dim rez As Byte = 0

        For i As Integer = 0 To Input.Length - 1
            bit(i) = CInt(Input.Substring(i, 1))
            rez += bit(i) * (2 ^ (7 - i))
        Next

        Return rez

    End Function

    Private Function ByteToBit(ByVal Input As Integer) As String

        Dim rez As String = Convert.ToString(Input, 2).PadLeft(7, "0"c)

        Return rez

    End Function

[HookDevil]

Observed another weird behaviour:
The main compression algorithm is in another thread yet the main form freezes frequently. I revised the code (god knows for how many times) and can't find any mistake in it. Used almost the same approach while making a Data Deduplication algorithm but the deduplication doesn't have any memory leaks and the main form doesn't freeze. Right now I'm clueless as to why this is happening... Any suggestions are greatly appreciated...

[Momerath]

Only thing I can see just glancing over the code is you don't dispose of the filereader that you create in line 36.

Also force a garbage collection when you are done processing the file and see if memory usage goes down.

[HookDevil]

Already tried disposing filereader but that didn't help. Will try GC thx for suggestion

[HookDevil]

GC didn't help either but I found out where it is leaking so much memory... commented out all the showProgress() calls and now it's working like a charm. Two problems left tho: 1. can't figure out why it is leaking memory there, 2. I can't see the progress of the algorithm... Just to be sure that I didn't do some stupid mistake here that I'm not seeing I'll post the showProgress() code:

Public Sub showProgress()

        If Me.InvokeRequired Then
            sProgress = New showProgressDelegate(AddressOf showProgress)
            Me.BeginInvoke(sProgress)
        Else
            pbarUP.Value += 1
            pbarTP.Value += 1
        End If

    End Sub

Interesting:
The above code is recommended from M$ (saw it on MSDN). Basically it just updates controls on main form (main thread) from my 2nd thread by invoking itself automatically when needed.
More Interesting:
Almost identical code works like a charm on my other App but here it leaks memory.

Clueless again....

[HookDevil]

Found out what the problem was: compression code is to fast for progress bar. I'm suspecting that it can't update fast enough so that VB puts the new value somewhere into memory and doesn't clear it properly after the progressbar has been updated hence it stays in memory while the main form is active. Since the showProgress is called several thousand times this means, in theory, (several thousand * value) bytes are written to memory. I already suspected this before but didn't really think that the answer could be so easy. Don't know how M$ coded their progressbar control but if it can't update that fast I'm suspecting the code is poorly written (no offense here really M$). Nonetheless, now I'm a little bit wiser... Anyway thx for the suggestions Momerath...