Member Avatar for raheel88

All,

What is the most efficient way of writing several lines of data to a *.txt file? Currently, I store the required text in memory then write it all to a user-specified text file at the end, similar to the following;

Dim text As String = ""
text = text + "1st line of text" & vbNewLine
text = text + "2nd line of text" & vbNewLine
text = text + "3rd line of text" & vbNewLine

Dim path As String = "C:\Temp\fileName.txt"

My.Computer.FileSystem.WriteAllText(path, text, True, System.Text.Encoding.Default)

This works well enough for up to about 1000 lines of text. However, I often need to write 3000+ lines to a single file and it is noticeably slow using the above method. Is there a more efficient way?

Thanks in advance!

  • 5 Contributors
  • 5 Replies
  • 4K Views
  • 1 Hour Discussion Span
  • Latest Post Latest Post by dionterry

Dani AI

Generated

+1 to and — the real bottleneck is building one giant string in memory, not the disk write itself. WriteAllText is fine when the whole payload already exists, but for thousands of lines streaming to disk and letting the runtime buffer is faster and much kinder on memory. 's StringBuilder idea helps when lines must be assembled first, and 's StreamWriter approach is the right direction; the pattern below combines an explicit buffer and encoding for predictable, efficient writes.

Imports System.IO
Imports System.Text

Sub WriteLines(path As String, lines As IEnumerable(Of String), append As Boolean)
    Dim mode = If(append, FileMode.Append, FileMode.Create)
    Using fs As New FileStream(path, mode, FileAccess.Write, FileShare.Read, bufferSize:=65536, options:=FileOptions.Asynchronous)
        Using w As New StreamWriter(fs, New UTF8Encoding(encoderShouldEmitUTF8Identifier:=False))
            w.NewLine = Environment.NewLine
            For Each line In lines
                w.WriteLine(line)
            Next
        End Using
    End Using
End Sub

Why this helps: no O(n^2) concatenation, far fewer small system calls, and explicit UTF‑8 avoids platform surprises. Additional tips: build text in moderate chunks (8–64 KB) if in‑memory assembly is unavoidable; keep AutoFlush off so buffering works; in ASP.NET open the FileStream with asynchronous options and use WriteLineAsync/FlushAsync to get true overlapped I/O (otherwise WriteAsync can still consume threadpool threads). For safe appends under concurrency, consider writing to a temp file and atomically replacing the target.

Recommended Answers

All 5 Replies

Member Avatar for deceptikon

File operations are inherently slow, relative to in-memory operations. I'd probably go with writing smaller blocks than dumping all of the file contents to a line for a single WriteAllText call. The framework can handle its own buffering for performance and you can limit the amount of string processing you're doing in memory as well as how much memory you're using. A big hit to performance often involves crossing cache lines, which large strings are wont to do.

The absolute fastest you can get would probably force you to drop down to WriteFile using PInvoke, but WriteAllText is one of the fastest options from the framework from benchmarks I've seen.

Member Avatar for tdrosiadis

You could use the StringBuilder class and write chuncs of text to your file. As a .net class it might be less resource concuming. Havn't tryed it though. Just a though

Edited by deceptikon because: Snipped fake signature
Member Avatar for Reverend Jim

Assuming that you are replacing the text in the file with the new text, I would do the following

Dim text As New System.Text.StringBuilder

text.Append("1st line of text" & vbCrLf)
text.Append("2nd line of text" & vbCrLf)
text.Append("3rd line of text" & vbCrLf)
.
.
.    
System.IO.File.WriteAllText(filename,text.ToString())

Depending on how many lines you have, repeated concatenation as in your code can get noticibly slow. StringBuilder is much more efficient. Try the following and see for yourself.

Dim text As String = ""

For i As Integer = 1 To 1000
    text &= "this is line " & i & vbCrLf
Next

and compare it to

Dim text As New System.Text.StringBuilder

For i As Integer = 1 To 1000
    text.Append("this is line " & i & vbCrLf)
Next
Member Avatar for raheel88

Thanks all, what Reverend Jim suggested is the best option by the looks of it.

Having internet problems so will mark as solved when I get a chance :)

Thanks again!

Member Avatar for dionterry

Best way is:

Dim sr as new system.io.streamwriter("Location of txt file")
sr.write("Text to write") 'You can also use writeline etc.
sr.close

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.