Member Avatar for oliflorence

Hi,
i am working on a project where I have to read a .txt file for a feed and parse the data line by line.

I am using FSO in ASP to do this and it is working fine on small versions of the file, unfortunatly, the actual file is massive in size (670,000 kb) and I am not able to interprete the full file.

There is about 200,000 lines in the file of up to 4,000 characters each and i can read the first thousands fine but the problem starts when going further.

Due to the size of the file, I am unable to store it in a variable or array etc to manipulate it, the best way I have found is by looping through the lines and reading them one by one.
I can think of 2 solutions here, I could read a line and then remove it, do this for let say 1000 lines, move out of the file (which is now smaller by 1000 lines) and reaccess it and do the same again. The problem here is that I can't find a way to replace the line, is this possible to do??
I can write a line but it is a new line, it doesn't replace an existing one.

The second idea would be to loop through a number of line, for instance 1000, but only starting starting at line 1001 for instance, this way i could access the file several times. The only way I have found to do this is by using SkipLine but it means I still have to start the loop from the first line and ultimately, it ends crashing the same way.
Is there a way to start at loop at line???

Any pointer would be appreciated.

  • 2 Contributors
  • 1 Reply
  • 240 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by closetosane

Dani AI

Generated

Short answer: classic ASP's TextStream does not support in-place line replacement or a built-in "start at line N" seek. For very large text feeds the reliable patterns are (a) stream and process the file in a single pass without accumulating lines in memory, or (b) rewrite the file in chunks using a temporary file (the approach suggested). Long-running parsing also belongs outside a single web request (use a scheduled script or background worker) to avoid IIS/ASP timeouts and memory pressure.

A simple, memory-safe chunk-and-replace pattern (implements 's idea) — open the original for reading, read/process the first N lines, copy the remainder to a temp file, then atomically replace the original with the temp file:

<%
Const ForReading = 1
Const ForWriting = 2
Set fso = Server.CreateObject("Scripting.FileSystemObject")
src = Server.MapPath("bigfeed.txt")
tmp = Server.MapPath("bigfeed.tmp")

Set inFile = fso.OpenTextFile(src, ForReading)
Set outFile = fso.CreateTextFile(tmp, True)

' Process first chunk
maxLines = 1000
For i = 1 To maxLines
  If inFile.AtEndOfStream Then Exit For
  line = inFile.ReadLine
  ' process line here (do not accumulate in memory)
Next

' Preserve the remainder
Do While Not inFile.AtEndOfStream
  outFile.WriteLine inFile.ReadLine
Loop

inFile.Close
outFile.Close

fso.DeleteFile(src, True)
fso.MoveFile tmp, src
%>

Notes and cautions: use unique temp names and keep backups before deleting the original; ensure the swap happens on the same volume (rename/move is atomic on the same drive); avoid Response.Write inside long loops (it can fill buffers); consider recording progress (line count or a marker) in a database if the job must resume after failures. For best throughput and reliability on very large files, run parsing as a scheduled background task or a small compiled utility (PowerShell/.NET) that can seek by byte offsets or use streaming APIs — these avoid web-request timeouts and improve robustness.

Member Avatar for closetosane

Here is a thought that is a little dirty but may work for you.
Before reading the text file create a temp text file. when you start reading your file write the line into the temp file then delete it from the original text file. when you are finished reading all of the original text file close the temp file then delete the original, then rename the temp file to that of the original filename.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.