Member Avatar for Fortinbra

My web application has the ability to process data from word docs and PDF files. We recently had a client who attempted to run files that are of a significantly larger size than anything we tested with. 50MB-500MB PDFs compared to our test files of 1MB-30MB files. If they don't get a timeout error, they get a System.OutofMemoryException. I know what to do about the out of memory exception, but I can't tell the client they can't use those files.

I'd like to run the processing of the files as a background process, but I need to show progress on the files to the user. We're using Aspose.Total and .NET 4, what are everyone else's thoughts on how to accomplish this? We use MSSQL as the backend, and out application supports 1000+ concurrent users, so performance is key.

  • 2 Contributors
  • 4 Replies
  • 139 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by riteshbest

Dani AI

Generated

The easiest reliable pattern for large-file work is to treat each upload as a queued job handled by a separate worker process (Windows Service, console worker, Hangfire job, or cloud WebJob). This keeps the heavy PDF work out of IIS, avoids request timeouts and reduces the risk of System.OutOfMemoryException on the web tier. was right to suggest an external worker; ’s AJAX polling idea for UI progress can be reused or replaced by a push channel for better responsiveness.

A minimal, practical flow:

  • Client uploads file; server persists the file to shared storage or blob storage (avoid storing 100s of MB in a SQL row) and inserts a Job row with metadata and status = Pending.
  • One or more worker processes atomically claim Pending jobs, process the file in a streaming/page-by-page fashion (don’t load the entire PDF into memory), and write periodic progress to the Job row (percent, step, logs).
  • Web UI either polls the Job row or receives pushes (SignalR) to show live progress and final result URL.
  • Workers set Completed/Failed, increment Attempts, and move dead-lettered jobs after N retries.

A simple job table and claim pattern (atomic claim inside a transaction):

Jobs (JobId PK, FilePath, Status, Progress INT, Attempts INT, WorkerName, CreatedAt, UpdatedAt)

BEGIN TRANSACTION;
DECLARE @jobId INT;
SELECT TOP 1 @jobId = JobId
FROM Jobs WITH (ROWLOCK, READPAST)
WHERE Status = 'Pending'
ORDER BY CreatedAt;

IF @jobId IS NOT NULL
  UPDATE Jobs
  SET Status = 'Processing', WorkerName = 'worker-1', UpdatedAt = GETDATE()
  WHERE JobId = @jobId;
COMMIT TRANSACTION;

Consider these practical points: run workers as 64-bit processes so they can use more memory; throttle concurrency so only a few large PDFs are processed at once; implement retries and a dead-letter queue; keep files outside the DB (filesystem or blob store) to avoid bloating SQL Server; and use SignalR for push updates if you want sub-second UX (SignalR introduction). For managed job orchestration on .NET 4, Hangfire provides durable jobs, retries and a dashboard to simplify workers (Hangfire docs).

Recommended Answers

All 4 Replies

Member Avatar for riteshbest

My web application has the ability to process data from word docs and PDF files. We recently had a client who attempted to run files that are of a significantly larger size than anything we tested with. 50MB-500MB PDFs compared to our test files of 1MB-30MB files. If they don't get a timeout error, they get a System.OutofMemoryException. I know what to do about the out of memory exception, but I can't tell the client they can't use those files.

I'd like to run the processing of the files as a background process, but I need to show progress on the files to the user. We're using Aspose.Total and .NET 4, what are everyone else's thoughts on how to accomplish this? We use MSSQL as the backend, and out application supports 1000+ concurrent users, so performance is key.

Hi Fortinbra,
If you want to perform background processing, implement background worker.
Also, you can show progress:

worker.WorkerReportsProgress = true;

and then implement ProgressChanged event handler to indicate the progress.

private void bw_ProgressChanged(object sender, ProgressChangedEventArgs e)
{
    //Your Code
}

Please find below link.
Check Link: Implement Background Worker

Edited by riteshbest because: n/a
Member Avatar for Fortinbra

We have something similar in place where the event handler updates a database, and we have an AJAX enabled data grid that is updating on a 5sec timer from that database. But in this case I have nothing in the database I can update.

I'm not sure how to display the progress to the user, when everything is happening server side, and the server can't say anything to the client except as a response.

Member Avatar for Fortinbra

From another source, it was recommended that we create an entire separate process, outside the website, that shares a database with the site, and does all the long processing. That way it can be offloaded to a separate server all together.

Member Avatar for riteshbest

From another source, it was recommended that we create an entire separate process, outside the website, that shares a database with the site, and does all the long processing. That way it can be offloaded to a separate server all together.

Yes you can do this with a windows service.
Basically the idea is..

•You submit a request for processing the doc/pdf file in database table with some status as not started.
•Then your windows service picks up the request from database table which are not started and update them as in progress status.
•Once the processing is complete succesfully /unsuccesfuly your service updated the database table with status as Completed / Failed.

but you will have problems with that:
•what if the sql fails? should there be any response to the client
•if it fails, how do you ensure the file on a later request

Background processing is good way to keep track of current status. You can also, cancel operation if needed and show status as operation aborted.
You can show progress using status label.
Check: http://forums.asp.net/t/1266633.aspx/1

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.