Member Avatar for Aeonix

If user issues a query, I would like AJAX to download the result of query into browser so that JavaScript can do the parsing. The results could be around 1MB-5MB (at most, doubt it would get anywhere near 0.1MB), and I would send it from PHP to front-end using JSON, then convert it into array, which I would then parse depending on users expectations. Given that 57KB is what my asset is worth (project not finished), DaniWeb is for example 1.232kB (as my browser receives it). Would containing 5MB of data in a single array-variable hurt the performance of a computer browser? I'm not talking about download, I'm talking after it's done. Will containing 5MB and making data[2198124]["text"] hurt a lot?

Edited by Aeonix
  • 4 Contributors
  • 5 Replies
  • 322 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by ryantroop

Dani AI

Generated

Short answer for : 5MB of JSON over the wire is often fine for modern desktop browsers, but it can become a problem once parsed and rendered. was right to call out parsing/iteration cost, and 's experience highlights that transport or server limits can also bite when a single response gets very large.

Why this matters: the raw JSON size is not the whole story. After parsing the text into JS objects you typically see a several‑times memory inflation (strings and object/property overhead), and creating many DOM nodes (1000s–100k+) is what produces the worst slowdowns and layout/paint stalls. Network compression hides that risk, and low‑end devices or older browsers will feel it first.

Practical, immediate fixes you can apply:

  • Return only the fields the UI needs (server projection) and cap results (pagination or cursor).
  • Stream results incrementally (NDJSON/newline JSON) and process records as they arrive to avoid holding the whole payload.
  • Offload parsing/aggregation to a Web Worker so the UI thread stays responsive.
  • Virtualize rendering: only create DOM nodes for visible rows and recycle elements as the user scrolls.
  • Use gzip/brotli compression; consider a binary format (MessagePack) if transport size is critical.
  • Persist very large result sets to IndexedDB and page through them locally instead of keeping everything in RAM.
  • Measure on target devices with DevTools (network size, heap snapshots, CPU flame charts) to find the real bottleneck.

Small examples (patterns, not a copy of posts above):

// incremental NDJSON consumer (main thread)
fetch('/api/stream')
  .then(res => {
    const reader = res.body.getReader();
    const dec = new TextDecoder();
    let buf = '';
    function pump() {
      return reader.read().then(({done, value}) => {
        if (done) { if (buf.trim()) handle(JSON.parse(buf)); return; }
        buf += dec.decode(value, {stream: true});
        const lines = buf.split('\n');
        buf = lines.pop();
        for (const line of lines) if (line.trim()) handle(JSON.parse(line));
        return pump();
      });
    }
    return pump();
  });
// worker.js (parse/aggregate in background)
self.onmessage = e => {
  const chunk = e.data; // e.g. NDJSON text or array of records
  const processed = parseAndSummarize(chunk); // keep CPU off the UI thread
  self.postMessage(processed);
};

If the UI only needs a small window (e.g., 50–200 rows), prefer server-side limits + client virtualization; stream/worker techniques are for when the client truly needs large volumes.

Recommended Answers

All 5 Replies

Member Avatar for ryantroop

On a "modern" machine, probably not. You will have a longer delay simply downloading the data.

However, if this is the path you plan on taking, I would let PHP format it into the array so that the browser just has to JSON.parse() the value and then you don't have to worry about "processing" (so.. pretty much everything in JS is a string. If you send a raw string of whatever along to script, it will have to take that string and basically "eval" it to figure out what is inside said string. You would then turn it into an "array" which means JS then has to iterate through the string again to break everything apart. Not such a good idea...).

That said, 5MB in an array may be a bit demanding, and actually stall processing a bit (if you do a straight up for (;;) loop, and you have 100000 results, you will likely be sitting for a while to process all of that (seconds, if not minutes). It also depends on what the compare values are, if they are strings, etc... or what actions you are taking with the data as you iterate over your array.

Really, the only way to "know" is to make an HTML page that has your 5MB of data on it already formatted as you need it, and start processing it in script to check your timings. It likely will not be "usable" without some sort of spinny wheel of death to show processing.

So... what are you trying to accomplish? And why all that data? Why must it be that much?

Edited by ryantroop
Member Avatar for Aeonix

I would let PHP format it into the array so that the browser just has to JSON.parse() the value and then you don't have to worry about "processing"

It will be straight JSON, that will be JSON.parse()d, I won't eval anything, they're ready to use values, strings and integers.

if you do a straight up for (;;) loop, and you have 100000 results, you will likely be sitting for a while to process all of that (seconds, if not minutes)

What would happen, if I had this entire 100000 entries, and I would for(;;) 100 results? Would that hurt that much?

if they are strings, etc... or what actions you are taking with the data as you iterate over your array.

Just display, document.getElementById("result" + x).innerHTML = oneofresults[x];.

Really, the only way to "know" is to make an HTML page that has your 5MB of data on it already formatted as you need it

This will take A LONG time... also, why HTML page? The result comes from SQL databased, that is parsed into usable array in PHP, then json_encode(), passes through AJAX to front-end JavaScript, the JSON.parse() into a variable, and use it. Where does HTML come from?

So... what are you trying to accomplish? And why all that data? Why must it be that much?

Knowing my skill to explain things, it will only take 5 years to get my point over. So let's just skip why and how :D

P.S: Oh, I didn't explain anything... yea... user makes request into database and I want it to be parsed so that users access is faster, than parsing time of AJAX/PHP and SQL.

Edited by Aeonix
Member Avatar for jayashreemarg

It is of course too much for the browser. In that case you can go ahead with readucing the queries so that downloading can be a bit easier.

Edited by jayashreemarg because: more information
Member Avatar for Member #905211

I work on something that recieves a lot of data and displays it in a grid. We are able to recieve about 70MB worth of data without issue. After 70MB, the webapi calls start to choke, but not because javascript can't handle it. There seems to be a limit as to how much data can come through a single webapi call. So we ended up chunking the call to be able to bring in over 70MB worth of data. Basically, you should have no problem with 1-5MB of data coming from a WebAPI/webservice/ajax/whatever call. It's now up to the speed of the connection from the client machine and the server.

Member Avatar for ryantroop

For the record, my suggestion was to make a stub html page that has a single script element that has your formatted data on the page already. Not really a whole lot of work there

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.