Member Avatar for Charles_25

My script merges 18 files and returns all numbers that occur >=13 times within the merger. I timed my script and array_count_values is so slow it accounts for 80% of the 2.35 sec time. The files are large, 200,000 numbers per file, so the merged array is well over 2 million.

Any ideas how I can kick out the array_count_values function or write it in a better way and still get a return of all numbers that occur >= 13 times in the merged array?

Note: I shortened code to reflect only 3 files out of 18 to be merged.

for($b=0; $b<1; $b++)
{
    echo $b."\n";
for($a=0; $a<10; $a++)
{

    for($i=0; $i<30; $i++)//30
{
    $linespreset=file_get_contents("/users/history/".$folder."/".$round."/masterspeedrandom_randompick_less13_".$b."_".$a."_".$i.".txt");

    $holdpreset=explode(" ",$linespreset);
    $holdpreset=array_map("trim", $holdpreset);
$print1=file_get_contents('/users/'.$a.'/masterspeed_round3_xxx_'.$holdpreset[0].'.txt');
$print2=file_get_contents('/users/'.$a.'/masterspeed_round3_xxx_'.$holdpreset[1].'.txt');
$print3=file_get_contents('/users/'.$a.'/masterspeed_round3_xxx_'.$holdpreset[2].'.txt');

$healthy = " ";
$yummy   = "_";
$print1= strtr($print1,$healthy,$yummy);
$print2= strtr($print2,$healthy,$yummy);
$print3= strtr($print3,$healthy,$yummy);

$resultround=$print1."\r\n".$print2."\r\n".$print3."\r\n".$print4."\r\n".$print5."\r\n".$print6."\r\n".$print7."\r\n".$print8."\r\n".$print9."\r\n".$print10."\r\n".$print11."\r\n".$print12."\r\n". $print13."\r\n".$print14."\r\n".$print15."\r\n".$print16."\r\n".$print17."\r\n".$print18;

$somearray = str_word_count($resultround, 1, '1234567890:@&_');

$frequency = array_count_values($somearray);

$result = array_filter($frequency, function ($x) { return $x >=13; });

//fwrite to print out $result array with numbers that occur >=13 times in the merged array

unset($somearray);

}//END OF I
}//END OF A

}//END OF B
  • 4 Contributors
  • 3 Replies
  • 671 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by benanamen

Dani AI

Generated

A much faster pattern is to avoid building a huge merged array and then calling array_count_values. Stream each file and increment counts on the fly; keep a small associative counter for candidates and move a token to a "qualified" set as soon as it reaches the threshold (then unset the counter to save memory). This eliminates the large intermediate array and the extra pass that array_count_values does.

$threshold = 13;
$qualified = [];
$counters = [];

foreach ($files as $file) {
    $fh = fopen($file, 'r');
    while (($line = fgets($fh)) !== false) {
        $tok = strtok($line, " \t\r\n"); // adjust delimiters as needed
        while ($tok !== false) {
            if (!isset($qualified[$tok])) {
                if (isset($counters[$tok])) {
                    if (++$counters[$tok] >= $threshold) {
                        $qualified[$tok] = true;
                        unset($counters[$tok]); // free memory: only membership required
                    }
                } else {
                    $counters[$tok] = 1;
                }
            }
            $tok = strtok(" \t\r\n");
        }
    }
    fclose($fh);
}

$results = array_keys($qualified);

Notes and tradeoffs: this is usually far quicker than creating a 2M-element array then calling array_count_values because memory allocation and copying are the real bottlenecks. As noted, faster storage and RAM help I/O, but algorithmic changes give the biggest win here. As suggested, SplFixedArray can help for dense numeric indexed arrays, but it is not a win for associative frequency counting.

Other options: if installing native tools is acceptable, an OS pipeline (sort | uniq -c) often outperforms PHP for raw counting; for extreme scale, use a lightweight DB (SQLite) or Redis to increment counters. Always profile with microtime(true) and memory_get_usage() and verify tokenization matches the actual data format before committing to a solution.

Recommended Answers

All 3 Replies

Member Avatar for rproffitt

So 2.35 seconds? Is this on a SSD or HDD?
I've found payback to be good enough that a move to SSD and more RAM is worth it. I don't see anything outstanding in the code that would make a big difference.

Member Avatar for cereal

Hi,

in addition, have you tried with SplFixedArray? It should be faster than standard arrays. Also if you want to open files from the script, than use fopen() instead of file_get_contents(), because the latter will load the entire file in memory before starting processing, while the former will read in chunks and start the execution immediately.

See: http://php.net/manual/en/class.splfixedarray.php

Member Avatar for benanamen

I believe this question may be part of this thread Click Here

commented: indeed +15
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.