Member Avatar for Bavananth

Hello experts ! I'm a newbiee to this shell scripting! I try to write a "diff" kinda script which compares two text files and echos the differences. The only different is, my script doesn't care about the character "x" in the first file , being replaced with any other character in the second file. I wrote a script which was perfect(logically correct) with smaller files, takes a hell a lot of time to read 500kb files which I 'm supposed to play with for my work.

Those files contain all digital logic outputs dumped by an IP tool ( zeros(0) ,ones(1) and Don't care terms(x)). I hope now you may understand the reason why I'm trying to do this. Any help?

My code:

#!/bin/bash
dumpraw=bavan1                                      //file1
dumpafterpack=bavan2                              //file2
index1=0
index2=0
cat $dumpraw > $dumpraw.tmp                 // just make temp file of file 1
cat $dumpafterpack > $dumpafterpack.tmp  // temp file of file2

while read line1 ; do              
   MYARRAY1[$index1]="$line1"
    index1=$(($index1+1))
done < $dumpraw.tmp
rm -rf $dumpraw.tmp

while read line2 ; do
   MYARRAY2[$index2]="$line2"
    index2=$(($index2+1))
done < $dumpafterpack.tmp
rm -rf $dumpafterpack.tmp

v3=0
v3=`wc -c bavan1 | cut -d' ' -f1`
v4=`wc -c bavan2 | cut -d' ' -f1`

i=1
if [ $v3 -ne $v4 ] ;  then 
   echo "Error: You are comparing two files which dont have equal number of characters"
   exit 1
else
   while [ $i -lt $v3 ];
       do
		a1=`echo ${MYARRAY1[*]} | cut -c$i`  //this line runs forever,why?
        	a2=`echo ${MYARRAY2[*]} | cut -c$i` //this line runs forever,why?
       			if [  $a1 != $a2 ]
             			then if [ $a1 != "x" ]
                  			then 
                      			echo " $a1 : $a2 "
                  	             fi
       		        fi
       			 
       		let i++
     done
fi
  • 1 Contributor
  • 1 Reply
  • 127 Views
  • 6 Hours Discussion Span
  • Latest Post Latest Post by Bavananth

Dani AI

Generated

’s code is logically fine for small files but very slow on larger dumps because it repeatedly rebuilds and scans a huge string and spawns external programs inside the per-character loop. Concatenating all lines and calling cut/echo on every character forces the shell to do O(n^2) work and many process launches. The symptom looks like “hang” but is just extreme inefficiency.

A much faster approach is to use awk (a single C program, no per-character external calls) and compare corresponding lines character-by-character while ignoring an x in the first file:

awk '
FNR==NR { a[FNR] = $0; next }
{
  s1 = a[FNR]; s2 = $0
  max = (length(s1) > length(s2) ? length(s1) : length(s2))
  for (i=1; i<=max; i++) {
    c1 = substr(s1, i, 1)
    c2 = substr(s2, i, 1)
    if (c1 != c2 && c1 != "x")
      printf "line %d col %d: %s -> %s\n", FNR, i, c1, c2
  }
}' file1 file2

If the dumps are intended to be compared as one long byte stream rather than line-by-line, a small Perl slurp-and-compare is efficient and simple:

perl -0777 -e '
  my ($f1,$f2) = @ARGV;
  open my $A, "<", $f1 or die $!;
  open my $B, "<", $f2 or die $!;
  binmode $A; binmode $B;
  local $/;
  my $s1 = <$A>; my $s2 = <$B>;
  die "length mismatch\n" if length($s1) != length($s2);
  for (my $i=0; $i<length($s1); $i++) {
    my $c1 = substr($s1,$i,1); my $c2 = substr($s2,$i,1);
    print ($i+1) . ": $c1 -> $c2\n" if $c1 ne $c2 && $c1 ne "x";
  }
' file1 file2

Quick troubleshooting tips: measure with time, check for CRLF (use dos2unix), verify byte counts with wc -c, and avoid spawning external commands inside tight loops. For 500KB dumps the awk/perl solutions will finish near-instantly; buffer overflow is not the issue—inefficient looping and command spawning is.

Member Avatar for Bavananth

I guess there may be a buffer overflow or something? Anybody have a simple code block?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.