Member Avatar for rantnna

I have two files large files:
1) file1.txt as below: ( more than 20,000 records )

5020120475      |13ZJ38000813                  |20141204|R
5020127431      |13ZJ38001569                  |20141201|R
5020127689      |12ZJ44000606                  |20141203|R
5020127728      |13ZJ38001356                  |20141203|R
5020127956      |13ZJ62002544                  |20141205|R
5020127972      |13ZJ49000082                  |20141205|R
5020128325      |13ZJ57000785                  |20141210|R
5020128706      |13ZJ38002805                  |20141211|R
5020129084      |10XJ70107764                  |20141217|R
5020129102      |12ZJ54000041                  |20141217|R

and

2) other file as file2.txt as below: ( more than 2,00,000 records )

09Y903010552
12ZJ54000041
11XJ62118385
08Y909018946
09Y902011954
11XJ57120346
10XJ70107764
11XJ40165329
09XJ42008336
08Y912021435
11XJ51040272
07Y910027235

and I need expexted output as below:

5020129084      |10XJ70107764                  |20141217|R
5020129102      |12ZJ54000041                  |20141217|R

NOTE: it will compare first file of 2nd column and second file of ist column and print the compare records of ist file as above.

Many thanks for help in Perl or unix awk.

  • 3 Contributors
  • 2 Replies
  • 398 Views
  • 18 Hours Discussion Span
  • Latest Post Latest Post by 2teez

Dani AI

Generated

This thread is asking to print lines from file1 whose second (pipe-delimited) column matches any whole line in file2. As suggested, an in-memory hash is the simplest approach. A few practical gotchas to note: the second column in file1 is padded with spaces (trim it), file2 lines may have CRLF or extra whitespace (strip those), and decide whether output should follow file1 order or file2 order (that determines which file is loaded into memory). ’s point about making an effort stands — the snippets below are minimal, ready-to-run examples to try and adapt.

To preserve file1 order (load file2 keys, then scan file1):

awk 'NR==FNR{gsub(/\r/,""); gsub(/^[ \t]+|[ \t]+$/,"",$0); a[$0]=1; next}
     {split($0,f,"|"); key=f[2]; if(key) gsub(/^[ \t]+|[ \t]+$/,"",key);
      if(key in a) print $0}' file2.txt file1.txt

To minimize memory (load file1 into a hash and scan file2; output follows file2 order):

awk 'NR==FNR{split($0,f,"|"); key=f[2]; gsub(/^[ \t]+|[ \t]+$/,"",key); h[key]=$0; next}
     {gsub(/\r/,""); gsub(/^[ \t]+|[ \t]+$/,"",$0); if($0 in h) print h[$0]}' file1.txt file2.txt

A short, clear Perl version (loads file2 then prints matching file1 rows):

#!/usr/bin/perl
use strict; use warnings;
open my $fh2,'<','file2.txt' or die $!;
my %k; while(<$fh2>){ chomp; s/^\s+|\s+$//g; $k{$_}=1 }
open my $fh1,'<','file1.txt' or die $!;
while(<$fh1>){ chomp; my @f = split(/\|/); next unless defined $f[1]; $f[1]=~s/^\s+|\s+$//g; print "$_\n" if $k{$f[1]} }

Troubleshooting tips: run dos2unix if CRLFs are present; handle case-insensitive matches with tolower() (awk) or lc (Perl) on both keys; if files grow into millions of rows, prefer disk-based methods (create key|line files, sort, then use join) or import into a lightweight DB for best scaling.

Recommended Answers

All 2 Replies

Member Avatar for rubberman

Sorry, but we don't do your work for you. Make an effort to code this in the language of your preference and then we can help.

Member Avatar for 2teez

Hi rantnna,

It is quite easy to compare two or more files in Perl. In fact, if you have taken your time to filter through theis forum, you will disccover that that has been done times and again.

However, since you have several data you might consider using database, not that Perl can't handle it but I think I will be a lot better especially if you don't have so much RAM to play with.

That said, what you can do is this:

open the first file and read in your data row after row into an hash using the second column has your key.

Then open the second file and step through it, compare each key in your hash with each row of the second file. If there is a match then print out the hash values for which the key matched.

So, "we" expect to see your code since you now have a prototype in words what you need to do. Of course, you can do it!
cheers!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.