Search a file with multi-element contents of another file

Question

perly 0 Newbie Poster

14 Years Ago

Hi,

I need help to make a perl program work. I have two files - file 1 and file 2. The contents of File2 is to be searched with the contents of file 1.

File2: 2 tab-delimited columns

XM:1120002  complex-solution 
MM:0999111  blue-green solution
UX:1020022  activity unknown, (simple/complex?)

File1:(one column of space separated strings)

XM:1120002 MM:0999111 UX:1020022 
UX:1020022 XM:1120002
XM:1120002

Output required: 2 tab-delimited columns
Date:
The following were found in file2:

XM:1120002 MM:0999111 UX:1020022    complex-solution;blue-green solution;activity unknown, (simple/complex?)  
UX:1020022 XM:1120002           complex-solution;blue-green solution;activity unknown
XM:1120002              complex-solution

The code is as follows:

use warnings;
use strict;

my $time = scalar localtime(); # get date of search

my $heading = qq(Date of search: $time\nThe Following Matches were Found in File 2:\n);

my %hash1;
my %hash2;

my $file2 = 'file2.txt'; # file 2: file with two columns to be searched with contents of file1.
open my $fh, '<', $file2 or die "can't open $file2:$!";
while (<$fh>) {
    chomp;
    my ($ID, $value) = split /\s+/, $_;
    $hash1{$ID} = $value;
}
 close $fh;

my $file1 = 'file1.txt'; # file1: one column of probe names for searching the second file.

open my $fh_new, '>', 'ouput.txt' or die "can't open file:$!"; # Open output file for writing

my $i=(); 
open $fh, '<', $file1 or die "can't open this file:$!";  #Open query file for reading
print $fh_new $heading;
while ( defined( my $line = <$fh> ) ) {
    chomp $line;
    my ($checked_word[$i]) = split /\s+/,$line;
    $hash2{$checked_word[$i]} = $checked_word[$i];
    if ( $line ) {
        $checked_word[$i] = $line;
        exists $hash{$checked_word[$i}
        ? print $fh_new $checked_word[$i], "\t", $hash{$checked_word}[$i], $/
        : print $fh_new $checked_word, "\t", "######", $/;
    }
}
close $fh or die "can't close file:$!";
close $fh_new or die "can't close file:$!";

I will appreciate any help! Thanks

perl

2 Contributors
10 Replies
302 Views
1 Day Discussion Span
Latest Post 14 Years Ago Latest Post by perly

Dani AI

Generated 9 Months Ago

Short, practical approach (what to do and why)
Read file2 once and build a lookup that maps each ID to its full description. Then process file1 line by line: split the line into IDs, look up each ID in the map, collect the descriptions in the same order, join them with a semicolon, and write one output line of the form "IDs<TAB>joined-descriptions". That pattern is what outlined and it gives O(file2 + total_tokens_in_file1) performance, so it scales to tens of thousands of rows.

Clarifications of the confusing bits

When parsing file2 you must split into only two fields: the first token (the ID) and a second field that contains the rest of the line. That preserves multi-word descriptions instead of chopping them at the first space.
The q{} idiom is simply an empty single-quoted string literal (equivalent to ''). It’s just a compact way to make an empty string.
The arrow form used to get a value from the lookup is dereferencing a hash reference. The script created a hashref earlier; using the arrow accesses a value by key. If you prefer simplicity you can use an ordinary hash instead and index it directly.

Why truncation or “hangs” happen and how to fix them

Truncation usually means the description was split incorrectly (no “two-field” split) or you are viewing the output with an editor that has word-wrap turned on. Check the raw file with a pager like less -S or inspect bytes to confirm.
Slowness or a program that appears to hang is commonly caused by repeatedly opening the output file inside the per-line loop or by appending to one global string that grows without bound. Open the output once, build a small per-line buffer, print it, then discard it. Avoid nested scans of file2; use the lookup.

Small extra tips
Check for missing IDs with a defined/exists test and emit a placeholder when absent. Use join/map to build the joined description (faster and clearer than concatenating in a loop). For tab-separated inputs prefer splitting on tab if fields can contain extra spaces, or use a text-parsing module for robust handling. Note that eventually fixed the visible truncation by turning off editor word-wrap and using a clean copy of ’s script.

All 10 Replies

2teez 43 Posting Whiz

14 Years Ago

Hi perly,
Yeap, just like want you have done, but with a little modification.
The code below should solve your problem nicely. I hope this help.

#!/usr/bin/perl
use warnings;
use strict;
use Carp qw(croak);

croak "usage: <perl_script> <file1> <file2>" unless @ARGV;

my $time = scalar localtime();    # get date of search

print
  qq(Date of search: $time\nThe Following Matches were Found in  File 2:\n\n);

my ( $file1, $file2 ) = @ARGV;
my $file_data = {};

open my $fh, '<', $file2 or croak "can't open file:$!";
while ( defined( my $line = <$fh> ) ) {
    my ( $rec_name, $rec_description ) = split /\s+/, $line, 2;
    $file_data->{$rec_name} = $rec_description;
}
close $fh or croak "can't close file:$!";

open $fh, '<', $file1 or croak "can't open file:$!";
while ( defined( my $line = <$fh> ) ) {
    chomp $line;
    my $msg = "";
    foreach my $data ( split /\s+/, $line ) {
        $msg .= sprintf "%s", $file_data->{$data};
    }
    print $line, ' ', $msg;
}
close $fh or croak "can't close file:$!";

Files file1 and file2 has to be indicated on the Command Line Interface like so:

perl_script.pl file1.txt file2.txt

Else, this code wouldn't work and you get a message on how the script should be used.

Edited 14 Years Ago by 2teez

2teez 43 Posting Whiz

14 Years Ago

Hello Perly again,
The previous code I showed solved the problem, but I think is kind of noisy. For example, why should I use open function twice, only to end up doing almost the same thing? So, I came up with another one.
Put your open function in a subroutine, call it twice and use CODEREFs achieve what you want on the files. Using both file and CODEREFs as your subroutine parameters. Bingo! More organised and less noisy!
To run this just type the name of your perl script on the CLI, you don't need to specify file1.txt and file2.txt.

#!/usr/bin/perl
use warnings;
use strict;
use Carp qw(croak);

my $time = scalar localtime();    # get date of search
print
  qq(Date of search: $time\nThe Following Matches were Found in  File 2:\n\n);

my $file_data = {};               # a hash ref. to use

# read_file subroutine to read in the file
# and a subroutine ref to work on the file

read_file( 'file2.txt', \&file_handler2 );
read_file( 'file1.txt', \&file_handler1 );

sub read_file {
    my ( $filename, $code_ref ) = @_;

    open my $fh, '<', $filename or croak "can't open file: $0 - $!";
    while (<$fh>) {
        s/^\s+|\s+$//;
        $code_ref->($_);
    }
    return;
}

sub file_handler1 {
    my ($line) = @_;
    my $msg = q{};
    foreach my $data ( split /\s+/, $line ) {
        $msg .= sprintf "%s;", $file_data->{$data};
    }
    $msg =~ s/;$//;
    print $line, ' ', $msg, $/;
}

sub file_handler2 {
    my ( $rec_name, $rec_description ) = split /\s+/, $_[0], 2;
    return $file_data->{$rec_name} = $rec_description;
}

OUTPUT:

Date of search: Sat Jul  7 19:51:02 2012
The Following Matches were Found in  File 2:

XM:1120002 MM:0999111 UX:1020022 complex-solution;blue-green solution;activity unknown, (simple/complex?)
UX:1020022 XM:1120002 activity unknown, (simple/complex?);complex-solution
XM:1120002 complex-solution

I hope this helps

Edited 14 Years Ago by 2teez

2teez 43 Posting Whiz

14 Years Ago

HI Perly,

> Thanks for the two codes. They both worked. For some reason,
> my output was like this:
> XM:1120002 MM:0999111 UX:1020022 complex-solution;blue-green solution;ac
> tivity unknown, (simple/complex?)
> UX:1020022 XM:1120002 activity unknown, (simple/complex?);complex-solution
> XM:1120002 complex-solution

The two codes would work because the second code is just a refined of the first code.

The output you got was what you specified as what you wanted in the orginial post.
If you need to seperate each column with  tab space or write it out to another file, all you need do is to modify the subroutine called **file_handler1**, like so:



    sub file_handler1 {
        my ($line) = @_;
        open my $fh, '>', 'output.txt' or croak "can't open file: $0 - $!";
        print {$fh} $heading, $/;
        $msg .= sprintf "\n%s\t", $line;
        foreach my $data ( split /\s+/, $line ) {
            $msg .= sprintf "\t%s", $file_data->{$data};
        }
        print {$fh} $msg, $/;
    }


Please, note that the variable $msg is now been used as a global variable, i.e not declared within the subroutine, and also new variable $heading is now been used. Please, check the complete script below. 

> Kindly explain the following code lines:
> 1. split /\s+/, $_[0], 2
> 2. my $msg = q{};
> 3. $file_data->{$rec_name}


1. You know the "split" function am sure. If not, then you have to do this:    


    perldoc -f split    


 What maybe confusing will be the $_[0] and the number '2'
   I used the $_[0] to get the first element of the parameter passed to the subroutine, since, parameter are passed to a subroutine or a function as a flat list in perl.
   So, $_[0], could also have been written as:
   a.) shift @_; OR my $line = shift;
   b.) my ($line) = @_;
The number '2', represent LIMIT in the "split" function. Which defined the maximum number of field an expression would be split into.

2. my $msg = q{}. Is just the same way as saying '' i.e a single quote, just as qq, is a double quote.   


    perldoc perlop 


for more info.

3 Instead of using an hash, which of course would work, we used an hash reference, to get, both the name and description of which line in file2. Then we return the result.

> The initial confusion I had was how to search file2 with the multiple line elements in File1. I will be happy for some explanation.

For me, file2 is the key to handlering file1. How?
Since, file2 has all the names contained in file1, tied to a particular description. Then, we use an hash or hash ref., to get a key and value for file2. Then use that to sort out file1. That is we say foreach line of file1, we split into an array and use each element of the array as a key in our generated hash variable, then print the result.

The script below seperated each file with tab and printed output in a file called **output.txt**




    #!/usr/bin/perl
    use warnings;
    use strict;
    use Carp qw(croak);

    my $time = scalar localtime();    # get date of search
    my $heading =
      qq(Date of search: $time\nThe Following Matches were Found in  File 2:\n);

    my $file_data = {};               # a hash ref. to use
    my $msg       = qq{   };

    # read_file subroutine to read in the file
    # and a subroutine ref to work on the file

    read_file( 'file2.txt', \&file_handler2 );
    read_file( 'file1.txt', \&file_handler1 );

    sub read_file {
        my ( $filename, $code_ref ) = @_;

        open my $fh, '<', $filename or croak "can't open file: $0 - $!";
        while (<$fh>) {
            s/^\s+|\s+$//;
            $code_ref->($_);
        }
        return;
    }

    sub file_handler1 {
        my ($line) = @_;
        open my $fh, '>', 'output.txt' or croak "can't open file: $0 - $!";
        print {$fh} $heading, $/;
        $msg .= sprintf "\n%s\t", $line;
        foreach my $data ( split /\s+/, $line ) {
            $msg .= sprintf "\t%s", $file_data->{$data};
        }
        print {$fh} $msg, $/;
    }

    sub file_handler2 {
        my ( $rec_name, $rec_description ) = split /\s+/, $_[0], 2;
        return $file_data->{$rec_name} = $rec_description;
    }




I hope this helps.
Please, mark this as solved if satisfied.

Edited 14 Years Ago by 2teez

Reply to this topic

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.

perly 0 Newbie Poster · Answer 1 · 2012-07-07T20:28:04+00:00

Hi 2teez,

Thanks for the two codes. They both worked. For some reason,

 my output was like this:



    XM:1120002 MM:0999111 UX:1020022 complex-solution;blue-green solution;ac
    tivity unknown, (simple/complex?)
    UX:1020022 XM:1120002 activity unknown, (simple/complex?);complex-solution
    XM:1120002 complex-solution

I also tried to write the output into a file unsuccessfully, and also tried to have the columns separated by a tab. The amended code is as folows:



    use warnings;
    use strict;
    use Carp qw(croak);

    my $time = scalar localtime(); # get date of search
    print
    qq(Date of search: $time\nThe Following Matches were Found in File 2:\n\n);

    my $file_data = {}; # a hash ref. to use

    # read_file subroutine to read in the file
    # and a subroutine ref to work on the file
    read_file( 'file2.txt', \&file_handler2 );
    read_file( 'file1.txt', \&file_handler1 );
    sub read_file {
    my ( $filename, $code_ref ) = @_;
    open my $fh, '<', $filename or croak "can't open file: $0 - $!";
           while (<$fh>) {
                s/^\s+|\s+$//;
                $code_ref->($_);
            }
            return;
    }

    #open my $fh_new, '>', 'ouput.txt' or die "can't open file:$!"; 

    sub file_handler1 {
            my ($line) = @_;
            my $msg = q{};
            foreach my $data ( split /\s+/, $line ) {
                $msg .= sprintf "%s;", $file_data->{$data};
                }
                $msg =~ s/;$//;
                print $line, ' ',"\t", $msg, "\n", $/;
    }
    #print my $fh_new $line, ' ',"\t", $msg, "\n", $/;

    sub file_handler2 {
                my ( $rec_name, $rec_description ) = split /\s+/, $_[0], 2;
                return $file_data->{$rec_name} = $rec_description;
    }
    #close $fh_new or croak "can't close file:$!";


Kindly explain the following code lines:

1. split /\s+/, $_[0], 2

2. my $msg = q{};

3. $file_data->{$rec_name}


The initial confusion I had was how to search file2 with the multiple line elements in File1. I will be happy for some explanation.

Thank you!

perly 0 Newbie Poster · Answer 2 · 2012-07-08T01:26:26+00:00

Hi 2teez,

Thanks for the two codes. They both worked. For some reason,

 my output was like this:



    XM:1120002 MM:0999111 UX:1020022 complex-solution;blue-green solution;ac
    tivity unknown, (simple/complex?)
    UX:1020022 XM:1120002 activity unknown, (simple/complex?);complex-solution
    XM:1120002 complex-solution

I also tried to write the output into a file unsuccessfully, and also tried to have the columns separated by a tab. The amended code is as folows:



    use warnings;
    use strict;
    use Carp qw(croak);

    my $time = scalar localtime(); # get date of search
    print
    qq(Date of search: $time\nThe Following Matches were Found in File 2:\n\n);

    my $file_data = {}; # a hash ref. to use

    # read_file subroutine to read in the file
    # and a subroutine ref to work on the file
    read_file( 'file2.txt', \&file_handler2 );
    read_file( 'file1.txt', \&file_handler1 );
    sub read_file {
    my ( $filename, $code_ref ) = @_;
    open my $fh, '<', $filename or croak "can't open file: $0 - $!";
           while (<$fh>) {
                s/^\s+|\s+$//;
                $code_ref->($_);
            }
            return;
    }

    #open my $fh_new, '>', 'ouput.txt' or die "can't open file:$!"; 

    sub file_handler1 {
            my ($line) = @_;
            my $msg = q{};
            foreach my $data ( split /\s+/, $line ) {
                $msg .= sprintf "%s;", $file_data->{$data};
                }
                $msg =~ s/;$//;
                print $line, ' ',"\t", $msg, "\n", $/;
    }
    #print my $fh_new $line, ' ',"\t", $msg, "\n", $/;

    sub file_handler2 {
                my ( $rec_name, $rec_description ) = split /\s+/, $_[0], 2;
                return $file_data->{$rec_name} = $rec_description;
    }
    #close $fh_new or croak "can't close file:$!";


Kindly explain the following code lines:

1. split /\s+/, $_[0], 2

2. my $msg = q{};

3. $file_data->{$rec_name}


The initial confusion I had was how to search file2 with the multiple line elements in File1. I will be happy for some explanation.

Thank you!

perly 0 Newbie Poster · Answer 3 · 2012-07-08T01:34:26+00:00

Hi 2teez,

I really appreciate your explanations and the new code. The code works with the correct output on small and medium-sized files, but it hangs with large files (file2 ~ 35000 rows and file1 ~ 7000rows). I'm sorry I forgot to mention that the strings in file1 maybe more than 3 per line, sometimes up to 23.

2teez 43 Posting Whiz · Answer 4 · 2012-07-08T06:07:35+00:00

HI Perly,

The code works with the correct output on small and medium-sized files, but it hangs with large files (file2 ~ 35000 rows and file1 ~ 7000rows)

Which of the codes; the first, second or modified second code? And when you say hangs, is it the execution of the code, or openning of the output file generated?

I should also mention that you could have the same desired result with code number 1 and 2, by doing this from the CLI:

    perl_script.pl file1.txt file2.txt > output.txt    # code 1

    perl_script.pl > output.txt   # code 2

Please, also include a

    return;

just after

     print {$fh} $msg, $/;

in file_handler1 subroutine

Hope this helps

perly 0 Newbie Poster · Answer 5 · 2012-07-08T18:26:56+00:00

That helped a lot. Thanks. I got an output in a file. Unfortunately, there is still one problem. The output is truncated. For example I get something like:

XM:1120002 MM:0999111 UX:1020022    complex-solution    blue-green  activity

instead of :

XM:1120002 MM:0999111 UX:1020022    complex-solution    blue-green solution activity unknown, (simple/complex?)

The program seems to output the first string only for every search string.

2teez 43 Posting Whiz · Answer 6 · 2012-07-08T19:17:24+00:00

Hi Perly,
And you are sure your editor don't have it's word wrap enabled?
Test with the raw data you gave in the original post and see, if you are not getting the correct output, then you have alter the script somewhere, else you should have something like so:

XM:1120002 MM:0999111 UX:1020022 complex-solution blue-green solution activity unknown, (simple/complex?)

perly 0 Newbie Poster · Answer 7 · 2012-07-08T20:56:13+00:00

Hi 2teez,

You're right!!! I think I inadvertently altered something; I turned off wordwrap and I downloaded a fresh modidfied code 2 (added return;) and it worked perfectly. Thanks for your patience and help.

Search a file with multi-element contents of another file

Dani AI

Recommended Answers Collapse Answers

All 10 Replies

Recommended Answers