count characters in a string

[mank]

This code counts number of characters in a single line
how can I make it count(give total X Y Z) it in multiple lines seperated by ">"

while(<IN>) {
  chomp;
  my @line = split(//, $_);  
  if($line[0] eq ">"){print @line;}
          elsif (($line[0] eq "X"||"Y"||"Z")||($line[0] ne ">"))
                        {
                my %cnter;
                         foreach my $gc (@line) 
                             {$cnter{$gc} ++;}

                      print join(" ", @cnter{qw(X Y Z)}), "\n";

                        }


                }

close(IN);
exit;

[katharnakh]

open(IN, "readme.txt") || die "ERROR: $!"; 
my %cnter;

while(<IN>) {
  chomp;
  my @line = split(//, $_); 
  if($line[0] eq ">"){print @line, "\n";}
  elsif (($line[0] eq "X"||"Y"||"Z")||($line[0] ne ">"))
  {
      foreach my $gc (@line) 
      {
      	$cnter{$gc} = $cnter{$gc} ? $cnter{$gc} += 1 : 1;
      }
  }
}
foreach $k(keys %cnter){
	print "$k: $cnter{$k} | ";
}
close(IN);

# readme.txt
> this is first line
X this is second line
Y this is third line
Z this is forth line

# Output
> this is first line
 : 12 | c: 1 | d: 2 | e: 4 | f: 1 | h: 5 | i: 10 | l: 3 | n: 4 | o: 2 | r: 2 | s: 7 | t: 5 | X: 1 | Y: 1 | Z: 1 |

katharnakh.

[mank]

This is what I am looking for 
input file

>abc
XXYYZZ
XXYYZZ
>bcd
XXZZ
XXYY
>cde
ZZYYXX
>def
YYZZ
YYXX

output->

>abc 4 4 4
>bcd 4 2 2
>cde 2 2 2
>def 2 4 2

[katharnakh]

Hi,
since, X, Y and Z are repeating after every line having '>', make line having '>' as initial key in hash, and then add keys(X, Y, Z) and values.

open(IN, "readme.txt") || die "ERROR: $!";
my (%cnter, $marker);
while(<IN>) {
		chomp;
		my @line = split(//, $_);
		if($line[0] eq ">")
		{
				$marker = $_;
				print @line, "\n";
		}
		elsif (($line[0] eq "X"||"Y"||"Z")||($line[0] ne ">"))
		{
		  	foreach my $gc (@line)
				{
		 				$cnter{$marker}{$gc} = $cnter{$marker}{$gc} ? $cnter{$marker}{$gc} += 1 : 1;
		 		}
		}
}

foreach $k(keys %cnter){
		print "\n$k: ";
		foreach $kk(keys %{$cnter{$k}}){
 				print "\t$kk: $cnter{$k}{$kk}";
		}
}
close(IN);

Note: Please dont post your queries to private message, the forum is provided for that only, getting help.

katharnakh

[mank]

Thanks :)

[godevars]

I have been playing with this. I have tried updating it to count words. When I update the split to /\s+/ I receive an error for an unitialized hash value. How would I correct it? I tried by changing the elsif line to not include the X,Y and Z argument.

[KevinADC]

post your code and some sample data.

[godevars]

Below is the code I am working with:

1. use strict;
2.
3. open(FILEIN,'Sample.txt') || die "Can't open OUT file: $!\n";
4. open (FILEOUT, '>SampleOut2.txt') ||die "Can't open OUT file: $!\n";
5.
6. my (%cnter, $marker);
7. while(<FILEIN>) {
8.		chomp;
9.		my @line = split(/\s+/, $_);
10.		if($line[0] eq "=")
11.		{
12.			$marker = $_;
13.			print FILEOUT @line, "\n";
14.		}
15.		elsif ($line[0] ne "=")
16.		{
17.		  	foreach my $gc (@line)
18.				{
19.		 		$cnter{$marker}{$gc} = $cnter{$marker}{$gc} ? $cnter{$marker}{$gc} += 1 : 1;
20.		 		}
21.		}
22. }
23.
24. foreach my $k(keys %cnter){
25.		print FILEOUT "\n$k: ";
26.		foreach my $kk(keys %{$cnter{$k}}){
27. 				print FILEOUT "\t$kk: $cnter{$k}{$kk}\n";
28.		}
29. }
30. close FILEIN;
31. close FILEOUT;

Text for the sample is:

======Retirement Plan Fundamentals Part I

The objective of the Retirement Plan Fundamentals, Parts I and II, is to give an individual
beginning a career as a retirement plan professional a general background in qualified plans as a first step toward meeting the challenges of the profession.

=====Retirement Plan Fundamentals Part III

Retirement Plan Fundamentals Part III (RPF-3) covers plan administration, including census collection, benefit allocations and coverage and nondiscrimination testing. This course emphasizes daily valuation recordkeeping but includes discussions of balance-forward plans and conversions.

I am looking to count the words in each section seperately but reported in one output.

Thanks

[katharnakh]

open(IN, "readme.txt") || die "ERROR: $!";
open(OUT, ">seeme.txt") || die "ERROR: $!";

my (%cnter, $marker);
while(<IN>) {
		chomp;
		my @line = split(/\s+/, $_);
		if($line[0] =~ /=/)
		{
				$marker = $_;
				print @line, "\n";
				#print OUT @line, "\n";
		}
		elsif ($line[0] =! /=/)
		{
		  	foreach my $gc (@line)
				{
		 				$cnter{$marker}{$gc} = $cnter{$marker}{$gc} ? $cnter{$marker}{$gc} += 1 : 1;
		 		}
		}
}

foreach $k(keys %cnter){
		print OUT "\n$k: ";
		foreach $kk(keys %{$cnter{$k}}){
				print OUT "\n$kk: $cnter{$k}{$kk}";
 				#print "\t$kk: $cnter{$k}{$kk}";
		}
}
close(IN);
close(OUT);

You should use a regular expression to check whether line is having a character '=', and everything else is same. Also you need to one more line to inner foreach loop to output to a file.

sample output:
======Retirement Plan Fundamentals Part I:
plans: 1
background: 1
first: 1
individual: 1
...
=====Retirement Plan Fundamentals Part III:
conversions.: 1
plans: 1
course: 1
allocations: 1
...

katharnakh.

[KevinADC]

this might return false matches:

if($line[0] =~ /=/)

it would be better (I think judging by the sample data) to anchor it to the beginning of the string:

if($line[0] =~ /^=/)

at least that way you know it not somewhere else in the string. Might be better to just use the index() function though to avoid the unecessary overhead of a regexp.

This is also not correct syntax:

elsif ($line[0] =! /=/)

should be:

elsif ($line[0] !~ /=/)

[katharnakh]

...

This is also not correct syntax:

elsif ($line[0] =! /=/)

should be:

elsif ($line[0] !~ /=/)

Thanks Kavin...

[KevinADC]

Borrowing from katharnakh's code I corrected a couple of things and expanded on the processing.

use strict;
use warnings;
open(IN, "readme.txt") || die "ERROR: $!";
open(OUT, ">seeme.txt") || die "ERROR: $!";

my (%cnter, $marker);
while(<IN>) {
		chomp;
		my @line = split(/\s+/);
		if($line[0] =~ /^=/) {
            $line[0] =~ tr/=//d; # remove all the "=" from the section title
            $marker = join(' ',@line); # rejoin the section title into a string
		}
		else {
            tr/,.?!//d for @line; #remove some punctuation
            tr/A-Z/a-z/ for @line; #convert all text to lower case so 'Word' and 'word' are the same
			   $cnter{$marker}{$_}++ for @line; 
 		}
}

# sort words by count in descending order
foreach my $section (keys %cnter){
		print OUT "$section\n";
		foreach my $word (sort { $cnter{$section}{$b} <=> $cnter{$section}{$a} } keys %{$cnter{$section}}){
				print OUT "$word: $cnter{$section}{$word}\n";
		}
}
close(IN);
close(OUT);

[godevars]

Thank you. I updated my script and the output looks good. I did start using the 'warnings' and 'strict' commands as suggested to me in another thread. When I run the updated script I receive an 'unitialized value in pattern match' error for this line:

if($line[0] =~ /^=/) {

My output is correct, just not sure why I continue to receive this error. I tried to correct by updating it to read.

if(my $line[0] =~ /^=/) {

Didn't work. Also tried to add @line (as well as $line) to the line where 'my' variables are defined but no luck. Is this an error I would receive but could ignore? Thanks again for helping me understand.

[KevinADC]

it's probably being caused by a blank line in the file. It is also a warning, not an error. Errors terminate scripts, warnings alert you to possible problems but the script keeps running.

You may want to try and skip blank lines in the file:

while(<IN>) {
      <strong>next if (/^\s*$/;</strong>
		chomp;

[godevars]

I didn't realize the blank lines would create warnings. No more warnings now. In reviewing the script, it looks like a hash of hashes were made. I added (after the 'while' section' and before the '#sort..':

my ($key, $value);
while (($key, $value) = each %cnter) {
      print FILEOUT "$key => $value\n";
}

I wanted to see what the values in each hash were (wanted understand the logic behind each portion). My output looked something like this:

Retirement Plan Fundamentals Part I => HASH(0x18f0554)
Retirement Plan Fundamentals Part III => HASH(0x1876a74)
.....

I am looking to compare the words from each section in an easier way to review. For example, output would like more like:word/section Plan Part I : Plan Part II : Plan Part III

plans: 1 : 0 : 1
course: 0 : 1 : 1
.....

I know (keys %cnter) is the title part.
(keys %{cnter{$section}}) is each word in the title hash. Not sure where to start. If I do this:

(keys %{cnter{$section}}) eq (keys %{cnter{$section}})

I am only comparing the words in the section. I believe I need to assign each section a unique hash name so that it would look something like this:

(keys %{cnter1{$section}}) eq (keys %{cnter2{$section}})

I would use one of the sections as the base list so that when I compare other sections, I would just add a word if it's not there already.

Also, please let me know if I should start another thread. I want to make sure I follow proper form here.

Thanks-

[katharnakh]

... I know (keys %cnter) is the title part. (keys %{cnter{$section}}) is each word in the title hash. Not sure where to start. If I do this:

(keys %{cnter{$section}}) eq (keys %{cnter{$section}})

I am only comparing the words in the section. I believe I need to assign each section a unique hash name so that it would look something like this:

(keys %{cnter1{$section}}) eq (keys %{cnter2{$section}})

...

Hi,
Its a good idea to have two or three sections, if you have two or three sections only(not more than that). But if sections in the input file is more than 3(may be 5 or 10) then having unique hash for each sections is not a good idea. So we can change a little bit in the way data is stored in the hash, so that it is easy to handle.

Now the data is stored in the hash is like this,

cnter{
	section1 => {
				word1 => count1,
				word2 => count2
			},
	section2 => {
				word1 => count1,
				word2 => count2
			},
        ...
}

we will alter above hash, so that we will have one hash for entire section,

cnter{
		word1 => [sec1_count, sec2_count, sec3_count....],
		word2 => [sec1_count, sec2_count, sec3_count....],
		...
}

Now only word is stored as KEY and VALUE is an anonymous array holding count of word in each section.

Note: Each index in the array is dedicated to perticular section, so word count should be incremented accordingly.

katharnakh.

[KevinADC]

It seems that the array approach would require you know all the sections beforehand.

It may require more than one process to get the final output. First build the hash that counts the words per section, then have another routine that builds the hash of arrays then prints the final output.

[KevinADC]

I think the approach for the main data structure will have to be:

word => {
      section_titte => count
      section_title => count
}
word => {
      section_titte => count
      section_title => count
}

plus have a seperate array that holds the names of each section in the order it was found in the file.

[KevinADC]

A basic script, the output foramt is not real good but that will be up to you to change to your needs:

use strict;
use warnings;
open(IN, "readme.txt") or die "ERROR: $!";
open(OUT, ">seeme.txt") or die "ERROR: $!";
my (%cnter, $title, @order);
while(<IN>) {
      next if (/^\s*$/);
      chomp;
      my @line = split(/\s+/);
      if($line[0] =~ /^=/) {
            $line[0] =~ tr/=//d; # remove all the "=" from the section title
            $title = "@line";
            push @order, $title;
      }
      else {
            tr/,.?!//d for @line; #remove some punctuation
            tr/A-Z/a-z/ for @line; #convert all text to lower case so 'Word' and 'word' are the same
            $cnter{$_}{$title}++ for @line; 
      }
}

print OUT join("\t",@order),"\n";
foreach my $word (sort keys %cnter){
      print OUT "$word : ";
      my @t = ();
      foreach my $title (@order) {
            push @t, (exists $cnter{$word}{$title}) ?  $cnter{$word}{$title} : 0;
      }
      print OUT join("\t", @t),"\n";
}
close(IN);
close(OUT);

This does not allow for a lot of data analysis in and of itself. It simply lists the data by word and its count per section. If you wanted to sort by highest word frequency per section (for example) you would need to build a more robust data structure or open the file this script creates and parse that file with another script. You could look at the output of the above script as your basic statistics from which you could perform more analysis of the data.

[katharnakh]

It seems that the array approach would require you know all the sections beforehand. It may require more than one process to get the final output. First build the hash that counts the words per section, then have another routine that builds the hash of arrays then prints the final output.

Not really, only thing which we need to consider is extra overhead(which, i consider) to find the index of the section in the array, so that count of a word belong to that section is incremented correctly.

Here is the implementation according to the datastructure which i had mentioned earlier. Below code modifies Kavin's earlier posted code, just to show how the implementation goes. I have commented some lines just to compare two datastructure creation and manipulation

use strict;
use warnings;
open(IN, "readme.txt") or die "ERROR: $!";
open(OUT, ">seeme.txt") or die "ERROR: $!";
my (%cnter, $title, @order);
while(<IN>) {
      next if (/^\s*$/);
      chomp;
      my @line = split(/\s+/);
      if($line[0] =~ /^=/) {
            $line[0] =~ tr/=//d; # remove all the "=" from the section title
            $title = "@line";
            push @order, $title;
      }
      else {
            tr/,.?!//d for @line; #remove some punctuation
            tr/A-Z/a-z/ for @line; #convert all text to lower case so 'Word' and 'word' are the same
            
            # a local hash to store { section => its index in the array
            my %index;
            @index{@order} = (0..$#order);
            
            #$cnter{$_}{$title}++ for @line; 
            ${$cnter{$_}}[$index{$title}]++ for @line;
      }
}

print OUT join("\t",@order),"\n";
foreach my $word (sort keys %cnter){
      print OUT "$word :";
      my @t = ();
      my $index=0;
      foreach my $title (@order) {
            #push @t, (exists $cnter{$word}{$title}) ?  $cnter{$word}{$title} : 0;
            push @t, (exists ${$cnter{$word}}[$index]) ? ${$cnter{$word}}[$index] : 0;
      }
      print OUT join("\t", @t),"\n";
}
close(IN);
close(OUT);

I appreciate Kavin's approach which is more simple, clean and code is more readable.

katharnakh.