Member Avatar for PhoenixInsilico

I am trying to parse an html file. But unable to remove spaces using \s (matching character for whitespace)

use strict;
use warnings;
open(FILE,"<paragraph.txt")|| die "Can't open para.txt";
my @file = <FILE>;
my $all = join("",@file);
$all =~ s/\n/ /g;
$all =~ s/\./\. /g;
$all =~ s/\s\s*/ /g;
open (FIL,">paraone.txt")||die "Can't open para.txt";
print FIL $all;
close(FILE);
close(FIL);

I hava attached the paragraph.txt which contains spaces that are not removable.
<p style='text-align:justify'> this space Confusion and ensuing controversy also arose from the PANDAS
Please help...

<p style='text-align:justify'> Confusion and ensuing controversy also arose from the PANDAS criterion requiring that symptom onset and exacerbations be temporally associated with a GAS infection<sup>1</sup>. Difficulties establishing this association were predicted by observations of SC patients, where the onset of chorea typically may lag behind the inciting GAS infection              by 4  6 months or longer.
Two years later
Edited by PhoenixInsilico because: n/a
  • 3 Contributors
  • 2 Replies
  • 134 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by histrungalot

Dani AI

Generated

This is a classic encoding/byte-vs-character problem. saw that \s didn’t remove some gaps; suggested line breaks and showed there are non-standard bytes present. Those answers point to the right area: the file contains whitespace-like bytes that your current read/regex steps never treat as Perl characters, so \s (and simple byte-level fixes) won’t reliably match them.

Recommended workflow:

  • Inspect the raw bytes with a hex tool (for example hexdump/xxd) or print code points from Perl to confirm which code points are present.
  • Determine the file encoding (check HTML meta tags or use a detector). If the bytes are not already decoded into Perl’s internal strings, decode them first (or open the file with an appropriate encoding layer).
  • Once you have character strings, normalize and collapse whitespace using Unicode-aware classes rather than raw byte patterns. For example, read with an encoding layer and then replace all space-like characters with a single ASCII space.

Example snippets (not identical to the earlier posts) that illustrate reading with an encoding layer and using Unicode properties:

open my $fh, '<:encoding(UTF-8)', 'paragraph.txt' or die $!;
local $/; my $text = <$fh>; close $fh;
$text =~ s/\p{Space}+/ /g;

For HTML input, prefer parsing and entity-decoding (use an HTML parser and decode entities) before whitespace normalization — that avoids brittle regex fixes inside tags. See perluniintro for Perl/Unicode basics, [Encode] (https://perldoc.perl.org/Encode) for decoding, and [HTML::TreeBuilder] (https://metacpan.org/pod/HTML::TreeBuilder) / [HTML::Entities] (https://metacpan.org/pod/HTML::Entities) for robust HTML handling.

Troubleshooting tip: if a replace still fails, print the offending character’s code point (e.g., printf "U+%04X\n", ord $char) to identify exactly what you’re dealing with and choose the right decode/normalize step.

Recommended Answers

All 2 Replies

Member Avatar for pritaeas

According to my text editor they are line feeds (0x0A). Perhaps this link can help.

Edited by pritaeas because: n/a
Member Avatar for histrungalot

Your file has space and other things:

$ od -tx1 -c -A d paragraph.txt | more
0000000  3c  70  20  73  74  79  6c  65  3d  27  74  65  78  74  2d  61
          <   p       s   t   y   l   e   =   '   t   e   x   t   -   a
0000016  6c  69  67  6e  3a  6a  75  73  74  69  66  79  27  3e  a0  a0
          l   i   g   n   :   j   u   s   t   i   f   y   '   > 240 240
0000032  a0  a0  a0  a0  a0  a0  a0  a0  a0  a0  a0  a0  a0  20  43  6f
        240 240 240 240 240 240 240 240 240 240 240 240 240       C   o
0000048  6e  66  75  73  69  6f  6e  20  61  6e  64  20  65  6e  73  75
          n   f   u   s   i   o   n       a   n   d       e   n   s   u

Notice that what you think is space (0x20) is really (0xa0). Use this to start making it work. Think there are other non-ascii to filter out as well. Like after infection there is a 0x94, will need to filter that out.

#!/usr/bin/perl
use strict;
use warnings;
open(FILE,"<paragraph.txt")|| die "Can't open para.txt";
my @file = <FILE>;
my $all = join("",@file);
print $all;
$all =~ s/\n/ /g;
$all =~ s/\./\. /g;
# ---> Notice the use of \xa0
$all =~ s/\xa0+\s+|\s+/ /g;  # <------ Look here
print $all;
open (FIL,">paraone.txt")||die "Can't open para.txt";
print FIL $all;
close(FILE);
close(FIL);
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.