Member Avatar for wulaishiwo

Hi all,

In my current project, I need to write python code extracting tons of pages grabbed from the web. By extraction, I mean strip all tags and comments and if possible, filter out small sections like navigation links. The only thing should be left is the length paragraph, if there's any.

1. About stripping the tags

I tried the html2text lib, but it will stop when encontered an error, like some ill-formated tag, unclosed tag, etc. Didn't figure out how to have it ignore such error.

So I end up using the old BeautifulSoup, my code snippet is

soup = BeautifulSoup(html)
    comments = soup.findAll(text = lambda text: isinstance(text, Comment))
    [comment.extract() for comment in comments]
    c = soup.findAll('script')
    [i.extract() for i in c]
    s = soup.findAll('style')
    [i.extract() for i in s]
    content = ''.join(soup.findAll(text=True))

Currently, it worked fine with a dozen documents tested. But I think it's not robust enough. Do you know any library than can do the job?

2. About filtering out short text, like title, list items

My solution is to split the whole document by '\n\n' and for each 'paragraph', I used a simple predicate to determine whether it's lengthy paragraph(has at least one ',' and have more than 200 chars):

text.find(',') != -1 and len(text) >= 200

Of course it didn't work very well. Do you have any suggestions?

Thanks a lot.

It certainly can't exclude all

  • 2 Contributors
  • 1 Reply
  • 274 Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by snippsat

Dani AI

Generated

As discovered, stripping tags is the easy part — reliably isolating the main article from navigation, headers, footers and lists is the tricky bit. is right to point at a tolerant parser: use BeautifulSoup with an HTML5/LXML backend or lxml.html so malformed pages don't break the pipeline. Two practical paths work well in practice: (A) use a purpose-built extractor that implements boilerplate-removal heuristics, or (B) build a simple block-scoring extractor that combines text density, link density and sentence/stopword signals.

For quick wins try a dedicated extractor (they already encode sane heuristics): jusText, Readability-style extractors, trafilatura, newspaper3k or boilerpy3. If you prefer a custom approach, score DOM blocks (p/div/article) using signals such as block text length, number of sentences, stopword count, and link density (ratio of link text to block text). That beats the brittle "contains a comma and >200 chars" rule because it rewards real sentence structure and penalizes navigation-heavy blocks.

A compact scoring example (tune stopwords/thresholds to your corpus):

from lxml import html
import re

STOPWORDS = set("the a and to of in is it for on that".split())

def score_block(node):
    text = ' '.join(node.itertext()).strip()
    if len(text) < 80:
        return 0
    links = ''.join(a.text_content() or '' for a in node.findall('.//a'))
    link_density = len(links) / max(1, len(text))
    stopwords = sum(1 for w in re.findall(r'\w+', text.lower()) if w in STOPWORDS)
    sentences = text.count('.') + text.count('!') + text.count('?')
    return stopwords + sentences*5 + (len(text)/100.0) * (1 - link_density)

def extract_main(html_bytes):
    tree = html.fromstring(html_bytes)
    for t in tree.xpath('//script|//style'):
        p = t.getparent()
        if p is not None:
            p.remove(t)
    candidates = tree.xpath('//article|//div|//p')
    if not candidates:
        return tree.text_content().strip()
    best = max(candidates, key=score_block)
    return best.text_content().strip()

Practical notes: detect and fix encodings before parsing, prefer HTML5/LXML parsers, and only resort to headless browsers for JS-heavy pages. For large-scale scraping prefer fast extractors (trafilatura/newspaper) and batch processing; tune thresholds on a representative sample to reduce false positives.

Member Avatar for snippsat

Currently, it worked fine with a dozen documents tested. But I think it's not robust enough. Do you know any library than can do the job?

BeautifulSoup is very robust,not many parser are so good.
You have lxml that is good,it also has BeautifulSoup and html5lib build in.
lxml has also xpath.

Edited by snippsat because: n/a
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.