Member Avatar for roulette843

I want to create a method in Python that will read all of the text on a webpage and put it into a text file. Eventually I will have it distinguish between hyperlinks by having it exclude lines in the text containing href="....."

Any help would be greatly appreciated.

  • 2 Contributors
  • 1 Reply
  • 3K Views
  • 7 Hours Discussion Span
  • Latest Post Latest Post by vegaseat

Dani AI

Generated

you do not need to filter out href=... yourself if your goal is the page’s visible text. A proper HTML parser will ignore tag attributes when you extract text. Building on ’s idea, here is a Python 3 approach that handles encoding, strips scripts/styles, and lets you decide what to do with hyperlinks. It uses requests for fetching and Beautiful Soup for parsing.

import requests
from bs4 import BeautifulSoup

def page_text_to_file(url, out_path):
    headers = {"User-Agent": "Mozilla/5.0 (DaniWebExample/1.0)"}
    r = requests.get(url, headers=headers, timeout=15)
    r.raise_for_status()
    if not r.encoding:
        r.encoding = r.apparent_encoding

    soup = BeautifulSoup(r.text, "lxml")

    # drop non-content elements
    for tag in soup(["script", "style", "noscript"]):
        tag.decompose()

    # choose ONE of the following:
    # 1) exclude hyperlinks entirely:
    for a in soup.find_all("a"):
        a.decompose()
    # 2) or keep anchor text but remove tags:
    # for a in soup.find_all("a"): a.unwrap()

    text = soup.get_text(separator="\n", strip=True)
    lines = [ln for ln in (s.strip() for s in text.splitlines()) if ln]
    with open(out_path, "w", encoding="utf-8") as f:
        f.write("\n".join(lines))

if __name__ == "__main__":
    page_text_to_file("https://example.com/", "page.txt")

Tips: many sites block default clients, so set a User-Agent and a reasonable timeout. If a page builds content with JavaScript, a simple GET will not see it; use a headless browser (e.g., Playwright) for those cases. Be mindful of robots.txt and the site’s terms. For finer control, Beautiful Soup’s get_text() and decompose()/unwrap() behavior is documented here: Beautiful Soup docs. For robust HTTP usage (retries, headers), see Requests quickstart.

Member Avatar for vegaseat

One of the more simple ways to do this is to use Python module HTMLParser ...

# extract text from HTML code of a web site

import urllib2
import HTMLParser
import cStringIO

class HTML2Text(HTMLParser.HTMLParser):
    """
    extract text from HTML code
    """
    def __init__(self):
        HTMLParser.HTMLParser.__init__(self)
        self.output = cStringIO.StringIO()

    def get_text(self):
        """get the text output"""
        return self.output.getvalue()

    def handle_starttag(self, tag, attrs):
        """handle <br> tags"""
        if tag == 'br':
            # Need to put a new line in
            self.output.write('\n')

    def handle_data(self, data):
        """normal text"""
        self.output.write(data)

    def handle_endtag(self, tag):
        if tag == 'p':
            # end of paragraph. Add newline.
            self.output.write('\n')


# test it ...
if __name__ == '__main__':
    urlStr = 'http://www.python.org/'
    try:
      fileHandle = urllib2.urlopen(urlStr)
      html = fileHandle.read()
      fileHandle.close()
    except IOError:
      print 'Cannot open URL %s for reading' % urlStr
      
    #print html  # test only

    print '-'*50
    print 'Text ectracted from HTML code of URL =', urlStr
    print '-'*50
    
    p = HTML2Text()
    p.feed(html)
    text = p.get_text()
    # remove all the empty lines and leading/trailing white spaces from
    # the raw extracted text add back the newline character to each line
    raw_list = text.splitlines()
    new_list = []
    for line in raw_list:
        line = line.strip()
        if line != '':
            line = line + '\n'
            new_list.append(line)
        
    #print new_list  # test only
        
    clean_text = "".join(new_list)
    print clean_text
Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.