Member Avatar for Niloofar24

Hello, me again :)
With this code:

>>> from BeautifulSoup import BeautifulSoup
>>> import urllib2
>>> url = urllib2.urlopen('http://www.python.org').read()
>>> soup = BeautifulSoup(url)
>>> links = soup('a')
>>> print links

A list of links printed into the terminal. I want to send the list into a text file, i tried this:

>>> with open('python-links.txt.', 'w') as f:
...     f.write(links)

But there was an error:

  File "<stdin>", line 2, in <module>
TypeError: expected a character buffer object
What is the problem? How can fix that?

And one more question; as that list looks like this: (I will copy only small part of the list)

[<a href="#content" title="Skip to content">Skip to content</a>, <a id="close-python-network" class="jump-link" href="#python-network" aria-hidden="true">
<span aria-hidden="true" class="icon-arrow-down"><span>&#9660;</span></span> Close
                </a>, <a href="/" title="The Python Programming Language" class="current_item selectedcurrent_branch selected">Python</a>, <a href="/psf-landing/" title="The Python Software Foundation">PSF</a>,

So how can i drop each link into a new line?
I tried this:

>>> text = '\n'.join(links)

But i got this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: sequence item 0: expected string, Tag found

How can i do that?

  • 3 Contributors
  • 3 Replies
  • 762 Views
  • 19 Hours Discussion Span
  • Latest Post Latest Post by Niloofar24

Dani AI

Generated

The printed output is a Python list of BeautifulSoup Tag objects, not plain text, so methods that expect strings (like join or file.write) fail. Build on and by extracting and normalizing the anchor targets, then write one URL per line while handling relative links, duplicates and encoding issues.

A compact, robust workflow:

  • fetch the page with a modern HTTP client,
  • parse with BeautifulSoup 4,
  • for each <a> use a.get('href') (skip None), turn relative URLs into absolute ones, drop URL fragments, and ignore non-http(s) schemes,
  • preserve discovery order while deduplicating,
  • write using UTF-8 (or io.open/codecs.open on Python 2) and a simple newline join.

Example (Python 3):

from bs4 import BeautifulSoup
import requests
from urllib.parse import urljoin, urldefrag

resp = requests.get('http://www.python.org', timeout=10)
soup = BeautifulSoup(resp.text, 'html.parser')

seen = set()
out = []
for a in soup.find_all('a'):
    href = a.get('href')
    if not href:
        continue
    href = urljoin(resp.url, href)
    href, _ = urldefrag(href)
    if href.startswith(('http://', 'https://')) and href not in seen:
        seen.add(href)
        out.append(href)

with open('python-links.txt', 'w', encoding='utf-8', newline='\n') as f:
    f.write('\n'.join(out))

Troubleshooting notes: watch for anchors with no href, skip mailto:/javascript: links, remove the stray trailing dot from filenames (your example used python-links.txt.), and if you need visible anchor text instead of URLs use a.get_text(strip=True). This approach avoids the TypeError caused by trying to write Tag objects directly and produces a clean, one-URL-per-line file.

Recommended Answers

All 3 Replies

Member Avatar for Gribouillis

Python complains because the file's write() method needs a string argument. Here the correct way to handle things is to find the values of the href= attributes, which contain the link targets. If you want to write anything to the file, you can use write(str(anything)).

Member Avatar for snippsat

Use the new bs4,do not call old BeautifulSoup.
Do not use read(),BeautifulSoup detect encoding and convert to Unicode.

As mention you need take out href attributes,
and you most learn to study webpage with Firebug or Chrome DevTools.
So then you see that you only need adresses that start with http and have href attributes.

from bs4 import BeautifulSoup # Use bs4
import urllib2

url = urllib2.urlopen('http://www.python.org') # Do not call read()
soup = BeautifulSoup(url)
with open('python-links.txt.', 'w') as f:
    for link in soup.find_all('a'):
        if link['href'].startswith('http'):
            f.write('{}\n'.format(link['href']))
Edited by snippsat
Member Avatar for Niloofar24

Thank you @Grebouillis.

Thank you .

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.