Member Avatar for Niloofar24

Hi.
How i can ask my crawler to print only the text of all <li></li> tags in a url page?
I want to save the text of all <li></li> tags in a text file (without<li></li> words.)

  • 3 Contributors
  • 8 Replies
  • 3K Views
  • 3 Hours Discussion Span
  • Latest Post Latest Post by Niloofar24

Dani AI

Generated

Good call following and using a parser instead of regex. Two quick upgrades will make this robust and fix the issues ran into later: switch to Python 3 and use a proper HTTP client. Python 2 shows the leading u'' on unicode strings and is long past end-of-life, so moving to Python 3 removes that confusion and gives better defaults for text handling.

Here is a minimal, reliable pattern that:

  • fetches the page safely,
  • extracts just the text of every <li>,
  • writes one item per line to a UTF-8 text file, and
  • checks whether the first three items start with A.
import requests
from bs4 import BeautifulSoup

url = "https://example.com/page"  # put the target URL here

r = requests.get(url, headers={"User-Agent": "li-scraper/1.0"}, timeout=15)
r.raise_for_status()

soup = BeautifulSoup(r.text, "lxml")  # or "html.parser" if lxml is not installed

items = [li.get_text(strip=True) for li in soup.select("li") if li.get_text(strip=True)]

with open("li.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(items))

top3 = items[:3]
if len(top3) == 3 and all(s.startswith("A") for s in top3):
    print(top3)
else:
    print("sorry!")

Notes and gotchas:

  • If you only want list items from a specific section, narrow the selector, e.g. soup.select("ul.features > li").
  • For nested markup inside <li>, get_text(strip=True) already flattens and trims whitespace; use separator=" " if you need spaces between nested elements.
  • If the site is dynamic (content built by JavaScript), you will not see the final HTML with requests. In that case, use an API if available, or a headless browser.

Recommended Answers

All 8 Replies

Member Avatar for Slavi

Use regular expressions Click Here

Here's quick example ..

>>> import re
>>> html = 'randomstuff<li>I am some text 12345</li>randomstuff'
>>> re.findall(r'<li>(.+)</li>',html)
['I am some text 12345']
Member Avatar for Slavi

The problem is that your html variable is just a string containing this value
https://www.daniweb.com/software-development/python/threads/492669/how-to-print-only-the-content-of-all-tags-from-a-url-page
and not the actual HTML code ... the library that you have imported urllib2 .. use it to get the code from that page
Read urllib2
and also the example from there ..

import urllib2
response = urllib2.urlopen('http://python.org/')
html = response.read()

also.. should this >>> re.findall(r'<p>(.+),/p>', html)
be >>> re.findall(r'<p>(.+)</p>', html)?
and I am not sure if you read the link I gave you earlier about regular expression but the . matches any character including space. The + stands for that get all character that match the pattern stated which in our case was the . representing any character between <li></li> as in get all characters that match the pattern, as if it was only . without + it will simply return a single character that matches the pattern

Edited by Slavi
Member Avatar for snippsat

Use regular expressions Click Here

No no no just to make it clear :)
Have to post this link again.
Use a parser Beautifulsoup or lxml.

from bs4 import BeautifulSoup

html = '''\
<head>
  <title>Page Title</title>
</head>
<body>
  <li>Text in li 1</li>
  <li>Text in li 2</li>
</body>
</html>'''

soup = BeautifulSoup(html)
tag_li = soup.find_all('li')
print tag_li
for tag in tag_li:
    print tag.text

"""Output-->
[<li>Text in li 1</li>, <li>Text in li 2</li>]
Text in li 1
Text in li 2
"""
Edited by snippsat
commented: Great read, but this 'Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins.. he's gone too far=D +6
Member Avatar for Niloofar24

Thank you . Your example was exactly what i was looking for.

And thank you for your answer and explanation.

Member Avatar for snippsat

he's gone too far=D

Yes of course,to make it great humoristic read.
Regex can be ok to use some times,like you only need a singel text/value.

Both BeautifulSoup and lxml has build in support for regex.
Sometime it ok to use regex as helper to parser,when parsing dynamic web-sites
you can get a at lot of rubbish text.

Edited by snippsat
Member Avatar for Niloofar24

I wanted to post this question into a new Discussion but as it was related to this discussion so i will ask here. My code:

from bs4 import BeautifulSoup
import urllib2

mylist = []

url = ''
html = urllib2.urlopen(url).read()
soup = BeautifulSoup(html)
tag_li = soup.find_all('li')
for tag in tag_li:
    if tag.text.startswith('A'):
        mylist.append(tag.text)
if 'A' in mylist[0]:
    if 'A' in mylist[1]:
        if 'A' in mylist[2]:
            print mylist
else:
    'sorry!'

The output must be the else message but it print this output:

[u'Apple', u'Age', u'Am']

What is the problem? I want the script to check if the first 3 words (indexes) of mylist start with the letter 'A', print the list, but if not, print 'sorry!'. But as you can see here, it has printed even the index[4]!

And one more question, how i can remove those u letters that has printed into output?

Edited by Niloofar24
Member Avatar for Niloofar24

Well, it seems my question is basically wrong.
Forget that question, sorry!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.