Member Avatar for StepIre

Hey guys,
im trying to extract the top 10 links from a yahoo search results page. i can get all the links using the code below.. but that could be 70 links.

Any idea how i could get just those top 10 ranked ones? and not the adverts etc.

ie for this page..

http://uk.search.yahoo.com/search?p=python&fr=yfp-t-501&ei=UTF-8&meta=vc%3D

i would only want
1. www.python.org
2. www.pythonline.com
3.
.
.
10.
etc

heres main lump of my code that returns ALL links on that page.

Is there even anything to distinguish which are in the top ten that way i could try extract them.

if __name__ == "__main__":
    import urllib
    usock = urllib.urlopen("http://uk.search.yahoo.com/search?p=python&fr=yfp-t-501&ei=UTF-8&meta=vc%3D")
    parser = URLLister()
    parser.feed(usock.read())
    parser.close()
    usock.close()
    path = u"c:\\Users\\admin\\Desktop\\"
    i = 0
    for url in parser.urls: 
       if i <= (len(parser.urls)):
          print i
          print parser.urls[i]
          page = urllib.urlopen(parser.urls[i]).read()
          f = file(path + u"test" + str(i) + u".txt", "w+")   
          print >> f, page 
          f.close()
          print "Html file successfully printed to file!"

any help appreciated,

thanks guys :)

  • 2 Contributors
  • 2 Replies
  • 231 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by StepIre

Dani AI

Generated

Quick summary: was parsing every <a> on the Yahoo page with a simple SGML-based URLLister, so you get nav, footer, site links and ads — not just the ranked results. The reliable ways to get the "top 10" are (A) target the DOM elements that contain organic results (if you can identify them), or (B) use an official search API (recommended for production — scraping search pages is brittle and may violate terms).

Practical, low-friction approach (works well for experimenting):

  • Fetch the page with a real User‑Agent and reasonable timeouts.
  • Parse with BeautifulSoup or lxml.
  • Prefer anchors inside result-like containers (li/div/ol with class names containing "result", "res", "web", or the engine's pattern).
  • Skip anchors that are obviously wrappers or ad-related: rel="nofollow", javascript: links, internal search-engine hosts, or known ad domains.
  • Deduplicate by domain (so multiple pages on the same site count once) and keep order.
  • Stop when you have 10.

Example extractor (replace the heuristic checks for your target page as needed):

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse

def extract_top_links_from_html(html, base_url=None, limit=10):
    soup = BeautifulSoup(html, "html.parser")
    candidates = []

    # Heuristic: look for anchors inside result-like containers first
    for container in soup.find_all(['li', 'div', 'ol'], class_=lambda c: c and 'result' in c.lower()):
        a = container.find('a', href=True)
        if a and not a.get('rel'):
            candidates.append(urljoin(base_url or '', a['href']))

    # Fallback: scan anchors but skip non-http and obvious engine/ads
    if not candidates:
        for a in soup.find_all('a', href=True):
            href = urljoin(base_url or '', a['href'])
            p = urlparse(href)
            if p.scheme.startswith('http') and not p.netloc.startswith('search'):
                candidates.append(href)

    # dedupe by host and return first N
    seen = set(); top = []
    for h in candidates:
        host = urlparse(h).netloc.lower()
        if not host or host in seen: 
            continue
        seen.add(host)
        top.append(h)
        if len(top) >= limit: 
            break
    return top

Two quick troubleshooting tips: iterate with for url in urls[:10] or for i,url in enumerate(urls[:10],1) instead of manual indexing (that avoids off‑by‑one or skipping problems like i = i + 2). And always respect robots.txt, rate limits and the search engine terms; for stable, repeatable results use an API (Bing/Google/third‑party SERP APIs) rather than scraping.

Recommended Answers

All 2 Replies

Member Avatar for jrcagle

Your code is broken as it stands. What's the URLLister() class? I don't have it in my urllib.

Jeff

Member Avatar for StepIre

Your code is broken as it stands. What's the URLLister() class? I don't have it in my urllib.

Jeff

Sorry, its one of my own methods i didnt include all the ode just main chunk

from sgmllib import SGMLParser

class URLLister(SGMLParser):
    def reset(self):
        SGMLParser.reset(self)
        self.urls = []

    def start_a(self, attrs):
        href = [v for k, v in attrs if k=='href']
        if href:
            self.urls.extend(href)

if __name__ == "__main__":
    import urllib
    usock = urllib.urlopen("http://uk.search.yahoo.com/search?p=cinemas+in+dublin&fr=yfp-t-501&ei=UTF-8&meta=vc%3D")
    parser = URLLister()
    parser.feed(usock.read())
    parser.close()
    usock.close()
    path = u"c:\\Users\\Neil\\Desktop\\"
    i = 0
    for url in parser.urls: 
       if i <= (len(parser.urls)):
          print i
          print parser.urls[i]
          page = urllib.urlopen(parser.urls[i]).read()
          f = file(path + u"test" + str(i) + u".txt", "w+")   
          print >> f, page 
          f.close()
          print "Html file successfully printed to file!"
          i = i + 2

any idea how i can just get the top ten links?

thanks :)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.