Member Avatar for Ayush_5

I've tried inurl:http but it takes forever to get even a bunch of sites right and I have to think of new keywords everytime to get the sites. Is there kind of a directory or a script i could use to filter the http from all the sites on the web. Or some sort of software?

  • 4 Contributors
  • 5 Replies
  • 351 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by Ayush_5

Dani AI

Generated

Short answer (for ): no — there is no practical, authoritative “list of every HTTP site.” The public web is massive and constantly changing; large open crawls are snapshots of parts of it rather than a complete inventory. Use existing crawl/archive and scan datasets to approximate coverage instead of trying to enumerate everything yourself. Common Crawl. (commoncrawl.org)

Practical alternatives and what they give you:

  • Crawl archives (Common Crawl) give page captures and link graphs.
  • Internet-scan datasets and projects (Project Sonar / Rapid7, scans.io) publish forward-DNS lists, HTTP GET responses and TCP scan results that researchers use as a baseline.
  • Host/service search engines (Censys, Shodan) index banners, open ports and certificates so you can find hosts that serve HTTP vs only HTTPS.
    Use these instead of building a universal index from scratch. Rapid7 Open Data / Project Sonar · Censys API docs. (opendata.rapid7.com)

A simple, repeatable workflow (sampling/approximation):

  1. Start from a domain list (Tranco / Majestic top lists) or a forward-DNS dump.
  2. Resolve names to IPs / expand subdomains (use Rapid7 FDNS or Common Crawl indexes).
  3. Lightly discover hosts with port 80 open using an internet‑scale scanner (ZMap/masscan) — only on targets you own or have permission for.
  4. Probe application layer with an HTTP tool (httpx or curl) and record status codes and any Location header (many HTTP sites immediately redirect to HTTPS). Example probes:
# follow redirects and show status+location (works on a host list)
cat hosts.txt | httpx -silent -fr -status-code -location -o http_probe.jsonl

# quick manual check
curl -I -s -L http://example.com | sed -n '1,5p'

See ZMap and httpx docs for flags and responsible usage. (zmap.io)

Cautions / scope notes:

  • Internet‑wide active scanning has ethical, operational and legal implications; follow scanner best practices and your network/provider rules. ZMap/masscan project pages discuss “good‑citizen” guidelines.
  • Public archives and Sonar datasets intentionally exclude login‑protected “deep web” and Tor hidden services; those require Tor‑aware crawlers and specialized indexes (e.g., Ahmia) and are out of scope for surface‑web lists. As and hinted, scale and scope are the real blockers. (zmap.io)

Use the datasets above to build targeted, reproducible samples (by date and source) rather than chasing a nonexistent complete list.

Recommended Answers

All 5 Replies

Member Avatar for Dani

Member Avatar for JamesCherrill

All http sites in the world? Do you have any idea how many terabytes (zettabytes?) that is?

commented: Thanks, James. One Last thing? If that wont work then Can i create script which could filter out all the https results in a search result +0
Member Avatar for Dani

I provided you a link to a cached version of the Open Directory Project, which attempted to classify and organize every worthwhile website that existed. It no longer exists.

There are also tools such as Ahrefs and Moz that attempt to index every page based on ranking in Google search results for the purpose of gaining information and improving SEO.

Member Avatar for rproffitt

I have a follow up question for Ayush_5. Beyond the issue raised by JamesCherrill are you looking for dark web listings as well? Those are the part of the Internet not accessible by traditional web search engines also known as deep web.

Maybe you could share what you are trying to really do. Maybe you want to make your own index of all sites but to do so would have you building a small warehouse for the server farm to do the search of your databases. That's the only way to drive down the delay you mentioned up top.

Member Avatar for Ayush_5

Thank you guys so much for so much help. I think i got my answer.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.