Member Avatar for posterreal

Found this in my logs this morning: CrawlConvera0.1 When you visit convera . c o m / it says they are " a leading provider of mission-critical enterprise search and categorization solutions. More than 900 commercial and government customers in 33 countries rely on Convera's solutions to power mission-critical applications. "

  • 4 Contributors
  • 4 Replies
  • 55 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by bobchrist

Dani AI

Generated

Good catch, — an unfamiliar crawler in server logs should be validated. It may be a legitimate enterprise crawler, a third‑party indexer, or a spoofed/abusive bot. and are right to be cautious, and 's nudge to check search/webmaster tools is sound. Note that Yahoo Site Explorer (what was commonly used in 2009) was discontinued; current alternatives for indexing checks are Bing Webmaster Tools and Google Search Console.

Verification checklist:

  1. Capture the full log line: timestamp, client IP, complete user‑agent string and requested URL.
  2. Lookup IP owner and ASN with a whois/IP service such as ARIN or ipinfo. A legitimate crawler will usually come from the vendor's ASN.
  3. Perform a reverse DNS (PTR) on the IP and then forward-lookup the returned hostname; authoritative crawlers generally show consistent PTR→A records. Check whether the client first requested /robots.txt and whether its request velocity and depth look polite; see robotstxt.org for expected behavior.
  4. Correlate owner/PTR/behaviour with the claimed user‑agent. If these do not match, treat the crawler as suspicious.

If confirmed abusive, apply blocking or throttling at the network/webserver/CDN layer rather than relying on robots.txt. Prefer temporary rate limits or 429 responses while verifying; permanent blocks should only be applied after careful confirmation to avoid cutting off legitimate indexers.

Recommended Answers

All 4 Replies

Member Avatar for Dani

Not sure what you mean? Yahoo! logs? *confused*

Member Avatar for mokmok69

I think I never heard that before any ways can you give us link about that.

Member Avatar for Dani

I'm confused.

Member Avatar for bobchrist

Let us know what actually you're looking for? If you want to explore all the web pages indexed by Yahoo! Search you should check with Yahoo site explorer.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.