Member Avatar for jazardezign

MSN Live search has deindexed some of my pages. Many of them are property pages like http://www.zagat.com/Verticals/PropertyDetails.aspx?VID=8&R=87971&AJX=Ntk%253DGeoChildID%2526Ntt%253D1971%2526N%253D120%2526VID%253D8

which used to be indexed. I went through the Live Search Guidelines and did not find the answer I was looking for. Do anyone know what would cause MSN Live Search to drop a page from the index and more important, how to get it reindexed and have it stay for good.

  • 4 Contributors
  • 3 Replies
  • 100 Views
  • 2 Months Discussion Span
  • Latest Post Latest Post by bobchrist

Dani AI

Generated

As described, some property pages vanished from the index. , and raised useful ideas about timing, similarity and tools — the practical next step is to discover the exact signal the crawler saw, then fix that signal and request reindexing.

Run these crawler-visible checks (examples use a Bing-style user agent):

Check HTTP headers and status

curl -I -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "https://example.com/path?params"

Look for non-200 statuses, X-Robots-Tag headers or unexpected redirects. X-Robots-Tag: noindex or a redirect chain ending in a noindex/404 will remove a page. See the X-Robots-Tag header docs for exact behavior: X-Robots-Tag (MDN).

Inspect HTML as a crawler

curl -s -A "bingbot" "https://example.com/yourpage" | grep -i '<meta name="robots"\|<link rel="canonical"'

Confirm the canonical link points to the intended URL (wrong canonicals will consolidate or drop pages). Guidance on canonical usage: Consolidate duplicate URLs (Google).

Check server crawl logs

grep -i bingbot /var/log/access.log | awk '{print $9, $7}' | sort | uniq -c

Find the crawl times and response codes. If bingbot saw lots of 4xx/5xx or short/empty responses, the engine can prune the URL.

Sitemap and parameter strategy
Ensure the canonical URL is in your XML sitemap and that query-parameter variants are either canonicalized or excluded. See the sitemaps protocol: Sitemaps.org.

After fixing headers/canonicals/redirects and ensuring unique, crawlable content or structured data for each property, request re-crawl via the search engine’s webmaster tools (expect days to weeks). If the problem repeats, gather a small packet: example URLs, bot access logs (timestamps + response codes), and before/after header dumps — that will speed diagnosis by support or a consultant.

Recommended Answers

All 3 Replies

Member Avatar for Techie08

Search engines do that quarterly it seems, they clean out old links and cache. It likely just happened due to the changing year and everything re-updated again. I'd suggest updating your pages and adding more content, changing some content, or adding some photos to make them appear more active than they may have when msn cleared them.

Member Avatar for canadafred

I had a quick look. Without any further information I think I can still take a good guess at what the problem is.

Seems to me like you probably have too many similarly appearing web pages whereas the search engine doesn't find the content sufficiently unique one from the other. There's a large percentage (I'd say 97%) of the web page elements that's exactly identical one to the other. Look at it like your web site unknowingly triggers the content duplication sensors.

Member Avatar for bobchrist

Try to use this tools to tweak how MSN Live Search crawls and find if there any problem for indexing your website; http://webmaster.live.com/

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.