943,648 Members | Top Members by Rank

Ad:
Feb 26th, 2007
0

https question

Expand Post »
okay, it's been a while since I've asked any dumb simple questions you all know the answers to.
when a site is using https instead of http, can spiders still crawl that site?
My understanding is those pages load slightly slower and are just more secure. But I'm starting to suspect they cause all sorts of problems with spiders.
Thoughts?
Similar Threads
Reputation Points: 10
Solved Threads: 0
Newbie Poster
Track is offline Offline
3 posts
since Feb 2007
Feb 26th, 2007
0

Re: https question

14,550,000,000 for allinurl: http
196,000,000 for allinurl: https

Don't forget that https are reserved many times for checkouts and other secure parts of websites that aren't necessarily there to be crawled.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
Watt is offline Offline
3 posts
since Feb 2007
Feb 26th, 2007
0

Re: https question

In general https pages are not crawled or indexed by search engines. If you want a page to be crawled and indexed it is wise not encrypt its contents.
Last edited by stymiee; Feb 26th, 2007 at 8:55 pm.
Moderator
Reputation Points: 161
Solved Threads: 38
He's No Good To Me Dead
stymiee is offline Offline
1,422 posts
since May 2006
Mar 19th, 2007
0

Re: https question

Click to Expand / Collapse  Quote originally posted by stymiee ...
In general https pages are not crawled or indexed by search engines. If you want a page to be crawled and indexed it is wise not encrypt its contents.
I have seen this before and it really concerns me if I am understanding it correctly. I am new to "webmastering" and have read enough to be really confused at this point. I have a godaddy quick shopping cart that I have added about 80 products to with a goal to be web searchable and sell online. Godaddy HIGHLY recommended an SSL, so I added it. Now the entire domain is https (vtsupply) and all the product pages/descriptions are on https pages. I have submitted to all the major search engines for crawling indexing and am working on my keywords, inbound links, etc.

Am I just spinning my wheels though if I am https? Should I revoke the license? Godaddy says it's no problem with search engines crawling/indexing my site, but I'm not sure I believe it and I don't want to "wait 6 weeks" to view reports and see if there is a problem at that time. Another recommendation made was to host another identical site (another cost and setup) with links to SSL godaddycart (godaddy won't/can't apply the SSL to just the cart since currently, "my whole site is the cart".). It took a ton of time to setup my existing site and I'm not looking forward to doing it all over again for a duplicate (non SSL) site linked to the cart site. At this point I am really bummed and confused on this matter.:rolleyes: THANKS for any assistance. Aaron
Reputation Points: 10
Solved Threads: 0
Newbie Poster
Valhalla is offline Offline
1 posts
since Mar 2007
Mar 19th, 2007
0

Re: https question

The only pages that need to be secure are your order pages and any other page that captures and transmits sensitive data. Otherwise you are just spinning your wheels and making your server work hard encrypting pages that don't need to be encrypted.
Moderator
Reputation Points: 161
Solved Threads: 38
He's No Good To Me Dead
stymiee is offline Offline
1,422 posts
since May 2006
Mar 26th, 2007
0

Re: https question

if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.
For your http protocol (http://yourserver.com/robots.txt):
User-agent: *
Allow: /
For the https protocol (https://yourserver.com/robots.txt):
User-agent: *
Disallow: /
Reputation Points: 10
Solved Threads: 0
Junior Poster in Training
Candy_ME is offline Offline
65 posts
since Mar 2007
Mar 28th, 2007
-1

Re: https question

there is no way to do it via your robots.txt file i would recommend that you use the noindex meta tag!
Reputation Points: 10
Solved Threads: 4
Light Poster
nytrokiss is offline Offline
47 posts
since Jun 2006

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in Search Engine Optimization Forum Timeline: How to know that Google Crawler visits our website
Next Thread in Search Engine Optimization Forum Timeline: Exchange links with pages of different thematic





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC