Member Avatar for jeepj27

Hi All -

Forgive this naieve questions but I am not sure of the answer and know someone can help on Daniweb!

If I choose to ENCRYPT my HTML pages (for whole page) will the encryption scrample the text that the robots or crawlers will find?

That is - obviously encryption takes a whole language (text wise) on it's own and I want to be sure that I won't shy away the bots because there is no text visible in the coding.....

Hopefully this makes sense!!

Does anyoe have any experience with this or have any thoughts?

Thanks SO much!

-John

  • 3 Contributors
  • 4 Replies
  • 178 Views
  • 1 Week Discussion Span
  • Latest Post Latest Post by MidiMagic

Dani AI

Generated

Short answer: if the page is encrypted and only decrypted in the browser (client‑side), search engines will not reliably see the real text or links; if the server sends fully decrypted HTML, spiders see the content normally. This is the practical reason why client‑side “hide the source” tricks break SEO and why ’s point about when the page is decrypted matters. [Google’s JavaScript SEO guidance describes the crawl → render → index flow and why server‑side or pre‑rendering is preferable]. (developers.google.com)

For a musician site that needs public promotion but private downloads, separate the public content from the protected assets. Keep marketing pages plain HTML (indexable), and gate playable/downloadable files behind server‑side controls: authenticated endpoints, a streaming/proxy route, or short‑lived signed URLs. Use server logic to verify payment or session, then either (a) return an expiring presigned URL (S3/Cloud storage) or (b) hand off delivery to the web server (X‑Accel‑Redirect / X‑Sendfile) so the real file path never appears in page source. Example (Express + Nginx handoff):

app.get('/download/:id', verifyPurchase, (req,res)=>{
  res.set('X-Accel-Redirect', `/protected/${req.params.id}.mp3`);
  res.set('Content-Type','audio/mpeg');
  res.status(200).end();
});

See AWS presigned URLs for the time‑limited link pattern. (docs.aws.amazon.com)

A few operational cautions: client‑side obfuscation/“encryption” is security‑through‑obscurity and can be reversed or bypassed; sensitive controls belong on the server. Also, robots.txt is for crawl control, not secure hiding — a disallowed URL can still show up in results if other pages link to it, so don’t rely on robots.txt to hide downloads. Use meta robots / X‑Robots‑Tag or password protection when you must prevent indexing. (cheatsheetseries.owasp.org)

Testing and verification: compare the raw HTML (view‑source) and a rendered snapshot (or use Google Search Console’s URL inspection) to confirm what crawlers actually see; that will reveal whether client‑side tricks are preventing indexing. (developers.google.com)

Recommended Answers

All 4 Replies

Member Avatar for MattEvans

The important thing is how (and even more importantly when) you decrypt your page. If your page content is decrypted using javascript or other client-side means; then, because most search engines don't process client side scripts; your page wil not be indexed with relevant content.

If you decrypt during a server side operation, or use an encrypted transfer protocol (like SSL); and basically providing your page ends up as decrypted source being delivered as a result of a request; then search spiders won't know the difference.

If you're encrypting page content for any reason other than 'hiding the source code', then why do you want the pages to be indexed in search engines?

Member Avatar for jeepj27

The important thing is how (and even more importantly when) you decrypt your page. If your page content is decrypted using javascript or other client-side means; then, because most search engines don't process client side scripts; your page wil not be indexed with relevant content.

If you decrypt during a server side operation, or use an encrypted transfer protocol (like SSL); and basically providing your page ends up as decrypted source being delivered as a result of a request; then search spiders won't know the difference.

If you're encrypting page content for any reason other than 'hiding the source code', then why do you want the pages to be indexed in search engines?

It's actually because I am building a site that will be used for download of a musicians music... so I just wish to hide the links for downloads, etc...

Though I am using paypal for the actual links for the download and will encrypt those... I am using other links to PLAY the music online and do not wish people to be able to see those links...

Make sense?

Member Avatar for MattEvans

Sort of; but if you don't wish people to see the links, you probably should not put them on the Internet... If you want to restrict certain people from accessing the links; encrypting page content isn't the best way to go about it.

Do you want spiders to not see the links; or people to not to see the links?

Search spiders are like very fast people on very low-end browsers capable of following every link on your site, probably simultaneuosly. If you don't link to things; spiders can't see them. If you make your links difficult for a person to follow; spiders will find them difficult to follow. There's ways to stop spiders following links; and it's alot easier to stop (amicable) spiders from following links than it is to stop people following them.

Providing that enough of your page can be interpretted by a person on a browser without client-side-scripts; (even better, if it can be interpretted by looking at the source of the page as it is from the 'View Source' button); and providing that content is considered acceptable to the search engine's rules; then your pages will be indexed in some way.

If your page when viewed from the 'View Source' button is an unintelligable mix of encrypted HTML code; search spiders wouldn't understand it.

To answer your question (very generally); if you (effectively) stop users from seeing something, you stop spiders from seeing it to. If you stop spiders from seeing something (using many available means) you don't neccessarily have to stop users from seeing it. If you want to stop both; well that's easy.

Member Avatar for MidiMagic

You are forgetting that, once a user finds a web page, he can find any other file in the same folder using simple methods. He can also find subfolders.

I have a page which is not linked to from anywhere else, except that my browser starts with it (to save download MBs for real users). Yet I found it indexed on a search engine last week.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.