Member Avatar for JoseAmayo

Is there a way to implement a site search for secured pages? I am planning on changing the site search on an existing company page. We currently use Atomz and I am considering changing to Google Search Business Edition. My boss wants to keep the secured pages hidden from a regular Google web search, but create an internal search once people log in.

I believe the answer to this is "no," but I cannot find any specific information on this. Thank you for any help you can give.

  • 3 Contributors
  • 5 Replies
  • 125 Views
  • 5 Days Discussion Span
  • Latest Post Latest Post by JoseAmayo

Dani AI

Generated

The short, practical answer to the original question is that you can have a site search for pages that are hidden from public web indexing — but the implementation matters. correctly wants the secured area kept out of public Google results; and pointed to different routes (vendor vs self-hosted). The choice comes down to how you build the index and how you enforce access control.

Two reliable architectures are common: push-indexing (preferred for corporate sites) and authenticated crawling. With push-indexing you extract content from the CMS/database and send it to the search index via an API whenever content is created, updated or deleted. It is robust, auditable, and does not require exposing login flows to a crawler. Authenticated crawling has lower up-front development but is brittle (session handling, JavaScript, form logins) and harder to keep consistent at scale.

Important implementation notes and checklist:

  • Keep private and public indexes separate; never expose the private index endpoint to the open web.
  • Enforce access control at query time (map logged-in user to allowed scopes) or at index time (store ACLs with documents). Query-time filtering is simpler; index-time ACLs are faster for very large datasets.
  • Integrate search auth with your SSO/LDAP so results reflect user roles.
  • Use HTTPS, token-based query auth, and firewall rules to limit who can query the index.
  • Do not rely on robots.txt to protect sensitive pages; require authentication and avoid public links.
  • Plan for HA, replicas, snapshots/backups, monitoring, and load testing; simulate realistic query patterns and permission mixes.

For vendor evaluation and production rollout: require production references, SLAs, incident response procedures, and a security review. Run a pilot on a representative dataset and verify permission enforcement under failure scenarios. Given budget and skill constraints, weigh the tradeoff between a supported vendor and a self-hosted engine that can be integrated with your backend for push updates.

Recommended Answers

All 5 Replies

Member Avatar for JoseAmayo

Awesome, Passmark. Thank you very much. This is exactly what I was looking for. I searched everything I could think of, and could not find anything like it.

My only concern is reliability, as this is for a corporate website. I found several positive blog posts and c|net user reviews on this, but I was wondering if anyone here had experience with this product, or something similar. If you could let me know, it would be greatly appreciated.

Member Avatar for Passmark

No point asking me, I am hopelessly bias. I wrote the initial code for the Zoom product (and own the company :-) ). Needless to say, I think it is pretty good. Not perfect yet, but pretty good.

Member Avatar for Dani

I guess the only question to ask is if whether you want an inhouse or outsourced solution?

in house => try something called sphinx at sphinxsearch.com
outsource => google rack

Member Avatar for JoseAmayo

Awesome. Thank you, Dani. Unfortunately, I do not believe either of these solutions will work. Sphinxsearch is too technical for me, Google MiniSearch is outside of our budget, and Google's crawler cannot access our secured pages, as far as I can tell. Although Passmark should have been upfront about his affiliation with Zoom Search, it does seem like a very good product for the price. We are beta-testing the free version now.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.