Member Avatar for gutto786

Hey Everbody,

Iam quite new to web development. I have this issue to design a search engine and I cant figure out where to start from.Following is the problem statement

1.Users access our website and send a request. For each request, . In response to each query, the 3rd party system multiple results and store results in a repository on their side. Our system needs to retrieve those results and display them to the individual users that requested them. Our system is built on a Microsoft .NET / SQL Server platform

2.Multiple results can exist and some results might take minutes. We dont want the user to be waiting all this time and we want the results to be retrieved and displayed within seconds of them being available in the repository.

3.communication interface in whatever manner you request and build the necessary business logic on their side accordingly. We design protocol based on our specs: like cost, reliability, speed

Any help will be appreciated.

Edited by gutto786 because: Improvement in question
  • 3 Contributors
  • 2 Replies
  • 107 Views
  • 3 Weeks Discussion Span
  • Latest Post Latest Post by infinique

Dani AI

Generated

This is an asynchronous job/result problem more than a pure “search” feature. The reliable, low-latency pattern is: accept the user request quickly, assign a unique correlation id, queue the work, let background workers retrieve/normalize results from the third party, persist results, and push or let the client fetch updates as results arrive.

  • Presentation tier: lightweight API that accepts the query and returns a RequestId (HTTP 202). Provide a status endpoint and real-time update channel (SignalR / WebSocket / server-sent events) so the browser can receive incremental results without polling.
  • Application tier: an API/webhook endpoint + an internal queue. Decouple incoming notifications from result processing by enqueuing work. Worker processes handle communication with the third-party repo, normalize results, and write them to the DB.
  • Data tier: SQL Server Jobs table (RequestId, user, status, timestamps) and Results table (RequestId FK, result payload, partial-flag). Use proper indexes or a separate search index for heavy querying.

Suggested minimal workflow:

  1. Create Job row with GUID and status=Queued; return RequestId immediately.
  2. Enqueue a retrieval job (Message Queue / Service Broker / RabbitMQ / Azure Service Bus / Hangfire).
  3. Either accept third-party push (webhook) or run workers that poll the third-party repo with exponential backoff. Authenticate webhooks with HMAC/secret.
  4. Worker writes results (idempotent) and updates Job status; notify client over SignalR or let client poll /status/{RequestId}.

Operational notes and cautions: enforce idempotency and dedup, implement retries with exponential backoff, keep webhook handlers lightweight (persist then enqueue), secure endpoints with TLS and HMAC or tokens, and add monitoring/metrics for queue depth and latencies. is right that outsourcing to a third-party search provider reduces ops work, but building this pipeline gives full control over normalization and latency. Learning by building small prototypes will get you far—no need to join a search company first; start with a simple RequestId + worker prototype and iterate.

Recommended Answers

All 2 Replies

Member Avatar for canadafred

Any help will be appreciated.

Although I could not make much sense out of the last couple of comments (or how it all apllied), I'm still inclined to suggest you consider going with a third party search provider.

Edited by canadafred because: n/a
Member Avatar for infinique

Dude, by the time you figure that out the game is over. I would suggest that you try applying jobs with any of the Search Engine companies and learn from the basics once you starting working in these companies.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.