I'm about to get a little long winded here and I'm sorry for that but I have a couple of issues that I need guidance with.
I have assembled a procedure that basically performs a bunch of web browser clicks to query Statistics Canada (SC). The query itself is intensive on SC’s server as I will receive SC server side errors and fail the procedure. Sometimes there are no issues so I know that the SC server can handle the query. When the procedure is successful, it downloads a csv file that is only 7kb.
My questions are:
What is my best course of action for querying the SC website? I have a couple of thoughts but don’t know how to implement either so I’ll just put them out there.
Option 1: Query the website when demand is low on the server. For example, having the procedure run at say 3:00 am Central (guessing at the time) may work but I am unsure. If this is a good option, I have no clue regarding how to time when the procedure should start.
Option 2: Cut my query into 4 queries that run sequentially. For instance, in my procedure right now, I load a string type variable with comma separated values that I coded to be extracted from our database. If I could cut this string into say 4 strings and then somehow have my procedure repeat itself and start the query all over again (4 times), I could see it working.
Is there an Option 3?
As it is, I think Option 1 is more ideal but as I said, I don’t have a clue regarding how I can set the time to run.