I have a scrapper that was written in php that gathers info from a website and outputs it in csv format. The problem that I have encountered is that there are short connection breaks which are enough for my script to stop executing and I am not sure whether it is anti bot program on the server side or it is because internet break for brief amount of time.
I use it once a week to gather lots of data so it can stop at any time and I have to run it manually from same place again.
So what I was trying to figure out is how to allow the script to run until it gets some result; in other words to let it try access specific link until it gets there.
Maybe I should try set_time_limit (0) or maybe something else.
Hope it's understandable :)
Thanks for help guys.