I have an issue with running python scripts on linode cloud server. I create a scrapping script to scrape data by check in and check out so that i run the script using screen method. It run over one day very smoothly but today it hang. No response. When i call getRequest() method then i print the proxy number and send request with proxy and timeout=10.

def getRequest(self,url):
        self.proxieName = random.choice(self.proxyDict.keys())
        print getYellow("Proxy: ")+getCyan(self.proxieName)
        self.proxie = self.proxyDict[self.proxieName]
        try:
            self.getRqst=requests.get(url,headers=self.nrmlHeaders,proxies=self.proxie, timeout=10)
        except Exception, e:
            print e
            time.sleep(10)
            return self.getRequest(url)
        else:
            if self.getRqst.status_code==200:
                self.rqstErr = 0
                return self.getRqst
            else:
                self.rqstErr+=1
                print getYellow("Hi Dear, Don't worry. May be Server down. Exception in Url: ")+url
                print getRed("Error!!! ")+getBlue("I get Status Code: ")+str(self.getRqst.status_code)
                print getGreen("I am sleeping now for 10 second")
                time.sleep(10)
                return self.getRequest(url)

Here is the image snapshot link https://www.dropbox.com/s/yh2bs7z1mxcgkcd/script%20error.png?dl=0

after getting the content i parse them and save information into a CSV file and send another request.

And i see also that after 01:40 of 6th July 2015, the csv file alos not updated. So it's hanging.

Recommended Answers

You probably need to find some documentation about the timeout parameter in requests.get(), or perhaps you could try this solution with eventlet.Timeout().

Jump to Post

All 4 Replies

This function should be rewritten in a non recursive style. If the connection times out every time, it will call itself indefinitely until it reaches the recursion limit. Did you change the recursion limit ?

It is very easy to write this function with a while True loop.

Thank you, But after trying the request in line 6,

self.getRqst=requests.get(url,headers=self.nrmlHeaders,proxies=self.proxie, timeout=10)

it doesn't show any message in Exception Block in line 8 or line 17

print getYellow("Hi Dear, Don't worry. May be Server down. Exception in Url: ")+url

You probably need to find some documentation about the timeout parameter in requests.get(), or perhaps you could try this solution with eventlet.Timeout().

can i use post request with eventlet.Timeout()

Be a part of the DaniWeb community

We're a friendly, industry-focused community of 1.21 million developers, IT pros, digital marketers, and technology enthusiasts learning and sharing knowledge.