Search DaniWeb - scrapy

Re: Windows Media Player 11 Beta 2 Pretty Cool? 17 Years Ago by SamY Scrapy media player help to scrapy with python 5 Years Ago by zoro007 Hello, I want help in scrapy i want scrap this page full source code as Inspect …/recessed-lighting/led when i use this code import scrapy class TEST(scrapy.Spider): name = "quotes" start_urls = [ 'https://www.contechlighting… Re: help to scrapy with python 5 Years Ago by rproffitt Just for giggles I looked at the page source with a browser. The products do not show in the page source so scrapy and maybe CURL won't do that for you. Instead of scraping their web site, if you want product details from this company, just ask them for it. XPath help needed 14 Years Ago by Bash23 Ok I'm trying use scrapy (python module) to get stats off [URL="http://www.… Re: XPath help needed 14 Years Ago by somnia Try removing the tbody elements from the XPath. Those are added automagically by the browser, and are not present in the original HTML. More info here: [url]http://doc.scrapy.org/topics/firefox.html#caveats-with-inspecting-the-live-browser-dom[/url] Python - Installing 10 Years Ago by matthew.l.diefenbacher … from a Python site. I need to install easyinstall and scrapy. For easyinstall I enter and run: #!python """… Re: any suggestion for tutorial to create web crawler with python urllib? 9 Years Ago by vegaseat You could also take a look at Scrapy: http://scrapy.org/ and https://pypi.python.org/pypi/Scrapy Re: How can I write my web crawler for data extraction? 14 Years Ago by somnia Try [URL="http://scrapy.org"]Scrapy[/URL]. It's a very simple (though quite powerful) web crawling and screen scraping framework for Python. It's also pretty well documented, and has a growing community. Re: How To Check Whether An Html Element Or Plain Text Exists On A Page Or Not 6 Years Ago by cereal … are excellent solutions written in other languages, see for example Scrapy: * https://scrapy.org/ > Curious to see what the code would… Re: Get friend list from FaceBook 11 Years Ago by kindo I don't have any idea of Php to do such function but I think Python can do that through through scrapy. Re: How To Check Whether An Html Element Or Plain Text Exists On A Page Or Not 6 Years Ago by UI Cereal, I don't understand all that DOM stuffs. ALthought, heard of it before. Anyway, thankls for the Scrapy link. Re: XPath help needed 14 Years Ago by Bash23 [CODE]hxs.select("/html/body[@id='com-nfl']/div[@id='doc']/div[@id='doc-wrap']/div[@id='main-content']/div[1]/div[2]/div[1]/form/table[@id='result']/child/child::text()").extract() [] [/CODE] Still not getting a result :( Re: XPath help needed 14 Years Ago by somnia you could try with a simpler xpath like: //table[@id='result']//text() Re: XPath help needed 14 Years Ago by Bash23 [QUOTE=somnia;1128946]you could try with a simpler xpath like: //table[@id='result']//text()[/QUOTE] That worked but I got a bunch of jibberish. [code]\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\t\t26.6\r\n\t\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\r\n\t\t\t\t\t\r\n\t\t\t\t\r\n\t\t\t', u'\n', u'\r\n\t\t\t\t\r\n\t\t\t\t\… Re: XPath help needed 14 Years Ago by Bash23 Bump! Re: XPath help needed 14 Years Ago by varun0703 Hi, Dont wrry!!! i'll solve that but i need XML data ??? would u like to paste ur XML content here so that i might write correct x-Path????? Re: XPath help needed 14 Years Ago by Bash23 It's HTML and the link is in the OP. If it's easier you can use yahoo also [url]http://sports.yahoo.com/nfl/stats/byposition?pos=QB&conference=NFL&year=season_2009&timeframe=All&qualified=0&sort=49&old_category=QB[/url] Re: XPath help needed 14 Years Ago by varun0703 I am unable to open this link....sorry....Plz put some extracted content of that HTML , if u can.. Re: Python - Installing 10 Years Ago by renierdbruyn you need to install [setuptools](https://pypi.python.org/pypi/setuptools) for easy_install to work. I prefer to use pip though Re: Python - Installing 10 Years Ago by HatGuy Yesterday, I received the same uninformative message. On studying the apparently official setuptools install script the o.p. posted, I found a double negative. In the o.p.s code, line 66 in _install: if not _python_cmd('setup.py', 'install', *install_args): log.warn('Something went wrong during the installation.') log…