I want help in scrapy
i want scrap this page full source code as Inspect Element mode
when i use this code

import scrapy

class TEST(scrapy.Spider):
    name = "quotes"

    start_urls = [
        'https://www.contechlighting.com/en/products/recessed-lighting/led#t=Products&sort=%40wcs_site_tree_rank ascending&layout=card',

    def parse(self, response):
        filename = 'contechlighting.html'
        with open(filename, 'wb') as f:

get html source code and the site use dynamic content with javascript and i cant see the product items and results number on the top
how can get full source code with scapy as appear on Inspect Element on firefox ?
can any one edit on my code to do this ?


Just for giggles I looked at the page source with a browser. The products do not show in the page source so scrapy and maybe CURL won't do that for you. Instead of scraping their web site, if you want product details from this company, just ask them for it.