Hi folks

I want to universal website crawler using PHP, so my crawler will work on any given site.
By using my web application, user will input any site, will provide input, what he needs to get from given site and will click on Start button. Then my web application will begin to get data from source website.
I am using iframe for this purpose , load page in iframe and using jquery I get class and tags name of spacific area from user. But when I load external website like ebay or amazon etc it does not work, as these site are restricted.
Is there any way to resolve this issue, so I can load any site in iFrame. If there is any alternative to what I want to achieve.

I am actually inspired with mozenda, a software developed in .NET, http://www.mozenda.com/video01-overview/. They load a site in browser control and almost doing same thing.
Please help me on this!!

Thank you

3 Years
Discussion Span
Last Post by cereal

This happens because of the X-Frame-Options header, if the remote server uses a directive like this:

Header always append X-Frame-Options SAMEORIGIN

The browser will check and won't allow the iframes, unless you don't use a local script through Greasemonkey or Tampermonkey. It means your users should install it on their browsers to bypass that kind of block.

Otherwise you could contact some of these website to be allowed, they can create a directive like this:

Header always append X-Frame-Options ALLOWFROM http://yourwebsite.tld/ 

* https://developer.mozilla.org/en-US/docs/HTTP/X-Frame-Options
* http://tools.ietf.org/html/rfc7034

Edited by cereal

This topic has been dead for over six months. Start a new discussion instead.
Have something to contribute to this discussion? Please be thoughtful, detailed and courteous, and be sure to adhere to our posting rules.