Menu
Menu
DaniWeb
Log In
Sign Up
Read
Contribute
Meet
Search
Search
About 52 results for
crawl-requests
- Page 1
304 Not Modified Crawl Requests
Digital Media
Digital Marketing
Search Engine Strategies
2 Years Ago
by Dani
Starting September 1st, about 25% of
crawl
requests
started returning 304 Not Modified instead of 200 OK. Prior …to that, there were just a small handful of such
requests
per day. I did not make any code changes that…
Mobile-first Indexing
Digital Media
Digital Marketing
Search Engine Strategies
4 Years Ago
by Dani
Just got a popup notification in my Google Search Console that says: > Your site has been switched to Mobile First Indexing > The majority of Google's
crawl
requests
to your site will be made using a mobile crawler. > > Switch date: July 10, 2018 The notification is just a little late to the party.
Re: Terms of SEO
Digital Media
Digital Marketing
Search Engine Strategies
15 Years Ago
by shennon
crawl
rate is the frequency of a site is being indexed by search engines. PPC is also known as a CPC which is cost per clicks which advertisers only pay if some one clicks on the ads.
Re: Can we crawl password protected website?
Programming
Software Development
13 Years Ago
by masijade
HttpUrlConnection or HttpsUrlConnection. It is possible to use Cookies and post
requests
and authentication with both of them. Search for some tutorials concerning them.
Link Popularity
Digital Media
Digital Marketing
Search Engine Strategies
19 Years Ago
by Vinoth
…This way search engine spiders will find the site map,
crawl
the links and index your entire web site. [U…related sites from your site. Search engine spiders will
crawl
your site's outgoing links and determine that the … potential link partners by sending personalized reciprocal link email
requests
, after you visit and approve those sites you wish…
Instagram Web Crawler
Programming
Software Development
6 Years Ago
by Stefce
…page <= max_pages: url = 'https://instagram.com/xenia/' source_code =
requests
.get(url) plain_text = source_code.text soup = BeautifulSoup(plain_text, "…problem with this code or instagram does not allow to
crawl
? **EDIT:** Somehow instagram doesnt display the classes i …
Re: Instagram Web Crawler
Programming
Software Development
6 Years Ago
by rproffitt
… read this: > You cannot use the API Platform to
crawl
or store users' media without their express consent. So that… web scraping methods. https://www.quora.com/How-can-I-
crawl
-Instagram-without-using-API kicks around the 3 methods but…
Re: What To Lookout For To Prevent Crawler Traps ?
Programming
Web Development
11 Months Ago
by AndreRet
…'s important to respect robots.txt files,
crawl
politely, and not overload servers with excessive
requests
. The error you're encountering is…
Re: R.E.S.P.E.C.T
Community Center
Geeks' Lounge
13 Years Ago
by WaltP
…[/B]nippets of Code (sorry) are [I]not[/I] help
requests
[B]P[/B]osts should not be just an assignment… 5 posts I saw were "gimme codz plz"
requests
. And we get hundreds of posts a day. [QUOTE=camdaddy09…;]One other response may go like this, “you should just
crawl
into a hole and die you moron.” I fail to…
Servers sending false 404s all day
Digital Media
Digital Marketing
Search Engine Strategies
10 Years Ago
by Dani
… of our six web servers sending 404 responses for all
requests
. Obviously traffic was way down during the outage, but there… the outage ended. I'm worried that googlebot came to
crawl
us and our servers essentially said, "Nope, sorry, those…
Re: Why I Fail To Extract Link Path Extension ?
Programming
Web Development
11 Months Ago
by borobhaisab
…/ 36 The extension is: https://bytenota.com/apache-redirect-http-
requests
-to-https/ 36 The extension is: https://bytenota.com/php…-https/ 36 The extension is: https://bytenota.com/tag/https-
requests
/ 36 The extension is: https://bytenota.com/tag/remove-folder…
Re: Servers sending false 404s all day
Digital Media
Digital Marketing
Search Engine Strategies
10 Years Ago
by mmcdonald
Are you not able to get this sort of information from webmaster tools? perhaps request an increase to your
crawl
rate for a few days?
Re: Servers sending false 404s all day
Digital Media
Digital Marketing
Search Engine Strategies
10 Years Ago
by Dani
GWT is always a few days behind, so it hasn't showed up there yet. The problem with
crawl
rate is that, like it or not, Googlebot received mixed signals from the site, so now they are less likely to "trust" that site moving forward, me thinks.
Re: Servers sending false 404s all day
Digital Media
Digital Marketing
Search Engine Strategies
10 Years Ago
by bradly.spicer
In GWT if you wait for the page errors to come up you can remove them all and Google will automatically re-
crawl
the site. This should negate any errors ^^
Re: CGI Re-direct Question Best Practice
Programming
Web Development
7 Years Ago
by cereal
… data is filled in - the checkout file if a search
crawl
hits the page it sends blank info to the generic… random crawlers, as genuine robots usually do not send POST
requests
. You could also add a csrf token to the session…
Re: My first Personal interview-Evaluate Me :)
Community Center
14 Years Ago
by apegram
… everyone has to start somewhere -- walk before you run and
crawl
even before that -- but those answers tell me the candidate…
Re: Instagram Crawler
Programming
Computer Science
6 Years Ago
by cereal
… -r requirements.txt Which in practice are: * http://docs.python-
requests
.org/en/master/ * https://pypi.python.org/pypi/selenium Bye!
Re: Googlebot ignores robots.txt
Digital Media
Digital Marketing
Search Engine Strategies
3 Years Ago
by rproffitt
I'm hearing more about fake googlebot
requests
. Maybe that? PS. Google those 3 words to find out more. Also: https://support.google.com/webmasters/answer/80553
Re: Googlebot ignores robots.txt
Digital Media
Digital Marketing
Search Engine Strategies
3 Years Ago
by rproffitt
Time for Google's own to give up ideas. Given the Google Seach Console (GSC) gives it a passing grade tells me that that likelihood of fake googlebot
requests
just went up even if the useragent is legitimate. Sorry for not defining GSC first. I'll work harder on that.
Re: Mark Cuban's Plan to Kill Google
Digital Media
Digital Marketing
Search Engine Strategies
14 Years Ago
by Howard
… but important detail here: if a site owner
requests
that Google doesn't
crawl
their site, Google has to abide by that… site itself to determine search relevance (since it can't
crawl
the pages), but if the anchor text and links are… good enough for Google to figure out the "no
crawl
" result should be ranked high. Net result: the "…
Re: What are specific steps for optimizing on-page or technical SEO?
Programming
Web Development
3 Months Ago
by vexanshop
… times by compressing images, leveraging browser caching, and minimizing HTTP
requests
. XML Sitemap: Create and submit an XML sitemap to search… visibility of your content in search results.
Crawl
Errors: Regularly monitor and address
crawl
errors reported by search engine tools like Google…
Re: Floating IP Address in AOL
Programming
Web Development
17 Years Ago
by TopDogger
… Guidelines specifically cover this issue. "Allow search bots to
crawl
your sites without session IDs or arguments that track their… spiders and do not initiate a session when a spider
requests
a page. You can easily detect a spider using $_SERVER…
Re: Spam !!
Digital Media
Digital Marketing
Search Engine Strategies
12 Years Ago
by Seobytes
… at search engines). The technique involves making repeated web site
requests
using a fake referrer url that points to the site… in turn be indexed by the search engines as they
crawl
the access logs. This benefits the spammer because of the…
Re: Renderer.repaint(); null pointer
Programming
Software Development
6 Years Ago
by JamesCherrill
… do. Posting repaint
requests
as fast as the CPU will, at best, just slow your machine to a
crawl
. To display an…
Re: Link Building Tutorial for DA/Referral Traffic
Digital Media
Digital Marketing
Search Engine Strategies
3 Years Ago
by Dani
… duplicate content penalty from Google, nor have any copyright takedown
requests
, and so we'll almost always delete content like this… sure if they still do it, but Google used to
crawl
and index tweets, and I've noticed that Google would…
Re: What are Your best SEO techniques?
Digital Media
Digital Marketing
4 Months Ago
by Bunker
… loads faster by optimizing images, leveraging browser caching, minimizing HTTP
requests
, and using a content delivery network (CDN). Faster page loading….txt file to tell search engine crawlers which pages to
crawl
and which pages to avoid. Schema markup: Implement structured data…
Re: 304 Not Modified Crawl Requests
Digital Media
Digital Marketing
Search Engine Strategies
2 Years Ago
by AussieWebmaster
you may want to try removing the caching code you added and should fix it - the instructions in 304s screw with browsers especially older ones
Re: 304 Not Modified Crawl Requests
Digital Media
Digital Marketing
Search Engine Strategies
2 Years Ago
by Dani
Hey Frank! :) It's not caching code. I am simply returning a Cache-control HTTP header (specifying the page is cacheable by the web browser), same as I've done for the past 20 years.
Re: 304 Not Modified Crawl Requests
Digital Media
Digital Marketing
Search Engine Strategies
2 Years Ago
by rproffitt
I doubt that it's connected but I still see that bug with the spinning circle. 1. Go to https://www.daniweb.com/articles/latest/articles 2. Press the End keyboard button. ![image_2021-09-18_090120.png](https://static.daniweb.com/attachments/1/d2626cd81dbb00b2e01e6157b7c7d7d6.png)
Re: 304 Not Modified Crawl Requests
Digital Media
Digital Marketing
Search Engine Strategies
2 Years Ago
by Dani
Yeah, that’s unrelated. Just a UI but I was planning on fixing snd then got caught up with other things and completely forgot about it. Can you please refresh my memory and link me to the thread where you wrote reproducible steps?
1
2
Next
Search
Search
Forum Categories
Hardware/Software
Programming
Digital Media
Community Center
Latest Content
Newest Topics
Latest Topics
Latest Posts
Latest Comments
Top Tags
Topics Feed
Social
Meet People
Forums
Top Members
Community Functions
DaniWeb Premium
Newsletter Archive
Markdown Syntax
Community Rules
Developer APIs
Connect API
Forum API Docs
Tools
SEO Backlink Checker
Legal
Terms of Service
Privacy Policy
FAQ
About Us
Advertise
Contact Us
© 2024 DaniWeb® LLC