Google’s Feed Fetcher Bot

Messing around in your server logs is illuminating. Here’s one that you might want to know about.

Google uses Feedfetcher to get RSS and Atom feeds for Google Play Newsstand and PubSubHubbub. Feedfetcher saves and refreshes feeds that are requested by app or service users, however it does not index the feeds in Google’s search services. If Googlebot scans a feed, it will appear in our search results. Here are some from Google Developers of the most frequently asked questions about the operation of this user-controlled feed grabber.

When customers add a service or app that uses Feedfetcher data, Google’s Feedfetcher tries to get the feed’s content so it may be shown. Feedfetcher does not meet robots.txt requirements because its requests are made explicitly by human users rather than by automated crawlers.

Google won’t be able to prevent users from reading your feed if it’s public. Configure your site to serve a 404, 410, or other error status message to the Feedfetcher-Google user agent as one approach.

If your feed is hosted by a blog or website hosting service, please contact them directly to limit access to your feed.

On average, Feedfetcher should not retrieve feeds from most sites more than once every hour. Some sites that are often updated may be refreshed more frequently. However, due to network latency, Feedfetcher may appear to get your feeds more regularly for a short period of time.

Feedfetcher returns feeds on behalf of services or apps that users have installed. It’s possible that a user typed in a URL for a feed that doesn’t exist.

Only when users have started a service or app that seeks data from the feed can Feedfetcher receive feeds. Because Feedfetcher acts as a human user’s direct agent rather than a robot, it ignores robots.txt entries. Feedfetcher saves bandwidth by making requests for common feeds only once for all users that requested the feed through an app or service, because it acts as an agent for numerous users. RSS and Atom are two popular feed formats.

By configuring your server to provide a 404, 410, or other error status message to the Feedfetcher-Google user agent, you can prohibit Feedfetcher from crawling your site.

Only when users have started a service or app that seeks data from the feed can Feedfetcher receive feeds. Because Feedfetcher acts as a human user’s direct agent rather than a robot, it ignores robots.txt entries. Feedfetcher saves bandwidth by making requests for common feeds only once for all users that requested the feed through an app or service, because it acts as an agent for numerous users. RSS and Atom are two popular feed formats.

By configuring your server to provide a 404, 410, or other error status message to the Feedfetcher-Google user agent, you can prohibit Feedfetcher from crawling your site.

Feedfetcher’s IP addresses change on a regular basis. The best approach to spot Feedfetcher requests is to look for the Feedfetcher-Google user agent.

During a feed retrieval, Feedfetcher generally only downloads one copy of each file from your site. The computers are occasionally stopped and restarted, which may lead it to fetch pages it has recently visited again.

Feedfetcher, unlike traditional web crawlers, does not follow links; instead, it responds to requests made by users of a Feedfetcher-enabled service or app.

If you have questions not addressed here – you can there is a Search Central Forum Community. You can find them here https://support.google.com/webmasters/community

Leave a Reply