Messing around in your server logs is illuminating. Here’s one that you might want to know about.
Google uses Feedfetcher to get RSS and Atom feeds for Google Play Newsstand and PubSubHubbub. Feedfetcher saves and refreshes feeds that are requested by app or service users, however it does not index the feeds in Google’s search services. If Googlebot scans a feed, it will appear in our search results. Here are some from Google Developers of the most frequently asked questions about the operation of this user-controlled feed grabber.
How do I request that Google not retrieve some or all of my site’s feeds?
When customers add a service or app that uses Feedfetcher data, Google’s Feedfetcher tries to get the feed’s content so it may be shown. Feedfetcher does not meet robots.txt requirements because its requests are made explicitly by human users rather than by automated crawlers.
Google won’t be able to prevent users from reading your feed if it’s public. Configure your site to serve a 404, 410, or other error status message to the Feedfetcher-Google user agent as one approach.
If your feed is hosted by a blog or website hosting service, please contact them directly to limit access to your feed.
How often does Feedfetcher come to retrieve my feeds?
On average, Feedfetcher should not retrieve feeds from most sites more than once every hour. Some sites that are often updated may be refreshed more frequently. However, due to network latency, Feedfetcher may appear to get your feeds more regularly for a short period of time.
Why is Feedfetcher trying to download incorrect links from my server, or from a domain that doesn’t exist?
Feedfetcher returns feeds on behalf of services or apps that users have installed. It’s possible that a user typed in a URL for a feed that doesn’t exist.
Why isn’t Feedfetcher obeying the robots. txt file on my site?
Only when users have started a service or app that seeks data from the feed can Feedfetcher receive feeds. Because Feedfetcher acts as a human user’s direct agent rather than a robot, it ignores robots.txt entries. Feedfetcher saves bandwidth by making requests for common feeds only once for all users that requested the feed through an app or service, because it acts as an agent for numerous users. RSS and Atom are two popular feed formats.
By configuring your server to provide a 404, 410, or other error status message to the Feedfetcher-Google user agent, you can prohibit Feedfetcher from crawling your site.
Why are there visits from multiple machines at Google. com, all with the user-agent Feedfetcher?
Only when users have started a service or app that seeks data from the feed can Feedfetcher receive feeds. Because Feedfetcher acts as a human user’s direct agent rather than a robot, it ignores robots.txt entries. Feedfetcher saves bandwidth by making requests for common feeds only once for all users that requested the feed through an app or service, because it acts as an agent for numerous users. RSS and Atom are two popular feed formats.
By configuring your server to provide a 404, 410, or other error status message to the Feedfetcher-Google user agent, you can prohibit Feedfetcher from crawling your site.
What is the IP addresses from which Feedfetcher makes requests so that I can filter my server logs?
Feedfetcher’s IP addresses change on a regular basis. The best approach to spot Feedfetcher requests is to look for the Feedfetcher-Google user agent.
Why is Feedfetcher downloading the same page on my site multiple times?
During a feed retrieval, Feedfetcher generally only downloads one copy of each file from your site. The computers are occasionally stopped and restarted, which may lead it to fetch pages it has recently visited again.
What kinds of links does Feedfetcher normally follow?
Feedfetcher, unlike traditional web crawlers, does not follow links; instead, it responds to requests made by users of a Feedfetcher-enabled service or app.
If you have questions not addressed here – you can there is a Search Central Forum Community. You can find them here https://support.google.com/webmasters/community