I have a weather forecast site that offers weather widgets to other sites.
The widget is an iframe that pulls data from, let’s say, mysite.com/widget/yourforecast.php
Search engines follow the link and land on yourforecast.php page (which is not cached) causing not cached requests and fatiguing the server.
Can I disallow /widget/ from robots.txt without this penalizing me?
Doesn’t Google like robots.txt that inhibit access to scripts?
.(tagsToTranslate)webmaster forum(t)internet marketing(t)search engine optimization(t)web designing(t)seo(t)ppc(t)affiliate marketing(t)search engine marketing(t)web hosting(t)domain name(t)social media