As far as I know, the exclusion standard for robots is a way to inform the web robot about which areas of the website may not be processed or scanned. So I'm hiking, why not use a similar but "inclusion" standard? The website has a robot file that specifies which parts of the website can be accessed. Everything that is not included is not available to the web robot by default. Why is everything available to the web robot by default?
Tag: Robotstxt
Google Search Console error: Indexed but blocked by robots.txt
I have used shared hosting in the past and everything was fine. But when I switched to a VPS (Digital Ocean) Google search panel, the following error appears: "I … | Read the rest of https://www.webhostingtalk.com/showthread.php?t=1786978&goto=newpost
According to Google's URL check, my image URL is blocked by robots.txt – I do not even have one!
I've just noticed that our image system domain has long ceased to be crawled by Google.
The reason for this is that apparently all URLs are blocked by robots.txt – but I do not even have one.
Disclaimer: Due to some configuration checks, I now have a generic Allow-Allow-Robots file in the root of the site. I had none before this hour.
We operate a system for resizing images in a subdomain of our website.
I get a very strange behavior because the search console claims to be blocked robots.txtalthough I have none at all.
All the URLs in this subdomain give me this result when I test it live:
While trying to fix the problem, I created a robots.txt in the root directory:
The robot file is even visible in the search results:
The response headers also seem to be okay:
HTTP/2 200
date: Sun, 27 Oct 2019 02:22:49 GMT
content-type: image/jpeg
set-cookie: __cfduid=d348a8xxxx; expires=Mon, 26-Oct-20 02:22:49 GMT; path=/; domain=.legiaodosherois.com.br; HttpOnly; Secure
access-control-allow-origin: *
cache-control: public, max-age=31536000
via: 1.1 vegur
cf-cache-status: HIT
age: 1233
expires: Mon, 26 Oct 2020 02:22:49 GMT
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 52c134xxx-IAD
Here are some sample URLs for testing:
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_zg1YXWVbJwFkxT_ZQR534L90lnm8d2IsjPUGruhqAe.png.jpeg
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_FPutcVi19O8wWo70IZEAkrY3HJfK562panvxblm4SL.png.jpeg
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/09/legiao_gTnwjab0Cz4tp5X8NOmLiWSGEMH29Bq7ZdhVPlUcFu.png.jpeg
What should I do?
Google says my URL is blocked by robots.txt – I do not even have one!
I've just noticed that our image system domain has long ceased to be crawled by Google.
The reason for this is that apparently all URLs are blocked by robots.txt – but I do not even have one.
Disclaimer: Due to some configuration checks, I now have a generic Allow-Allow-Robots file in the root of the site. I had none before this hour.
We operate a system for resizing images in a subdomain of our website.
I get a very strange behavior because the search console claims to be blocked robots.txtalthough I have none at all.
All the URLs in this subdomain give me this result when I test it live:
While trying to fix the problem, I created a robots.txt in the root directory:
The robot file is even visible in the search results:
The response headers also seem to be okay:
HTTP/2 200
date: Sun, 27 Oct 2019 02:22:49 GMT
content-type: image/jpeg
set-cookie: __cfduid=d348a8xxxx; expires=Mon, 26-Oct-20 02:22:49 GMT; path=/; domain=.legiaodosherois.com.br; HttpOnly; Secure
access-control-allow-origin: *
cache-control: public, max-age=31536000
via: 1.1 vegur
cf-cache-status: HIT
age: 1233
expires: Mon, 26 Oct 2020 02:22:49 GMT
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 52c134xxx-IAD
Here are some sample URLs for testing:
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_zg1YXWVbJwFkxT_ZQR534L90lnm8d2IsjPUGruhqAe.png.jpeg
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/10/legiao_FPutcVi19O8wWo70IZEAkrY3HJfK562panvxblm4SL.png.jpeg
https://kanto.legiaodosherois.com.br/w760-h398-gnw-cfill-q80/wp-content/uploads/2019/09/legiao_gTnwjab0Cz4tp5X8NOmLiWSGEMH29Bq7ZdhVPlUcFu.png.jpeg
What should I do?
What's better – Meta Robot Tags or robots.txt? – SEO help (general chat)
Meta is superior. To exclude individual pages from search engine indices, the no index meta day is actually superior robot,TXT, You want to use & # 39;noindex, follow & # 39; in a Robots meta tag, rather than robot,TXT because it lets the connective juice pass. It is better from an SEO perspective
Why does Google index our robots.txt file and display it in the search results?
For some reason, Google indexes the robots.txt file for some of our sites and displays them in the search results. See screenshots below.
Our robots.txt file is not linked from any point on the site and contains only the following:
User-agent: *
Crawl-delay: 5
This only happens for some websites. Why is this happening and how do we stop it?
Screenshot 1: Google search console
Screenshot 2: Google search results
Why does Google index our robots.txt file and display it in the search results?
For some reason, Google indexes the robots.txt file for some of our sites and displays them in the search results. See screenshots below.
Our robots.txt file is not linked from any point on the site and contains only the following:
User Agent: *
Crawl Delay: 5
This only happens for some websites. Why is this happening and how do we stop it?
![[IMG]](https://forums.digitalpoint.com/proxy/ZFcxSj6xV4BoYdSW9HKuY1FkGWBsylPGS2dnnuN1ECngSIfEzL4xKHr6dCjG%2BKdMsdj3CO7d5HnnAo2%2FqdmLkAJ2mLOCWnKsuMpT/image.png)
Screenshot 1: Google search console
![[IMG] "data-url =" https://i.stack.imgur.com/V2UaU.png](https://forums.digitalpoint.com/proxy/ub7g1YQbXNJ82ZQnx4ibdtEGwx8p6zxAnigb1eURmgSKQRPc%2F1ywS4rjEa9o2FsvxccvoLPAtpRU8CwQd%2BUDwf%2BiuvfAun5d94XI/image.png)
Screenshot 2: Google search results
What is not allowed in the robots.txt file?
What is not allowed in the robots.txt file?
cms – Refers to a variable on the domain of a MediaWiki site in robots.txt
I have a website (created with MediaWiki 1.33.0 CMS) that contains a robots.txt File.
In this file is a line that contains the literal domain of this site:
Sitemap: https://example.com/sitemap/sitemap-index-example.com.xml
I usually prefer to replace literal domain references with a Variable Value Call this is somehow changed (depending on the specific case) in the execution to the value that is the domain itself.
An example of a VVC would be a Bash variable substitution.
Many CMS have a global directive file that usually contains the base address of the website:
In MediaWiki 1.33.0 this file is LocalSettings.php which contains the base address in line 32:
$wgServer = "https://example.com";
How could I call this value with a variable value robots.txt?
This helps me to avoid confusion and malfunction when the domain of the website is changed. There I would not have to change the value manually.
cms – Use a variable to refer to the domain of a website in robots.txt
I have a website (created with MediaWiki 1.33.0 CMS) that contains a robots.txt File.
In this file is a line that contains the literal domain of this site:
Sitemap: https://example.com/sitemap/sitemap-index-example.com.xml
I usually prefer to replace literal domain references with a Variable Value Call this is somehow changed (depending on the specific case) in the execution to the value that is the domain itself.
An example of a VVC would be a Bash variable substitution.
Many CMS have a global directive file that usually contains the base address of the website:
In MediaWiki 1.33.0 this file is LocalSettings.php which contains the base address in line 32:
$wgServer = "https://example.com";
How could I call this value with a variable value robots.txt?
This helps me to avoid confusion and malfunction when the domain of the website is changed. There I would not have to change the value manually.
