Tag: Robotstxt

Why does Google index our robots.txt file and display it in the search results?

For some reason, Google indexes the robots.txt file for some of our sites and displays them in the search results. See screenshots below.

Our robots.txt file is not linked from any point on the site and contains only the following:

User-agent: *
Crawl-delay: 5

This only happens for some websites. Why is this happening and how do we stop it?

Enter image description here

Screenshot 1: Google search console

Enter image description here

Screenshot 2: Google search results

Why does Google index our robots.txt file and display it in the search results?

For some reason, Google indexes the robots.txt file for some of our sites and displays them in the search results. See screenshots below.

Our robots.txt file is not linked from any point on the site and contains only the following:

User Agent: *
Crawl Delay: 5

This only happens for some websites. Why is this happening and how do we stop it?

[IMG]

Screenshot 1: Google search console
SEMrush

[IMG] "data-url =" https://i.stack.imgur.com/V2UaU.png

Screenshot 2: Google search results

cms – Refers to a variable on the domain of a MediaWiki site in robots.txt

I have a website (created with MediaWiki 1.33.0 CMS) that contains a robots.txt File.
In this file is a line that contains the literal domain of this site:

Sitemap: https://example.com/sitemap/sitemap-index-example.com.xml

I usually prefer to replace literal domain references with a Variable Value Call this is somehow changed (depending on the specific case) in the execution to the value that is the domain itself.

An example of a VVC would be a Bash variable substitution.

Many CMS have a global directive file that usually contains the base address of the website:
In MediaWiki 1.33.0 this file is LocalSettings.php which contains the base address in line 32:

$wgServer = "https://example.com";

How could I call this value with a variable value robots.txt?
This helps me to avoid confusion and malfunction when the domain of the website is changed. There I would not have to change the value manually.

cms – Use a variable to refer to the domain of a website in robots.txt

I have a website (created with MediaWiki 1.33.0 CMS) that contains a robots.txt File.
In this file is a line that contains the literal domain of this site:

Sitemap: https://example.com/sitemap/sitemap-index-example.com.xml

I usually prefer to replace literal domain references with a Variable Value Call this is somehow changed (depending on the specific case) in the execution to the value that is the domain itself.

An example of a VVC would be a Bash variable substitution.

Many CMS have a global directive file that usually contains the base address of the website:
In MediaWiki 1.33.0 this file is LocalSettings.php which contains the base address in line 32:

$wgServer = "https://example.com";

How could I call this value with a variable value robots.txt?
This helps me to avoid confusion and malfunction when the domain of the website is changed. There I would not have to change the value manually.

Search engines – Is the order of the lines "Disallow" and "Sitemap" in robots.txt important?

You can sort robots.txt this way:

User-agent: DESIRED_INPUT
Sitemap: https://example.com/sitemap-index.xml
Disallow: /

instead:

User-agent: DESIRED_INPUT
Disallow: /
Sitemap: https://example.com/sitemap-index.xml

I assume that both are okay, as it is likely that the file will generally be compiled by all crawlers in the correct order.
Is it a proven method to set Disallow: In front Sitemap: to avoid a highly improbable bug in the bad compilation of the crawler before it is ignored Disallow:?

Search engines – robots.txt execution

You can sort robots.txt this way:

User-agent: DESIRED_INPUT
Sitemap: https://example.com/sitemap-index.xml
Disallow: /

instead:

User-agent: DESIRED_INPUT
Disallow: /
Sitemap: https://example.com/sitemap-index.xml

I assume that both are okay, as it is likely that the file will generally be compiled by all crawlers in the correct order.
Is it a proven method to set Disallow: In front Sitemap: to avoid a highly improbable bug in the bad compilation of the crawler before it is ignored Disallow:?

How do I delete the robots.txt file from a site using the WordPress Elementor page builder with the Astra theme?

I went to my public HTML code and did not find the robots.txt file. Can you tell me which folder it is in? I have to pinpoint it. Much no access.

I have already disabled the "Search engine visibility (keep search engines from indexing this site)" setting. I have easily removed robots.txt files from HTML's java script.

Clear instructions are required. Using Elementor 2.6.8 with Astra Design.