The Google Search Console creates URLs that are not in my sitemap, and then complains that these pages are broken

You have a wrong idea of ​​what a sitemap is.

The sitemap is used to monitor the crawling of the site by the search engine bot. The sitemap and crawling a site are two different and independent things. Google continues to search your site independently of the sitemap. The sitemap is used to check if Google can properly crawl your site. For example, if pages are found in your sitemap and Google has not seen the page, Google may add the page to the fetch queue to include.

The opposite is not true. If a page is not found in the sitemap, Google will not remove it from the index. Why? Because Google found it by crawling the site.

What you seem to believe is that the sitemap is the absolute authority Google uses to identify which pages exist on a particular site. That's not the case. The crawl is. The sitemap only helps Google to know if You can properly crawl your site, and if not, what pages Google is missing that should be added to the fetch queue.

Your expectation that Google will stop trying to access pages because these pages are no longer in your sitemap is wrong. Sitemaps are cached and checked at regular intervals. Why? Because it is a testing process.

You have a real problem that you need to solve.

You return a 500 error for pages that were not found. This is bad. Your website should return a 404 Not Found error. The 500 error is a system error and is treated as temporary by Google. If your site returns a 404 error, Google will still try the page a few times over a period of time, until it finds that the page no longer exists. If possible, you want to output a 410 Removed error for pages that you have removed. If this is too much work or not possible, the 404 will mean the same over time.

You must fix your 500 error.

Google Search Console creates URLs that do not exist in my Sitemap, and then complains that these pages are broken

You have a wrong idea of ​​what a sitemap is.

The sitemap is used to monitor the crawling of the site by the search engine bot. The sitemap and crawling a site are two different and independent things. Google continues to search your site independently of the sitemap. The sitemap is used to check if Google can properly crawl your site. For example, if pages are found in your sitemap and Google has not seen the page, Google may add the page to the fetch queue to include.

The opposite is not true. If a page is not found in the sitemap, Google will not remove it from the index. Why? Because Google found it by crawling the site.

What you seem to believe is that the sitemap is the absolute authority Google uses to identify which pages exist on a particular site. That's not the case. The crawl is. The sitemap only helps Google to know if You can properly crawl your site, and if not, what pages Google is missing that should be added to the fetch queue.

Your expectation that Google will stop trying to access pages because these pages are no longer in your sitemap is wrong. Sitemaps are cached and checked at regular intervals. Why? Because it is a testing process.

You have a real problem that you need to solve.

You return a 500 error for pages that were not found. This is bad. Your website should return a 404 Not Found error. The 500 error is a system error and is treated as temporary by Google. If your site returns a 404 error, Google will still try the page a few times over a period of time, until it finds that the page no longer exists. If possible, you want to output a 410 Removed error for pages that you have removed. If this is too much work or not possible, the 404 will mean the same over time.

You must fix your 500 error.

Zsh treats a string as a file name pattern and complains about it (NOMATCH option)

I try Zsh and come from Bash. I had a bit of trouble understanding why Zsh complained about a regular expression.

usr@rk1 ~ % tty | grep ^/dev/tty(1-7)$ > /dev/null 2>&1
zsh: no matches found: ^/dev/tty(1-7)$

I've found that the output disappears and everything is fine if (1) I put the regex in quotation marks, (2) remove the slashes from the regex, or (3) use setopt NO_NOMATCH,

From the zshoptions manual:

NOMATCH (+3)

If a pattern for filename generation does not match, print an error instead of leaving it in the argument list. This also applies to the file extension of an initial & # 39; & # 39; or & # 39; = & # 39 ;.

It seems to me that the regular expression is treated as a filename pattern due to the slashes. Is this a bug?

Where can I find the Zsh version of the GreyCat Bashisms page?

seo – Alexa Rank complains about unsafe generator meta tag

The Alexa Rank auditing tool complains about a generator meta tag that complies with the HTML5 standard.

 

The Alexa Site Audit Report complains about the uncertainty of this tag:

What is this topic about?
HTML tags. on your pages, which contain information about the software used to create your website.

Why is that important?
Unsafe meta tags on your site contain information about the underlying software (such as WordPress) that can help someone attack or endanger your site.

I have read comments from here and here, but it does not indicate why using the generator meta tag can be unsafe (it states that it may be useless, but not uncertain).

8 – Site works well, but the composer complains about unmet needs. Do I have a problem?

I'm trying to get my permissions right so I do not need to run Composer as root. As part of this process, I deleted Vendor, Core, and Composer.lock and ran the Composer installation with my non-root user. The site seems to work fine even though this message composer appears:

        Problem 1
- drupal / core 8.2.x-dev requires symfony / psr-http-message-bridge v0.2 -> satisfying by symfony / psr-http-message-bridge[v0.2] However, these conflict with your requirements or minimum stability.
- do not install drupal / core-render 8.2.0 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.0-beta1 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.0-beta2 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.0-beta3 | Remove drupal / drupal dev-master
- do not install drupal / core render 8.2.0-rc1 | Remove drupal / drupal dev-master
- do not install drupal / core render 8.2.0-rc2 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.1 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.2 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.3 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.4 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.5 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.6 | Remove drupal / drupal dev-master
- do not install drupal / core-render 8.2.7 | Remove drupal / drupal dev-master

and so on

What does that mean? Is my website okay or did I take a wrong train?

I am not sure why the composer's message refers to "drupal / core 8.2.x-dev". Here is the "Requirement" section of my composer.json:

"need": {
"Composer / Installer": "^ 1.0.24",
"wikimedia / composer-merge-plugin": "^ 1.4",
"drupal / entity_clone": "^1.0@beta",
"drupal / features": "^ 3.8",
"drupal / metatag": "^ 1.8",
"drupal / webform": "^ 5.1",
"drupal / config_installer": "^ 1.8",
"drupal / entity_browser": "^ 2.1",
"drupal / media_entity_browser": "^1.0@beta",
"drupal / entity_embed": "^1.0@beta",
"drupal / file_browser": "^ 1.1",
"drupal / admin_menu_search": "^ 1.0",
"drupal / admin_toolbar": "^ 1.26",
"drupal / backup_migrate": "^ 4.0",
"drupal / block_field": "^1.0@alpha",
"drupal / contact_block": "^ 1.4",
"drupal / contrib": "^5.0@beta",
"drupal / ctools": "^ 3.2",
"drupal / entity_reference_revisions": "^ 1.6",
"drupal / facets": "^ 1.3",
"drupal / field_group": "^ 1.0",
"drupal / form_placeholder": "^ 1.0",
"drupal / formblock": "^1.0@beta",
"drupal / glazed_helper": "^ 1.3",
"drupal / google_analytics": "^ 3.0",
"Drupal / Honeypot": "^ 1.29",
"drupal / imce": "^ 1.7",
"drupal / insert_block": "1.x-dev",
"drupal / linkit": "^ 4.3",
"drupal / login_emailusername": "^ 1.1",
"drupal / material_admin": "^1.0@alpha",
"drupal / menu_link_attributes": "^ 1.0",
"drupal / minifyhtml": "^ 1.6",
"Drupal / paragraphs": "^ 1.6",
"drupal / pathauto": "^ 1.3",
"drupal / redirect": "^ 1.3",
"drupal / requir_login": "^ 2.0",
"drupal / search_api": "^ 1.11",
"drupal / simple_sitemap": "^ 3.0",
"drupal / simple news": "^1.0@alpha",
"drupal / tfa": "^1.0@alpha",
"drupal / token": "^ 1.5",
"drupal / video_embed_field": "^ 2.0",
"drupal / view_unpublished": "^1.0@alpha",
"drupal / viewport": "^ 1.1",
"drupal / viewsreference": "^ 1.4",
"drupal / workflow_state_config": "^1.0@alpha",
"drupal / xmlsitemap": "^1.0@alpha",
"drupal / Allowed_formats": "^ 1.1",
"drupal / asset_injector": "^ 2.4",
"drupal / taxonomy_access_fix": "^ 2.6",
"drupal / text_summary_options": "^ 1.0",
"drupal / migrate_source_csv": "^ 2.2",
"drupal / migrate_plus": "^ 4.1",
"drupal / migrate_tools": "^ 4.1",
"drupal / migrate_file": "^ 1.1",
"drupal / entityqueue": "^1.0@alpha",
"drupal / module_missing_message_fixer": "^1.0@beta",
"drupal / rules": "^3.0@alpha",
"ckeditor / autogrow": "^ 4.8",
"ckeditor / codemirror": "^ 1.17",
"ckeditor / fakeobjects": "^ 4.8",
"ckeditor / image": "^ 4.8",
"ckeditor / link": "^ 4.8",
"codemirror / codemirror": "^ 5.36",
"jquery / geocomplete": "^ 1.7",
"jquery / icheck": "^ 1.0",
"jquery / image picker": "^ 0.3.0",
"jquery / inputmask": "^ 3.3",
"jquery / intl-tel-input": "^ 12.1",
"jquery / rateit": "^ 1.1",
"jquery / select2": "^ 4.0",
"jquery / timepicker": "^ 1.11",
"jquery / toggles": "^ 4.0",
"jquery / word-and-character-counter": "^ 2.5",
"progress-tracker / progress-tracker": "^ 1.4",
"signature_pad / signature_pad": "^ 2.3",
"drupal / image_widget_crop": "^ 2.2",
"drupal / crop": "^ 1.5",
"drupal / advagg": "^ 3.5",
"drupal / advagg_js_minify": "^ 3.5",
"drupal / advagg_css_minify": "^ 3.5",
"drupal / better_exposed_filters": "^3.0@alpha",
"drupal / block_visibility_groups_admin": "^ 1.3",
"drupal / elected": "^ 2.6",
"drupal / chosen_field": "^ 2.6",
"drupal / content_export_csv": "^3.0@beta",
"drupal / contentimport": "^ 4.1",
"drupal / event": "1.x-dev",
"drupal / entity_reference_views_select": "^ 1.3",
"drupal / imce_search_plugin": "^ 1.0",
"drupal / inline_entity_form": "^1.0@RC",
"drupal / job": "^3.0@alpha",
"drupal / media_bulk_upload": "^1.0@alpha",
"drupal / memcache": "^ 2.0",
"drupal / purge_ui": "^3.0@beta",
"drupal / purge_processor_lateruntime": "^3.0@beta",
"drupal / purge_tokens": "^3.0@beta",
"drupal / twig_tweak": "^ 2.1",
"drupal / views_bulk_operations": "^ 2.5",
"drupal / block_visibility_groups": "^ 1.3",
"drush / drush": "^ 9.5",
"drupal / we_megamenu": "^ 1.5",
"drupal / views_infinite_scroll": "^ 1.5",
"drupal / embed": "^ 1.0",
"drupal / dropzonejs": "^2.0@alpha",
"drupal / purge": "^3.0@beta",
"drupal / address": "^ 1.4",
"drupal / acquia_connector": "^ 1.16",
"drupal / purge_purger_http": "^1.0@beta",
"drupal / smtp": "^1.0@beta",
"drupal / content_sync": "^ 2.1",
"drupal / console": "~ 1.0",
"drupal / devel": "^ 2.0",
"drupal / core": "8.6.13"
},

seo – The Google Webmaster tool complains about a blocked URL. This is not the case

We recently integrated with Google Webmaster Tools.

On one of the pages, Google complains:
Indexed, but blocked by robots.txt

Our robots.txt looks like this:

User Agent: *
Do not allow: / wp-admin /
Allow: /wp-admin/admin-ajax.php


Sitemap: http: ///sitemap.xml.gz

If I understand how it works, the rest of the URL (except / wp-admin /) should not be blocked.

Why am I getting this warning?