SEO – Google could not retrieve the sitemap

Sitemaps are a necessary part of our web work. And we need them to love Google more. But for about a month, I can not submit sitemaps to the Google search console.

I've used https://technicalseo.com/tools/fetch-render/ to validate and successfully pass. I used the sitemap validator and this one did not show any errors. It seems that something from the search console has to be wrong.

Yesterday I started another site and used the same WordPress plugin to generate the sitemap. Then I used the same Gmail ID to sign in to the search console and tried to add a property, and this time it worked.

If I submit a new Sitemap that could not be retrieved, and I try to resubmit an existing Sitemap, it succeeds, but the last reading date is 6 months ago, so I assume that it will not work.

What are possible solutions. The live test says it can not retrieve because it is not an index. I think there is nothing wrong with keeping sitemaps without an index, as sitemaps should not be displayed in the search. Infact my second domain also got sitemaps with noindex

I use WordPress PHP 7.3 and HTTP / 2

My sitemap URL is https://milyin.com/sitemap.xml

My HTTP headers are:

HTTP/2 200 
server: nginx
date: Thu, 22 Aug 2019 12:23:21 GMT
content-type: text/xml; charset=UTF-8
vary: Accept-Encoding
x-robots-tag: noindex, follow
link: ; rel="https://api.w.org/"
last-modified: Wed, 21 Aug 2019 14:03:48 GMT
etag: W/"59184deaadc3de11f553f5a8fbaac7f0"
x-httpd: 1
x-robots-tag: index, follow
host-header: 192fc2e7e50945beb8231a492d6a8024
x-proxy-cache: MISS
content-encoding: gzip

SEO – How do I change the structure of my URLs along with the sitemap for a live site?

I change the URL of my website to improve the SEO. The current sitemap contains URLs like these:

mysite/browse/1    /* 1 is the ID for fashion */
mysite/browse/2    /* 2 is the ID for real estate */

Now I changed the URLs to the following format:

mysite/browse/fashion
mysite/browse/real-estate

I have the code ready for publication along with the new sitemap. When I release the new code, the old URLs are no longer valid, ie: mysite/browse/2 would return "404 – page not found".

I think I need to share the code and send the new sitemap to Google Search Console. However, I think it would take some time for Google to crawl the new sitemap. During this time, all my old URLs in the search result returned the error "404 – page not found".

How can I mitigate this situation?

SEO – Will a Sitemap ensure that pages serviced by AJAX queries are crawled?

I have a website where I publish articles that I started just 2 weeks ago. I'd like to keep the pages as clean as possible and load more content (links to other articles) from AJAX requests to user action (for now, clicks). I read a bit. Most of the articles and blog posts on this topic were outdated. I understand that Google used to support crawling AJAX requests, but not anymore. Some papers also recommend using methods that provide content by pagination. I also read about sitemaps. I know that it gives search engine crawlers an indication of which pages to search.

However, will crawlers find inconsistencies because these links are out of reach and can only be accessed by clicking the Load More button? Does a sitemap make sure the crawlers visit the URL?

seo – Hierarchical vs flat sitemap

Your question is actually divided into a few categories, each of which has different considerations:

XML Sitemap
In your Sitemap, you can specify both the frequency of change and the priority ranking for the various pages. In this way, search engines can determine how important a page is on the site and how often it should be crawled for changes. Avoid listing duplicate content sites like / fashion / women / shoes and / fashion / shoes / women. In such cases, select a URL as the canonical URL that you want to send to search engines.

URL structure
By maintaining a minimum hierarchy in your URL structure for each page or for faceted search results, a search engine can index your website and display top-level category pages. (The search result of a department store can, for example, show the different departments.)

user experience
A menu with too much depth can lead to a bad user experience, as the user may be overwhelmed by the choices or simply tired of browsing through many navigation levels. A well-structured search interface with the most common categories as predefined filters will help you avoid an overly complex menu. Similarly, users can use breadcrumbs on the page to return to the top category or subcategory pages.

Look for products in Engjne Optimization
For each product on your site, consider adding schema microdata to your markup. That way, search engines can identify the individual products that are being offered for sale on your E-Comm site. See https://schema.org/product

Also, consider adding an RSS or XML feed and sending it to multiple sources to improve the reputation of your site.

sitemap – How to tell bots to forget a site and index it from scratch

It does not work You need to map your old URLs to the new ones with redirects for SEO and user experience.

Google never forgets about old URLs, even after a decade. If you are migrating to a new CMS, you must implement page-level redirects

If there is no match for a particular page, you can allow it 404 and Google will remove it from the index. If you use "410 Gone" instead, Google deletes the URLs from the index as soon as they have been crawled without the Google-defined "404 Not Found" deadline of 24 hours.

There is no instruction that instructs bots to forget an old site in the search console or robots.txt.

What if you do not redirect?

Redirecting may be too time-consuming, or your new CMS may not simplify the implementation of the redirect.

If you do not implement the redirects, it will start from scratch. Google recognizes that your legacy URLs return the status 404 and removes them from the search index.

Your new URLs may be indexed, but it may take a while. Changing all of your redirect URLs is a big sign that your site is not stable and can not be trusted. All your placements will be lost and your website will be restarted.

Googlebot will search the old URLs for years to come. The hope is eternal that you can open these pages again someday.

If you redirect, all inbound links, users' bookmarks, and most of your current leaderboards will be preserved.

Why?

Why do search engines have no "reset" button? Because there are almost always better options. In your case, it is much better to divert.

In the event of a site being penalized, Google will not offer a reset button as it may remove all penalties.

As?

How do you implement the redirects? You need a list of your old URLs. You may have a sitemap from your old website that you can start with. You can also retrieve the list from your server logs, Google Analytics, or even from Google's search panel.

If you've planned in advance, your URLs will be similar in your new CMS and you can implement a rewrite rule to handle them. If there is a pattern between the old and the new URL, it can be a one-liner in a URL .Access File to output the redirects for the entire website.

If you have to manually search for the new URLs and assign thousands of them one after the other, you can look it up RewriteMap Functionality.

seo – GSC: Sitemap could not be retrieved

I'm trying to send a very simple sitemap (for testing only) to the Google Search Console, but I'm getting the following error message:

╔══════════════╤═════════╤══════════════╤═════════ ══╤══════════════════╤═════════════════╗
║ Sitemap │ Type │ Submitted │ Last read │ Status │ Found URLs ║
╠══════════════╪═════════╪══════════════╪═════════ ══╪══════════════════╪═════════════════╣
║ /sitemap.txt │ Unknown │ July 17, 2019 │ │ * Could not retrieve * │ 0 ║
╚══════════════╧═════════╧══════════════╧═════════ ══╧══════════════════╧═════════════════╝

If you click on it, an additional error message will be displayed: "(!) Sitemap could not be read".
However, clicking on "OPEN SITEMAP" will open it normally.

question
Any idea what's going on?


Domain: world-hello.ddns.net
Sitemap file: sitemap.txt
Server: Apache (Debian)