We wrote this article in January, thatpointed directly to Yoast SEO as causing issues with sites being not seen by Google. We later learned Google was suggesting the code that Yoast was using it to all major SEO software companies we’ve discovered. Update: Yoast requested to have this post deleted and our plugin removed that fixes this issue. Yoast SEO is one of the most popular WordPress Plugins with over 5 million sites using their software. While Yoast provides a template to fill in the meta description field, it also generates a sitemap as do many other SEO software products. Sitemaps are extremely important for Google to be able to see your site and be able to decide what it should index and what it shouldn’t index.

Yoast, All in one SEO, and many other seo products are essentially telling Google to not look at your sitemap by putting no-index in their header. This makes it much harder for Google to find links on your site.

While the list of SEO software that we’ve found containing the code Google suggests, we are happy Yoast will be fixing this in 11.7.

The SEO software companies currently include Yoast, All In One SEO & an independent plugin called Google XML sitemap Genator for WordPress, which was not made by Google.

This started around July 5th, 2018 and cost one of our clients tourslosangeles.com over 100,000 dollars and another client an undisclosed amount of money from all in one seo. We made a free fix with our plugin Airtight Security so you can continue to use these programs and Google can see your sitemap. This means Google can find your site a lot easier. All you have to do to get our plugin is go to your WordPress site and log in. Then go to your plugins and click add new. After clicking add new search for airtight security and download our plugin and activate it. It will automatically fix the issue without any intervention by you.

The rest of this article is kept intact for historical purposes with minor updates, as this article was written in January.

Does A sitemap guarantee Google Will Index My Links?

No, it does not and Moz pointed this out in an extremely honest and detailed piece on xml sitemaps they wrote. What a sitemap does is let Google see you have content, rank the content using their algorithm and decide if the content should be included on Google and how valuable it is. The reason this is so important is that serious companies will be writing content that is of value every single day and expect their sitemap to help Google find it quickly instead of having to manually go to Google Webmasters and enter each link manually.

What Happens When Google Sees my Sitemap When using one of these SEO companies?

Here are screenshots of what Google sees when we tested this using Yoast.

Is that not enough proof for you that Yoast among others are blocking your sitemap from being indexed? This is what happens when you try to index a Yoast sitemap without going to the live test view, which is what we displayed above.

Prove SEO Software Is Causing this!

While Our team initially thought the site was infected with malware, we found no malware. We finally found the code SEO products making it not possible to index the sitemap. All that is required is to remove a few lines of code to stop the problem.

When we turn on our program airtight security the no-index header created by Yoasts sitemaps are removed.

When we turn off airtight security and use a chrome extension that let’s you see http headers, you can visually see the no-index header on the sitemap.

DO Premium Versions Fix This?

Since we were not aware that Google was telling SEO companies to do this when this was first published we noticed Yoast, which is where this investigation began at  was pushing their premium version what seemed to be more than usual. At the time we thought Yoast  had possibly  patched  the  issue in their free version if you upgraded to their paid version and bought the product for analysis to learn that isn’t the case.

 Yoast did not want to remove these lines as they believe this is helpful and makes sitemaps not rank higher in the search engines. While it is true it makes sitemaps not rank higher, since you’re not indexing them at all, our data also shows all these companies, not just Yoast are making companies less visible online. So far we’ve identified All In One SEO, Yoast & Google Sitemap Generator which is a WordPress plugin.

Leafly is a website that suffered a massive depletion of users, yet magically jumped back up and are doing great. So how is this possible if they use Yoast? Because they abandoned WordPress when customers started dropping off as the chart shows in September and October. We know they used Yoast thanks to this site that tracks users of Yoast. When they stopped using yoast the no-index code was removed and they became more visible.

Cheatsheet.com has had a major decrease at the same time of all the other sites, though it is worth noting they currently use Yoast SEO Premium. It is of no surprise to us that cheatsheet.com has had a massive decrease of visitors to their site since Yoast put a no-index on the sitemap. While we could compare millions of  sites this helps give you a visual of the issue that Yoast caused. It is also important to note that around the time this issue started, someone filed a github complaint that they noticed the rss feeds were not being indexed.

Does Yoast Know About the No-index issue?

Yoast initially considered this a feature in January, not a bug or an issue as that is what Google told them, but in July announced they are releasing a fix.

Their employee jono-alderson addressed the feature. Jono said on August 26th, 2018 when this started about the RSS Feed issue that ”
From an SEO perspective, it’s generally worthwhile preventing Google from indexing RSS feeds via the x-robots HTTP header. Note for reference, that when this has a value of noindex, that doesn’t prevent Google from accessing or consuming the information – just from indexing it.
That aside, we should definitely add the ability to filter this value, so that we can be podcast-friendly. Easy fix! “

Let’s break that down into easily, consumable pieces. First they claim no-index does not stop Google from “accessing or consuming the information”.

Google and any other search engine goes to a link, checks the headers and if the header says no-index, they go away, since that is what no-index means. So, from the statement by the Yoast employee since they can access the site and be told to leave, that is fine. What we haven’t mentioned is that Yoast uses noindex, follow which is very misleading and we clear up how this messes up your site in the words of Google’s Webmaster Round Table John Mueller who is in Charge of Webmaster Trends Analyst at Google.

Let’s be very clear, they do not consume the information on your sitemap, meaning they can not use it they ignore it per Yoast’s instructions. Google explains why they ignore it in the next paragraph. Also, one person pointed out they are violating Google’s rules on RSS feeds for podcasts. 

Google who is an industry leader in SEO says the exact opposite about Yoast’s noindex, follow technique in their Google SEO round table. John Mueller who is the Webmaster Trends Analyst at Google essentially said that if you put noindex, follow they won’t index that page or follow any of the links.

John Mueller explained how Google handles the exact type of code that Yoast and other SEO companies are using in a 2017 Google webmaster round table.
“It’s tricky with noindex, which I think is something of a misconception in general within the SEO community. With a noindex and follow it’s still the case that we see the noindex. In the first step we say ‘okay you don’t want this page shown in the search results’. We’ll still keep it in our index, we just won’t show it and then we can follow those links.”

That part seems to support Yoast’s claim, but the next paragraph debunks Yoasts claim.

“If we see the noindex there for longer than we think this page really doesn’t want to be used in search so we will remove it completely. And then we won’t follow the links anyway. So noindex and follow is essentially the same as a noindex, nofollow. There’s no really big difference there in the long run. “

So, what John Mueller is saying is that if you put noindex, follow on a page for a few days they would still follow the links and add the content into Google, like Yoast claims. However, if the noindex, follow stays on the same page for say a few weeks they will ignore that page and all the links on it. So, in short Google is addressing the exact code Yoast is using months before Yoast released it. Since the sitemap made by Yoast never removes the no-index header Google now ignores the sitemap and all of it’s links. However since we’ve found this issue John Mueller is trying to say that Google processes XML differently, but the search engine results are not reflecting that statement.

This disproves everything that Yoast claims and is why your site is having so much trouble. When it comes to how search engine optimization works, I listen to data.

Joost De Valk in July has announced this will be fixed in 11.7 and has tested it to make sure it works properly.

Joost De Valk from Yoast in January commented “

I’m sorry but this just isn’t true AT ALL. XML sitemaps aren’t indexed like normal webpages. Or at least: they shouldn’t be. Google reads them differently and doesn’t obey the indexing directives when it ingests them like that. Sometimes they get linked to on the web as well. At that point, Google *does* index them normally, and follows indexing directives. So we set the noindex header on the XML sitemaps so as to make it impossible for XML sitemaps to start showing up in search results. They do *not* prevent Google from using them for what they’re important for: getting URLs into the index.

We talk regularly to Google and are in fact looking at making XML sitemaps better for everyone together with them, so I’m 100% certain of this.”

None of what he said is supported by information from Google, charts showing damage as we showed above or even from Yoast’s customers.

.