Google’s John Mueller has confirmed in a Webmaster Hangout that they no longer crawl content behind JavaScript “load more” buttons. Also, Mueller confirmed that they might not crawl all content loaded via the infinite scroll technique either.
While pagination is the most popular way to manage content on home, category, and category product pages, two other methods have seen increased usage over the past few years:
- JavaScript “load more” buttons - This is where, instead of pagination, you dynamically load additional content to the page via a “load more” button at the end of the page.
- Infinite Scrolling - This is where more content automatically loads as you scroll to the bottom of the page.
I suspect the increased usage is due, at least in part, to Google announcing its ability to understand websites better by executing Javascript in 2014 and crawling AJAX in 2017.
Surprisingly, news of this change has not been better disseminated to the SEO and Developer communities.
In more detail, let’s look at what John Mueller had to say about this.
Google can’t crawl content behind JavaScript “Load More” buttons
Mueller has admitted that in the past, they would try and render content behind “load more” buttons, which adds to the confusion.
But, in a clear statement, Mueller said that because the cost of rendering “load more” content is so expensive, they no longer do it.
I in the past, we would try to figure this out and try to trigger some of these to see what would happen, but that’s extremely expensive when it comes to rendering the whole web, so I believe we don’t do that at all.
Later in the statement, Mueller confirmed it again:
So if you’re using a load more button with JavaScript, then that’s something we probably wouldn’t use.
Google MAY NOT crawl content loaded via infinite scroll
The position with infinite scroll is a little more complicated.
Google will render your page on a very long viewport; if the infinite scrolling increases the viewport, they will crawl that new content.
But they will only do it one time.
It probably won’t be SEO-friendly if you have a lot of content that needs to load via infinite scrolling.
Here is what Mueller had to say:
We do something we call frame expansion, which is we take the page that you serve, and we try to render it on an extremely long viewport, and we see if the page does anything like infinite loading where it expands the viewport.
That’s something where we would expand that new port once and see what all loads and try to index all of that, but we would not do this multiple times in a row.
Especially when it comes to infinite scrolling like the name says some websites actually do have pretty much an infinite scrolling where you could expand this a whole bunch of times, and you wouldn’t necessarily find new content you just find kind of repetitive sections across the whole website
Mueller continued that if you do intend to use infinite scrolling anyway, then you should:
Double check our documentation to make sure that you’re implementing it in a way that makes that works well for Google for indexing
You can view the whole discussion from the Webmaster Central Office-Hours Hangout below: