Website vital signs: watch your indexed pages

It's getting more complex to manage your own websites search engine performance, as you know. It used to be that you could create great content and then you let the shares and links happen naturally. That's what my impression of what Google wanted was. However, we have discovered this isn't the case.

We use Moz and Google Analytics to monitor traffic and rankings. But we didn't use Webmaster tools so much. Big mistake. For a while we saw that we had a large number of duplicate page titles and too many internal links. We couldn't figure out why. This went on for months. Then one day when a crawl was happening our agency asked us what was going on as they had to put on two extra servers to cope with the load. 

Duplicate disaster

Unfortunately, this wasn't a site visit spike, but a simple crawl to gather SEO data. Then the penny dropped. We have multiple pages. The answer was staring us in the face! We discovered that in Google Webmaster tools you can track the number of pages Google indexing…. and it was showing over 100,000 for our site! 

We had between 1,000 to 2,000 actual pages on our site, but Google was seeing over 100,000. How could this be? This was obviously a technical problem and after looking into the reasons why, we pin pointed it down to blog and event blocks that list content. Whenever the block appeared, it created new and original URLs for the content they were pointing to - on every page. So on every page we had a blog block it would be secretly creating unto 800 new URLs - and creating our duplicate content problem.

We deleted a number of old blogs (you need to set up 301 redirects afterwards). And added new do not follow links into our robots text, to try and fix the issue. We've dropped to around 60,000 in eight weeks. This is a slow process. Really slow.

The learning is: always check your indexed pages on Google Webmaster tools. It's one of the many vital signs you need to monitor.