When I started this blog 7 years ago, webmasters don’t really need to put much effort into making the website friendly for the search engine robot. Having a lot of unique content is all it takes but since last year, Google is focusing more on quality content and pages that are useless should not be indexed using the noindex meta tag. Since this blog started as a personal blog, a lot of the earlier articles were very thin in content and without quality. Spotting them is not really that difficult because this blog only has nearly 2000 articles after 7 years.
The biggest problem is actually the forum itself because it generates tons of index pages that refers to the threads created by members. I was able to implement a code at the header that automatically inserts a noindex meta tag for those pages but I still need to remove the URLs from Google search engine via Webmaster Tools.
As you can see from the screenshot below, these are the index pages for a forum category and they basically have the same description with links to threads and without any content. They are useless and should be removed from Google search index.
The forum currently has over 20,000 threads and by default the forum shows 20 threads per page. This would mean that there are at least 1000 index pages in the forum. I don’t to spend hours sitting on the computer copying and pasting the links to Google Webmaster Tools so I’ve figured out an automated solution.
What I need to do is create a list of index page URLs and save them in a text editor such as Notepad++. Then use a macro software to automatically perform the copying and pasting of the links to Google Webmaster Tools. After analyzing the index link, I simply need to generate number that are ordered in between index and .html at the URL for example, www.raymond.cc/forum/chat/index49.html. I would have to start with the number 2 because I want the first page to be indexed and I will have to refer to the forum to get the last page number.
After searching around, I found a website called TextMechanic that offers a free online service to generate number that are sequential with the ability to add prefixes before and after the numbers. All I need to do is enter the low and high number plus the prefixes and click on the Generate List of Numbers button which will instantly generate the list of URLs with numbers in sequence in between the prefixes.
I copied the list, saved it to a text file and continued generating more lists by referring to the forum category. Once I have the full list of URLs that I want to remove from the Google index, I used a macro software called TinyTask to record the action so that it can automatically perform the removal for me while I spend the precious time on doing something that is more meaningful.
TextMechanic is able to generate number based on other rules because it is an online text manipulation tool. Click on the Open Tool Menu button to check out other text manipulation functions.
Generate Number that are Sequential or Ordered at TextMechanic is a post from: Raymond.CC Blog