Help > Forum > Search Engines > Search engine spidering

Search engine spidering

Getting indexed

Your forum will be spidered (indexed) by search engines if it is linked to a page that the search engine can get to. For example, you should place your forum link somewhere on your website so that the search engine can follow that link and then index your forum. Alternatively, you can manually add your forum to Google's index.

How forums rank

Forums generally get very high rankings on search engines because they are filled with content all related to the same general topic. Search engines love this because it is exactly what they are looking for. Forums will typically help boost your website's ranking in the search engines as well since there is a link to your website on your forum.

Additionally, Website Toolbox forums have been developed with search engine optimization concepts in mind. For example, topic URLs contain the keywords used in the topic title, increasing the changes that a search engine will find those keywords relevant to that page.

Boosting your ranking

Be sure to customize your forum domain name to match your website. That will allow your main website to receive full credit for all the valuable content on your forum. Search engines will not be able to tell that your forum is hosted on Website Toolbox.

Search engines love content. If your forum is new, create as many helpful topics as you can. That will give the search engines something to index and will help bring people searching for that content to your forum. The topics should be relevant to the general theme of your forum and should be valuable to the reader. Toward the end of your post, encourage readers to ask questions or give their feedback so you can turn readers into posters and help your forum grow. The more useful content you post, the more search engines will refer users to your forum.

Create more quality links on the Internet pointing to your forum. When other websites link to a page, search engines considers that like a vote for the page. The links should preferably contain the keywords you are trying to get a higher ranking for.

Target long-tail keywords rather than short highly competitive keywords. For example, forum would be a highly competitive keyword that would be very difficult to rank well for. However, it would be significantly easier to target a long-tail keyword such as create a website forum.

Use the keywords that you are targeting in your forum titles, forum descriptions, and posts. Search engines will only think those keywords are relevant to your forum if they see them being mentioned.

Also, be sure to add meta tags to your forum.

If you're embedding your forum, consider linking to it and making it match your website layout instead. That will ensure that search engines can give full value to inbound links (backlinks).

Reasons for index exclusion

If all of the pages in your forum are not being indexed, it may be due to the following:

  • You have not linked to your forum from a prominent page on your website.
  • Your forum is not viewable by the general public due to password protection, mandatory login, etc.
  • Your pages include the robots meta tag mentioned above.

If some of the pages in your forum are not being indexed, it may be due to the following:

  • The page does not have enough content (wordage) on it.
  • The page has mirrored content. Search engines sometimes avoid indexing mirrored content to prevent spamming.
  • The page is blocked by the robots.txt file.

Checking if your forum is indexed

You can check if your forum is indexed in Google, for example, by searching for "", where is replaced by your forum address.

Page views

When a search engine views a page of your forum, our system still has to process the request the same way it would for a normal human. The same amount of memory, CPU, and bandwidth is used. Therefore, pages viewed by a search engine while indexing your forum are counted as page views.

If you still need help, please contact us.