We know that it is important to ensure that search engine bots crawl the whole website and index each webpage. This will allow all of our information to be included in search results, allowing people to find it. However, dynamic URLs system could mean that each of our webpage could have changing URLs. It means that, it would mean that people who click on our links at external websites could eventually find 404 Error. This is a problem for both search engine bots and real people. So, what we can do? The most straightforward way is to change dynamic URLs into static ones and avoid using random STOP characters like =, ? and &. URLs should have clean structure like [http://carsalesite.com/cars/BMW/index.html]. This URL doesn’t have STOP characters and it can be easily understood by both search engine bots and real people. Also, we should consider whether dynamic structure is really needed, if our website is small. Many business websites have only less than 20 pages and a static structure should work well enough.
It is true that creating a static website will require some effort, but it should be quite easy to do if our website is small. In general, static websites with simple HTML files are the best for SEO purposes; although they are inefficient if our website is large. However, static structure allows us to have a full control on our script and layout. Websites could load faster and they will be easier to modify. But, if we plan to add hundreds of new webpages in the future, we should consider making them look more static to bots, despite their dynamic technology. Apache is a popular web server technology and it features the “mod rewrite” tool to convert dynamic-looking URLs into static one. We could do this by opening the .htaccess file. We should read the instruction on how to do this and better, we should ask our web developer to create static URLs. Microsoft IIS is also an popular web server platform and it has a similar feature.
We could also use robots.txt file to prevent bots from doing specific actions; such as searching for parts of the website that contain duplicate contents. Dynamic codes could create webpages with nearly identical content for specific purposes. In this case, robots.txt should direct bots to crawl primary webpages, instead of their clones. However, we should be very careful when using robots.txt, because it is possible that we block bots from reaching parts of our website. Other thing that we should do is eliminating STOP characters from the URL. Cookies should also be used and we need to avoid using Session IDs. Sitemap is an old SEO tactic, but it is essential for dynamic websites. It provides search engines bots with direct access to all important parts of the website. Search engines will be able to read the whole structure, despite the use of complex dynamic URLs. Overall, we should find that more and more of our webpages will be included in search results.
