Demartek Comments on Internet Bots
16 August 2017
By Dennis Martin, Demartek President
The Internet has grown up over the years, and with the popularity of web sites, automated bots have been developed to crawl websites, indexing their content. This process helps popular search engines such as Google, Bing, Yahoo and others to make it easier to find content on the Internet. However, not all bots that crawl the Internet have good intentions. Many of these bots have malicious intent and many of these malicious bots also try to hide themselves by impersonating real human users. This is accomplished by changing the “user agent” field that is provided to web servers to appear to be a web browser, typically the more popular browsers.
In the past, we published and regularly updated two pages on the Demartek web site. One was the Demartek Top Ten highlighting the ten most popular pages on our website, using a rolling four-week average. The other page was our Monthly Browser Usage Statistics showing various statistics for the type of browser used, host operating system and other similar data.
While reviewing our web logs and noticing the number of automated Internet bots accessing our site, we have concluded that it no longer makes sense to continue to publish the data for the two pages mentioned above. Effective today, we are no longer updating those two pages on our website.
Much research has been published recently on Internet Bot activity. The following list provides a few resources on this topic and insight into the reasons behind our decision.