Search engine spiders are by far one of the most useful things to come around in the last 10 years of the internet. They are useful not only to the web sites (Google and many others) that use them, but also to people who are searching for a particular site and those who run web sites. Spiders allow your site to be seen by the millions of people who use search engines every day. In this newsletter, we will discuss what search engine spiders do, how they work, and how to set up a robots.txt file and upload that to your site to keep spiders from visiting your site.

Web page creation is no child’s play. A huge amount of time is spent in planning the look and feel of a website, before the actually development begins. From a user’s perspective, apart from the website itself, the only tangible evidence that vouches for this planning is the visible operational elements. In other words, front-end elements that are related to the functionality of the website. A sitemap is one such element.