It is not obligatory to include a robot.txt file on your website but it is incredibly beneficial to an SEO campaign as it helps search engine robots to index the correct pages of a website. It does not prevent a robot from crawling your webpage, however it does instruct them on how to do so correctly. By doing so your search engine result ranking can be improved tremendously. Correct indexing is one of the key factors of a successful SEO campaign.
A robot.txt file will always be located in the main directory of your website in order to help the search engine robots to find it. This is because these robots are not specifically looking for these files, but will obey them when they come across them in the main directory. In your robot.txt file you can specify exactly which pages to disregard and this means that the search engine robot will focus solely on those files that have not been excluded.