However this isn't an easy task, since there are so many ways search engines analyze and rank websites, meaning that there are a lot of factors that affect how a website or single website shows up on a search engine. txt file you can guidebook bots on which are the critical pages and this will do away with the chances of indexing duplicate and irrelevant subject matter.