To stay away from unwanted content inside the research indexes, site owners can instruct spiders not to crawl selected documents or directories throughout the conventional robots.txt file in the foundation Listing in the area. Additionally, a webpage could be explicitly excluded from a search engine's databases by using a meta tag unique to robots