# All robots will spider the domain User-agent: * Disallow: ######################################################### # Disallow the Windows-Update-Agent User-agent: Windows-Update-Agent Disallow: / #Disallow the Microsoft CryptoAPI User-agent: Microsoft-CryptoAPI/5.131.2195.6824 Disallow: / User-agent: Microsoft-CryptoAPI/5.131.2195.6926 Disallow: / User-agent: Microsoft-CryptoAPI/5.131.2600.2180 Disallow: / ######################################################### # Disallow certain directories User-agent: * Disallow: /future/ User-agent: * Disallow: /stats/ User-agent: * Disallow: /storing/ User-agent: * Disallow: /webmaster/ User-agent: * Disallow: /lab/ User-agent: * Disallow: /random.php