# All robots will spider the domain User-agent: * Disallow: Disallow: /cgi-bin/ Disallow: /database/ Disallow: /prjdata/ Disallow: /antherica/