Dataverse:Robots.txt
Ir para navegação
Ir para pesquisar
User-agent: * # Disallow: / User-agent: Googlebot User-agent: soscan (+https://dataone.org/) Allow: /$ Allow: /dataset.xhtml Allow: /sitemap* Allow: /api/datasets/:persistentId/thumbnail #Allow: /javax.faces.resource/images/ Disallow: /dataverse/*?q Disallow: /dataverse/*/search Disallow: /dataset.xhtml?*&version=&q= # Disallow: / Crawl-delay: 100 Sitemap: https://arcadados.fiocruz.br/sitemap.xml # Created initially using: http://www.mcanerin.com/EN/search-engine/robots-txt.asp # Verified using: http://tool.motoricerca.info/robots-checker.phtml
