User-agent: * Disallow: /t/trackback Disallow: /t/comments Disallow: /t/stats Disallow: /t/app Disallow: /.m/ # block against duplicate content User-agent: * Disallow: /*.html?cid=* Disallow: /*/comments/page/* Disallow: /*/comments/atom.xml Disallow: /*/comments/rss.xml Disallow: /*/comments/index.rdf User-agent: Googlebot-Mobile Allow: /.m/ Disallow: / User-agent: Y!J-SRD Allow: /.m/ Disallow: / User-agent: Y!J-MBS Allow: /.m/ Disallow: / # block MSIE from abusing cache request User-agent: Active Cache Request Disallow: * user-agent: Pinterestbot Crawl-delay: 1.0 User-agent: msnbot Crawl-delay: 5 User-agent: Yandex Crawl-delay: 1 User-agent: gsa-crawler Disallow: / User-agent: Twitterbot Disallow: