Выбираем одноименный пункт и смотрим и тестим.
для вордпреса получил такой роботс:
User-agent: Yandex
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-trackback
Disallow: /wp-feed
Disallow: /wp-comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Host: prokomputer.ru
Sitemap:
prokomputer.ru/sitemap.xml
User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-trackback
Disallow: /wp-feed
Disallow: /wp-comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Host: prokomputer.ru
Sitemap:
prokomputer.ru/sitemap.xml
в вебмастере пишет Найдено несколько директив Host.
--Важно: Директива Host в файле robots.txt может быть только одна. В случае указания нескольких директив, использоваться будет первая —
help.yandex.ru/webmaster/?id=996567#996574
посему переписал так:
User-agent: Yandex
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-trackback
Disallow: /wp-feed
Disallow: /wp-comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=
Host: prokomputer.ru
Sitemap:
prokomputer.ru/sitemap.xml
User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /wp-trackback
Disallow: /wp-feed
Disallow: /wp-comments
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /?feed=