Заказывал проверку прогой дублей нет
Allow: /
Disallow: /*.php
Disallow: /search*
Disallow: /comments*
Disallow: /auth
Disallow: /rss
Disallow: /redirect?url=
Disallow: /go
Disallow: /*/top*
Disallow: /*?iframe*
Disallow: /index.php
Disallow: /geo
Disallow: /users
Disallow: /index
Disallow: /tag
Disallow: /*.php
Disallow: /search*
Disallow: /comments*
Disallow: /auth
Disallow: /rss
Disallow: /redirect?url=
Disallow: /go
Disallow: /*/top*
Disallow: /*?iframe*
Disallow: /index.php
Disallow: /geo
Disallow: /users
Disallow: /index
Disallow: /tag