robots.txt 内容:
User-agent: *
Disallow: /subject_search
Disallow: /amazon_search
Disallow: /search
Disallow: /group/search
Disallow: /event/search
Disallow: /celebrities/search
Disallow: /location/drama/search
Disallow: /forum/
Disallow: /new_subject
Disallow: /service/iframe
Disallow: /j/
Disallow: /link2/
Disallow: /recommend/
Disallow: /doubanapp/card
Disallow: /update/topic/
Disallow: /share/
Allow: /ads.txt
Sitemap: https://www.douban.com/sitemap_index.xml
Sitemap: https://www.douban.com/sitemap_updated_index.xml
# Crawl-delay: 5
User-agent: Wandoujia Spider
Disallow: /
User-agent: Mediapartners-Google
Disallow: /subject_search
Disallow: /amazon_search
Disallow: /search
Disallow: /group/search
Disallow: /event/search
Disallow: /celebrities/search
Disallow: /location/drama/search
Disallow: /j/
1./group/topic 在标注为 Disallow 和 Allow 中都没有出现,那么应该默认为 Allow 还是 Disallow ?
2."# Crawl-delay: 5"的单位是什么?
User-agent: *
Disallow: /subject_search
Disallow: /amazon_search
Disallow: /search
Disallow: /group/search
Disallow: /event/search
Disallow: /celebrities/search
Disallow: /location/drama/search
Disallow: /forum/
Disallow: /new_subject
Disallow: /service/iframe
Disallow: /j/
Disallow: /link2/
Disallow: /recommend/
Disallow: /doubanapp/card
Disallow: /update/topic/
Disallow: /share/
Allow: /ads.txt
Sitemap: https://www.douban.com/sitemap_index.xml
Sitemap: https://www.douban.com/sitemap_updated_index.xml
# Crawl-delay: 5
User-agent: Wandoujia Spider
Disallow: /
User-agent: Mediapartners-Google
Disallow: /subject_search
Disallow: /amazon_search
Disallow: /search
Disallow: /group/search
Disallow: /event/search
Disallow: /celebrities/search
Disallow: /location/drama/search
Disallow: /j/
1./group/topic 在标注为 Disallow 和 Allow 中都没有出现,那么应该默认为 Allow 还是 Disallow ?
2."# Crawl-delay: 5"的单位是什么?
4 条回复 • 2021-06-02 09:38:56 +08:00
|
1
zengxs 2021 年 6 月 1 日 |
|
2
AoEiuV020 2021 年 6 月 1 日 如果有 Disallow: /的话是继承的,
|
|
4
marktask 2021 年 6 月 2 日
如果未定义任何蜘蛛,默认就是允许。例如 robots 为空,就是允许任何蜘蛛抓取任何目录
|
• 请不要在回答技术问题时复制粘贴 AI 生成的内容