搜索引擎的蜘蛛人（ROBOTS）秘密(四)

2007-8-8 9:36:46

例子:
# robots, scram

User-agent: *
Disallow:   /cgi-bin
Disallow:   /TRANSCRIPTS
Disallow:   /development
Disallow:   /third
Disallow:   /beta
Disallow:   /java
Disallow:   /shockwave
Disallow:   /JOBS
Disallow:   /pr
Disallow:   /Interactive
Disallow:   /alt_index.html
Disallow:   /webmaster_logs
Disallow:   /newscenter
Disallow:       /virtual
Disallow:       /DIGEST
Disallow:       /QUICKNEWS
Disallow:       /SEARCH

User-agent: Mozilla/3.01 (hotwired-test/0.1)
Disallow:   /cgi-bin
Disallow:   /TRANSCRIPTS
Disallow:   /development
Disallow:   /third
Disallow:   /beta
Disallow:   /java
Disallow:   /shockwave
Disallow:   /JOBS
Disallow:   /pr
Disallow:   /Interactive
Disallow:   /alt_index.html
Disallow:   /webmaster_logs
Disallow:   /newscenter
Disallow:       /virtual
Disallow:       /DIGEST
Disallow:       /QUICKNEWS
Disallow:       /SEARCH

User-agent: Slurp
Disallow:   /cgi-bin
Disallow:   /TRANSCRIPTS
Disallow:   /development
Disallow:   /third
Disallow:   /beta
Disallow:   /java
Disallow:   /shockwave
Disallow:   /JOBS
Disallow:   /pr
Disallow:   /Interactive
Disallow:   /alt_index.html
Disallow:   /webmaster_logs
Disallow:   /newscenter
Disallow:       /virtual
Disallow:       /DIGEST
Disallow:       /QUICKNEWS
Disallow:       /SEARCH

User-agent: Scooter
Disallow:   /cgi-bin
Disallow:   /TRANSCRIPTS
Disallow:   /development
Disallow:   /third
Disallow:   /beta
Disallow:   /java
Disallow:   /shockwave
Disallow:   /JOBS
Disallow:   /pr
Disallow:   /Interactive
Disallow:   /alt_index.html
Disallow:   /webmaster_logs
Disallow:   /newscenter
Disallow:       /virtual
Disallow:       /DIGEST
Disallow:       /QUICKNEWS
Disallow:       /SEARCH

User-agent: Ultraseek
Disallow:   /cgi-bin
#Disallow:   /TRANSCRIPTS
Disallow:   /development
Disallow:   /third
Disallow:   /beta
Disallow:   /java
Disallow:   /shockwave
Disallow:   /JOBS
Disallow:   /pr
Disallow:   /Interactive
Disallow:   /alt_index.html
Disallow:   /webmaster_logs
Disallow:   /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH

User-agent: smallbear
Disallow: /cgi-bin
Disallow: /java
Disallow: /images
Disallow: /development
Disallow:   /third
Disallow: /beta
Disallow: /webmaster_logs
Disallow: /virtual
Disallow: /shockwave
Disallow: /TRANSCRIPTS
Disallow:   /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
Disallow:   /alt_index.html
User-agent: GoogleBot
Disallow: /cgi-bin
Disallow: /java
Disallow: /images
Disallow: /development
Disallow:   /third
Disallow: /beta
Disallow: /webmaster_logs
Disallow: /virtual
Disallow: /shockwave
Disallow: /TRANSCRIPTS
Disallow:   /newscenter
Disallow: /virtual
Disallow: /DIGEST
Disallow: /QUICKNEWS
Disallow: /SEARCH
Disallow:   /alt_index.html

责任编辑:

相关文章