名创801
|
本文以linux云主机或VPS为例，讲述如何拦截搜索引擎蜘蛛抓取页面，下面列出规则: Linux主机的规则文件时 .htaccess，可以手工创建.htaccess文件到站点根目录 <IfModule mod_rewrite.c> RewriteEngine On #Block spider RewriteCond %{HTTP_USER_AGENT} "Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot|WangIDSpider|WBSearchBot|Wotbox|xbfMozi lla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl|Python|Wget|Xenu|ZmEu" [NC] RewriteRule !(^robots\.txt$) - [F] </IfModule> 注：规则中默认屏蔽部分不明蜘蛛，要屏蔽其他蜘蛛按规则添加即可 |
| 由中国名创www.chuang.top 整理发布。 |
