robotstxt-parser 0.1.1

Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Gemfile:
=

安裝:
=

版本列表:

0.1.1 September 18, 2014 (15.0 KB)
0.1.0 September 18, 2014 (15.0 KB)

Development 相依性套件 (2):

fakeweb ~> 1.3

rake >= 0

Show all transitive dependencies

擁有者:

作者:

Garen Torikian

SHA 256 總和檢查碼:

=