Robotstxt Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.
Required Ruby Version
None
Required Rubygems Version
>= 1.2
Authors
Simone Rinzivillo