RubyGems Navigation menu

robotstxt-parser 0.1.0

Robotstxt-Parser allows you to the check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap: rules.

Versions:

  1. 0.1.1 - September 18, 2014 (15 KB)
  2. 0.1.0 - September 18, 2014 (15 KB)

Development Dependencies (2):

  • fakeweb ~> 1.3
  • rake >= 0
  • Owners:

    Authors:

    • Garen Torikian

    SHA 256 checksum:

    9e7ba258e6275c63afa5abe8ef5843d608da23fc365bf4af43dc9178bb31fe00

    Total downloads 80,691

    For this version 1,202

    Gemfile:
    = Copy to clipboard Copied!

    install:
    =

    License:

    MIT

    Required Ruby Version: >= 0

    Links: