RubyGems Navigation menu

cobweb 1.2.1

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Gemfile:
=

install:
=

Versions:

  1. 1.2.1 January 09, 2021 (9.28 MB)
  2. 1.2.0 February 25, 2019 (9.28 MB)
  3. 1.1.0 November 06, 2015 (9.28 MB)
  4. 1.0.29 November 05, 2015 (9.28 MB)
  5. 1.0.28 March 20, 2015 (9.28 MB)
Show all versions (94 total)

Runtime Dependencies (9):

addressable >= 2.3.8
haml >= 4.0.7
json >= 1.8.3
nokogiri >= 1.6.0
rake >= 0
redis >= 3.2.1
redis-namespace >= 1.6.0
sinatra >= 1.4.6
slop >= 4.2.0

Development Dependencies (7):

bundle-audit >= 0
coveralls >= 0
mock_redis >= 0
rspec >= 0
rspec-core >= 0
sidekiq >= 0
thin >= 0
Show all transitive dependencies

Owners:

Pushed by:

Authors:

  • Stewart McKee

SHA 256 checksum:

=
← Previous version

Total downloads 333,449

For this version 2,909

Version Released:

License:

MIT

Required Ruby Version: >= 0

Links:

Homepage Documentation Download Review changes Badge Subscribe RSS Report abuse Reverse dependencies