Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.

Required Ruby Version

>= 0

Authors

Stewart McKee

Versions

  1. 1.2.1

    1. source

      1. 1.2.1 source January 09, 2021 (9.28 MB)
  2. 1.2.0

    1. source

      1. 1.2.0 source February 25, 2019 (9.28 MB)
  3. 1.1.0

    1. source

      1. 1.1.0 source November 06, 2015 (9.28 MB)
  4. 1.0.29

    1. source

      1. 1.0.29 source November 05, 2015 (9.28 MB)
  5. 1.0.28

    1. source

      1. 1.0.28 source March 20, 2015 (9.28 MB)
Show all versions (94 total)

Pushed by

stewartmckee

SHA 256 checksum

← Previous version