A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

Required Ruby Version

>= 1.9.3

Authors

Andrei Beliankou

Versions

  1. 0.3.0

    1. source

      1. 0.3.0 source January 20, 2016 (10 KB)
  2. 0.2.0

    1. source

      1. 0.2.0 source January 11, 2016 (10 KB)
  3. 0.1.2

    1. source

      1. 0.1.2 source September 03, 2015 (9.5 KB)
  4. 0.1.1

    1. source

      1. 0.1.1 source August 25, 2011 (8 KB)
  5. 0.1.0

    1. source

      1. 0.1.0 source May 19, 2011 (5.5 KB)
Show all versions (6 total)

SHA 256 checksum

← Previous version