riktoken | RubyGems.org | your community gem host

RubyGems

Sign in Sign up

Gemfile

install

A pure Ruby implementation of OpenAI's tiktoken library for BPE tokenization

Required Ruby Version

>= 3.1.0

Authors

moznion

Versions

0.0.1 June 04, 2025 (14.5 KB)

Pushed by

SHA 256 checksum