rz_tiktoken_ruby 0.0.6
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.
Gemfile:
=
In die Zwischenablage kopieren
Kopiert!
installieren:
=
Runtime Abhängigkeiten (1):
rb_sys
~> 0.9.81
Besitzer:
Pushed by:
Autoren:
-
IAPark, judy
SHA 256-Prüfsumme:
3cc11ddc93574df5bc41ed5d8c1a3eecb2e16266c40102b118c6beb077340695