rz_tiktoken_ruby 0.0.6
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.
Gemfile:
=
クリップボードにコピー
コピー完了!
インストール:
=
Runtime依存関係 (1):
rb_sys
~> 0.9.81
所有者:
プッシュ者:
作者:
-
IAPark, judy
SHA 256チェックサム:
3cc11ddc93574df5bc41ed5d8c1a3eecb2e16266c40102b118c6beb077340695