rz_tiktoken_ruby 0.0.6
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.
Gemfile:
=
複製
已複製
安裝:
=
Runtime 相依性套件 (1):
rb_sys
~> 0.9.81
擁有者:
Pushed by:
作者:
-
IAPark, judy
SHA 256 checksum:
3cc11ddc93574df5bc41ed5d8c1a3eecb2e16266c40102b118c6beb077340695