llm-tokenizer

KevinKWZheng
esm
llm-tokenizer JS library on GitHub llm-tokenizer JS library on npm Download llm-tokenizer JS library

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

Version 1.0.2 License Apache-2.0
llm-tokenizer has no homepage
llm-tokenizer JS library on GitHub
llm-tokenizer JS library on npm
Download llm-tokenizer JS library
Keywords
OpenAIGPT-3GPT-4GPT-3.5ChatGPTLLMsGPTtokenizerBPEencoderdecoderMachine Learning
INSTALL
Version:
Learn more
Readme Files Statistics Browse CDN