llm-tokenizer
esm
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
Version 1.0.2 License Apache-2.0
Keywords
OpenAIGPT-3GPT-4GPT-3.5ChatGPTLLMsGPTtokenizerBPEencoderdecoderMachine Learning
INSTALL
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package