gpt-tokenizer

esm cjs
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models
Version 2.9.0 License MIT
Keywords
BPEencoderdecodertokenizerGPTGPT-2GPT-3GPT-3.5GPT-4GPT-4oNLPNatural Language ProcessingText GenerationOpenAIMachine Learning
INSTALL