Glossary · AI & Development
Token
The unit of text that language models process, roughly corresponding to a word or a few characters.
Browse all definitionsIn detail
Language models do not process characters or words directly. They split text into tokens using a tokeniser, which breaks text into sub-word units. In English a token is roughly three to four characters or three quarters of a word on average, though common words like 'the' are a single token and long uncommon words may be several tokens. Pricing for API calls to hosted models is denominated in tokens (per million tokens input and output). The context window is also measured in tokens. Code, structured data and non-English languages typically tokenise less efficiently than plain English prose.
Apply the definition
Want to talk through how this applies to your business?
Start with the decision in front of you. We will help map the fit.