Tokenization for LLMs — Corporate Training Module v2

Foundation Keywords

Before diving into tokenization, these are the core concepts you need. Each entry includes a definition, historical context, worked examples, and how it connects to modern LLMs.

Encoding & Character Representation

Encoding

Character

The atomic unit of written language

A character is a single symbol in a writing system — a letter, digit, punctuation mark, emoji, or whitespace. It is an abstract concept; the way a character is stored as numbers is determined by an encoding.

Why it matters: Every text input to an LLM starts as characters. The choice of encoding determines which characters can be represented, and mismatches cause garbled output.

Examples

A · 5 · ! · 你 · → · 😀 · ▁

Encoding

ASCII

American Standard Code for Information Interchange · 1963

ASCII maps 128 characters — English letters, digits, punctuation, and control codes — to integers 0–127. Each character fits in 7 bits. It was the dominant encoding for early computers and the internet.

1963

ASCII finalised by ANSI. Covers A–Z, a–z, 0–9, basic punctuation.

1980s

Extended ASCII variants (ISO 8859-x) add 128 more slots for European accents — but incompatibly across vendors.

1991

Unicode supersedes ASCII, retaining its first 128 code points for compatibility.

ASCII values

A = 65 B = 66 a = 97 0 = 48 space = 32

Limitation: ASCII cannot represent non-English scripts, Chinese, Arabic, emoji, or most of the world's languages — a critical gap for multilingual LLMs.

Encoding

Unicode

Universal character standard · First published 1991

Unicode is a universal standard that assigns a unique code point (a number) to every character in every human writing system. Unicode 15 (2022) defines over 149,000 characters across 161 scripts.

1991

Unicode 1.0 released. Goal: one encoding to represent all of the world's writing systems.

1996

Unicode 2.0 introduces surrogates, enabling code points beyond U+FFFF.

2010

Emoji added (Unicode 6.0), now >3,600 emoji in the standard.

2022

Unicode 15 — 149,186 characters. Used by every major OS, browser, and LLM.

Code points (U+XXXX notation)

A → U+0041 你 → U+4F60 😀 → U+1F600

Key insight: Unicode is an abstract standard — it defines what a character is, not how to store it in bytes. That job belongs to encodings like UTF-8.

Encoding

UTF-8

Unicode Transformation Format – 8-bit · 1993

UTF-8 encodes every Unicode code point as 1–4 bytes. It is backward-compatible with ASCII (the first 128 code points use exactly 1 byte each) and is by far the most common encoding on the web and in modern software.

1 byte

ASCII chars (A–Z, 0–9)

2 bytes

Latin extended, Greek, Arabic

3 bytes

CJK (Chinese, Japanese, Korean)

4 bytes

Emoji & rare scripts

UTF-8 byte representation

A → 0x41 (1 byte)
é → 0xC3 0xA9 (2 bytes)
你 → 0xE4 0xBD 0xA0 (3 bytes)
😀 → 0xF0 0x9F 0x98 0x80 (4 bytes)

Why it matters for LLMs: Byte-level BPE tokenizers (GPT-2, GPT-4, LLaMA) operate directly on UTF-8 bytes, making them robust to any language or emoji without out-of-vocabulary errors.

Encoding

UTF-16 & UTF-32

Alternative Unicode encodings

UTF-16 uses 2 or 4 bytes per character and is used internally by Windows, Java, and JavaScript. UTF-32 uses a fixed 4 bytes for every character — simple but memory-heavy.

Encoding comparison for "Hi 你"

UTF-8: 48 69 20 E4 BD A0 (6 bytes)
UTF-16: 48 00 69 00 20 00 60 4F (8 bytes)
UTF-32: 16 bytes total

LLM relevance: Almost all LLM text pipelines use UTF-8. Awareness of UTF-16 matters when reading data from Windows systems or JavaScript APIs where string lengths differ from byte lengths.

→ UTF-8 → Unicode

Encoding

Code Point

The unique ID for a Unicode character

A code point is the abstract integer assigned to each character in the Unicode standard, written as U+XXXX. There are 1,114,112 possible code points (U+0000 to U+10FFFF).

Code point examples

U+0041 → A
U+00E9 → é (e with acute)
U+4F60 → 你 (you, in Chinese)
U+1F600 → 😀 (grinning face)

Distinction: A code point is an abstract number. The bytes used to store it depend on the encoding (UTF-8 uses 1–4 bytes per code point).

Tokens & Tokenization

Token

The atom of LLM text processing

A token is the smallest chunk of text an LLM processes. Tokens can be whole words, subwords, characters, or bytes — depending on the tokenizer. Every token is mapped to an integer ID from the model's vocabulary.

Pre-2015

NLP models used whole words as tokens. "Running" and "runs" were separate, unrelated entries — vocabulary explosion problem.

2016

BPE adapted for NLP (Sennrich et al.). Subword tokenization balances vocabulary size with coverage.

2019+

GPT-2, BERT, T5 all use subword tokenizers. "Running" → ["Run", "##ning"] or ["run", "ning"].

How "Tokenization" is split

GPT-4 (BPE): ["Token", "ization"]
BERT (WP): ["Token", "##ization"]
char-level: ["T","o","k","e","n","i","z","a","t","i","o","n"]

~100K

GPT-4 vocab size

~4 chars

avg token length (English)

~0.75

words per token (English)

Token

Vocabulary

The complete set of tokens a model knows

A model's vocabulary (or vocab) is the fixed list of all tokens it recognises, each assigned a unique integer ID. During inference, every input token must map to a vocab ID; during generation, the model picks the next ID from this same list.

Vocabulary sizes (approx.)

BERT-base: 30,522 tokens
GPT-2: 50,257 tokens
GPT-4 / CL3: ~100,000 tokens
LLaMA-3: 128,256 tokens

Trade-off: Larger vocab → fewer tokens per sentence (efficient) but more parameters in the embedding matrix. Smaller vocab → more tokens per sentence, slower and more expensive to run.

Token

Token ID

The integer a model actually processes

Each token in the vocabulary is assigned a unique integer, its Token ID. This is the actual input to a neural network — not the text itself. The tokenizer converts text → IDs (encoding) and converts IDs → text (decoding).

Text → Token IDs (GPT-2 example)

"Hello world"
↓ tokenize
[15496, 995]

"I love AI"
↓ tokenize
[40, 1842, 9552]

Important: Two different tokenizers will produce different IDs for the same text. A BERT token ID of 1234 is completely unrelated to a GPT-2 token ID of 1234.

Token

Special Tokens

Control signals in the token stream

Special tokens are reserved tokens that convey structural meaning rather than text content. They tell the model where a sequence starts, ends, is padded, or where a different speaker begins.

Common special tokens

[CLS] – classification token (BERT)
[SEP] – sentence separator (BERT)
[PAD] – padding to equal length
[UNK] – unknown / out-of-vocab
<s> – start of sequence
</s> – end of sequence
<|im_start|> – speaker turn (ChatML)

Why it matters: Incorrectly handled special tokens are a common source of subtle bugs when integrating LLM APIs — e.g., double-adding [CLS] or mismatching chat template tokens.

Token

Context Window

Maximum tokens a model can see at once

The context window (or context length) is the maximum number of tokens — input + output — that a model can process in a single call. Tokens beyond this limit are silently dropped.

Context window sizes (2024)

GPT-3.5: 4,096 tokens (~3,000 words)
GPT-4: 128,000 tokens (~96,000 words)
Claude 3: 200,000 tokens (~150,000 words)
Gemini 1.5: 1,000,000 tokens

Practical rule: 1,000 tokens ≈ 750 English words ≈ 1.5 pages of text. Always estimate token usage before sending long documents to an LLM.

Embeddings & Representations

Embedding

A dense numeric representation of meaning

An embedding converts a token ID into a dense vector of floating-point numbers (e.g., 768 or 4096 dimensions). These numbers encode semantic meaning: similar words cluster together in vector space.

1986

Hinton et al. introduce distributed representations — the conceptual origin of embeddings.

2013

Word2Vec (Mikolov, Google). First scalable word embeddings. "king" − "man" + "woman" ≈ "queen".

2014

GloVe (Stanford). Global co-occurrence statistics produce richer embeddings.

2018

ELMo & BERT. Contextual embeddings — "bank" means different things in different sentences.

2020+

GPT-3 / LLaMA / Claude. Embeddings are the first layer of every transformer LLM.

Conceptual vector (simplified)

"king" → [0.32, -0.14, 0.87, 0.05, ...] (768 dims)
"queen" → [0.31, -0.13, 0.86, 0.12, ...]
"apple" → [-0.44, 0.72, -0.21, 0.63, ...]

Connection to tokenization: The embedding layer is the bridge — it converts each token ID (an integer) into a vector the neural network can compute with. Better tokenization → better embeddings.

Embedding

Embedding Matrix

The lookup table that maps IDs to vectors

The embedding matrix is a parameter table of shape vocab_size × d_model. Given a token ID, the model simply looks up the corresponding row. These weights are learned during training.

Size calculation

GPT-2 embedding matrix:
50,257 tokens × 768 dims
= 38.6 million parameters

LLaMA-3 (70B) embedding matrix:
128,256 tokens × 8,192 dims
= ~1.05 billion parameters

Design impact: This is why vocabulary size matters — a 2× larger vocab means a 2× larger embedding matrix and more memory, even before the transformer layers.

Embedding

Word2Vec

Mikolov et al., Google · 2013

Word2Vec trains shallow neural networks to predict a word from its neighbours (CBOW) or predict neighbours from a word (Skip-gram). The internal weights become word embeddings that capture semantic relationships.

Famous analogy property

vec("king") − vec("man") + vec("woman")
≈ vec("queen") ✓

vec("Paris") − vec("France") + vec("Italy")
≈ vec("Rome") ✓

Historical significance: Word2Vec showed that neural networks could learn meaningful linguistic structure without hand-crafted rules — a key step toward modern LLMs. However, each word had one fixed vector regardless of context.

→ Embedding → GloVe → BERT

Embedding

Semantic Similarity

Measuring meaning closeness in vector space

Semantic similarity is quantified as the angle between two embedding vectors — specifically, cosine similarity. A score of 1.0 means identical meaning, 0 means unrelated, −1 means opposite.

Cosine similarity examples

sim("cat", "kitten") = 0.92 (very similar)
sim("cat", "dog") = 0.76 (related)
sim("cat", "democracy")= 0.04 (unrelated)
sim("hot", "cold") = 0.31 (antonyms — closer than you'd think!)

Applications: Semantic search, retrieval-augmented generation (RAG), recommendation systems, and duplicate detection all rely on fast cosine similarity over embeddings.

Model & Pipeline Concepts

Model

Corpus

The training text dataset (pl. corpora)

A corpus is the body of text used to train a tokenizer or language model. The tokenizer vocabulary is built to reflect the most frequent substrings in the corpus — so the corpus composition directly shapes what gets a token.

Notable training corpora

GPT-2: WebText — 8M web pages (40 GB)
GPT-3: Common Crawl + Books + Wikipedia (570 GB)
LLaMA-3: ~15 trillion tokens of curated web text
BERT: Wikipedia + BookCorpus (16 GB)

Consequence: A tokenizer trained on English-heavy data will give more efficient (shorter) tokenizations for English than for Hindi or Swahili — a real fairness and cost concern.

Model

Normalisation

Cleaning text before tokenization

Normalisation is the first stage of the tokenization pipeline. It transforms raw text into a consistent form — lowercasing, removing accents, applying Unicode normal forms (NFC, NFD, NFKC, NFKD) — before splitting into tokens.

Unicode normal forms

Input: "Héllo" (é as U+00E9)
NFD: "Héllo" (decomposed: e + combining accent)
NFC: "Héllo" (recomposed: single U+00E9)
NFKC: "Hello" (compatibility: drops accent)

BERT uses: NFD + lowercase + strip accents

Why it matters: The same visual character can have multiple Unicode representations. Without normalisation, "café" and "café" would tokenize differently even though they look identical.

Model

Attention & Attention Mask

The mechanism that connects tokens to each other

Attention is the core mechanism of transformer models. Each token "attends to" other tokens to build context. The attention mask is a binary tensor (1 = real token, 0 = padding) that prevents the model from attending to padding tokens.

Attention mask example

Input: ["The", "cat", "sat", "[PAD]", "[PAD]"]
IDs: [ 464, 3797, 3332, 0, 0]
Mask: [ 1, 1, 1, 0, 0]

Tokenizer's role: The tokenizer outputs both the input IDs and the attention mask. Forgetting to pass the mask is a silent bug — the model will waste computation attending to meaningless padding tokens.

Model

Out-Of-Vocabulary (OOV)

Words the tokenizer has no token for

An out-of-vocabulary word is one that the tokenizer cannot represent as a single token from its vocab. Word-level tokenizers replace it with a special [UNK] token. Subword tokenizers handle OOV by decomposing the word into smaller known pieces.

OOV handling comparison

Word "ChatGPT" (not in 2018 vocab):

Word-level: [UNK] ← information lost
WordPiece: ["Chat", "##GP", "##T"]
Byte-level BPE: ["Ch","at","G","PT"] or bytes
← no information lost

Why subwords won: Eliminating OOV was the primary motivation for subword tokenization (BPE, WordPiece, Unigram). Byte-level BPE guarantees zero OOV for any UTF-8 input.

Model

Padding & Truncation

Making sequences uniform length for batching

Padding extends short sequences to a fixed length by appending [PAD] tokens. Truncation cuts long sequences to fit within the model's max length. Both are necessary for batching multiple inputs together in a single tensor.

Padding to max_length=6

"Hi" → [ 9906, 50256, 50256, 50256, 50256, 50256]
"Hello" → [ 15496, 50256, 50256, 50256, 50256, 50256]
"Hello world" → [15496, 995, 50256, 50256, 50256, 50256]

Best practice: Always use padding=True, truncation=True with the Hugging Face tokenizer. Never manually truncate — the tokenizer handles edge cases correctly.

How These Concepts Connect

Raw text flows through these layers in order. Understanding each layer helps you debug problems, optimise costs, and choose the right model.

Raw Text (Unicode)

→

UTF-8 Bytes

→

Normalisation

→

Tokenization

→

Token IDs

→

Embedding Vectors

→

Transformer Layers

Learning objectives

Explain what a token is and why LLMs use tokens instead of raw words
Describe the full tokenization pipeline: normalisation → pre-tokenisation → model → post-processing
Compare the 5 major tokenization algorithms (Word, Character, BPE, WordPiece, SentencePiece/Unigram)
Read and interpret tokenizer output including token IDs, attention masks and special tokens
Estimate token counts and understand the cost, context-window and quality implications in the workplace
Apply practical strategies to reduce token usage when working with LLM APIs

2 hrs

Duration

7

Module sections

5

Tokenizer types

4

Pipeline stages

10

Quiz questions

80%

Pass mark

Module at a glance

#	Section	Format	Time
1	Welcome & scene-setting	Discussion + warm-up	10 min
2	What is a token?	Lecture + visual	20 min
3	Full tokenization pipeline HF	Lecture + code walkthrough	30 min
4	Why it matters at work	Case studies	25 min
5	Common pitfalls & tips	Facilitated discussion	15 min
6	Knowledge check quiz	Individual	15 min
7	Debrief & close	Q&A	5 min

Materials needed

Projector / shared screen for live code demos
Access to platform.openai.com/tokenizer (no login needed)
Access to Google Colab for optional Hugging Face code demos
Printed quick-reference cards (one per learner)
Whiteboard or flip chart for BPE step-through exercise
Printed or digital 10-question quiz

Source: Hugging Face LLM Course — Chapter 6

The 4-stage tokenization pipeline

Every modern tokenizer executes exactly four steps before the model ever sees your text. Understanding each step helps you debug unexpected outputs and write better prompts.

Raw text

→

1. Normalisation

→

2. Pre-tokenisation

→

3. Model (BPE / WP / SP)

→

4. Post-processing

→

Token IDs → LLM

1

Normalisation

Cleans the text before any splitting occurs. Operations include: lowercasing, Unicode normalisation (NFC / NFKC), accent removal, and whitespace stripping.

BERT example: "Héllò hôw are ü?" → "hello how are u?"

GPT-2: preserves case and formatting — produces richer but more complex token sequences.

Why it matters: Inconsistent text (e.g. "Résumé" vs "Resume") produces different tokens for the same meaning — wasting context window and increasing cost.

from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") # Inspect the normalizer result = tokenizer.backend_tokenizer.normalizer.normalize_str("Héllò hôw are ü?") # Output: 'hello how are u?'

2

Pre-tokenisation

Splits text into rough "words" before the subword algorithm runs. Different tokenizers split differently:

Tokenizer	Pre-tokenisation rule	Example
BERT (WordPiece)	Split on whitespace + punctuation	"Hello," → ["Hello", ","]
GPT-2 (BPE)	Split on whitespace, keep spaces as Ġ prefix	"Ġhow", "Ġare"
T5 (SentencePiece)	Treat raw bytes, space = ▁ symbol	"▁Hello", "▁world"

# BERT pre-tokenizer tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str("Hello, how are you?") # [('Hello', (0,5)), (',', (5,6)), ('how', (7,10)), ('are', (11,14)), ('you', (16,19)), ('?', (19,20))] # GPT-2 keeps spaces as Ġ gpt2_tok = AutoTokenizer.from_pretrained("gpt2") gpt2_tok.backend_tokenizer.pre_tokenizer.pre_tokenize_str("Hello, how are you?") # [('Hello', (0,5)), (',', (5,6)), ('Ġhow', (6,10)), ('Ġare', (10,14)), ('Ġyou', (15,19)), ('?', (19,20))]

3

Model (subword algorithm)

The core tokenization step. Takes pre-tokenised words and applies the learned merge rules (BPE), likelihood scores (WordPiece / Unigram), or language-agnostic byte-stream rules (SentencePiece) to produce subword tokens.

Output: a sequence of subword strings. e.g. "tokenization" → ["token", "ization"]

BPE in one sentence: Start with characters; repeatedly merge the most frequent adjacent pair into a new token until vocabulary size is reached.

4

Post-processing

Adds model-specific special tokens and produces final model inputs. This is also where padding and truncation happen.

Special token	Meaning	Model
[CLS]	Classification / start of sequence	BERT
[SEP]	Separator between sequences	BERT
[PAD]	Padding to uniform length	Most models
[UNK]	Unknown token (out-of-vocab)	Most models
<s> / </s>	Start / end of sequence	RoBERTa, T5
<\|endoftext\|>	End of document	GPT-2, GPT-4
<\|im_start\|>	Chat message boundary	GPT-4, Mistral

Padding & Truncation: When batching multiple inputs, all sequences must be the same length. Padding adds [PAD] tokens to shorter sequences. Truncation removes tokens from sequences that exceed the model's context window. Attention masks (1 = real token, 0 = padding) tell the model which tokens to ignore.

# Full tokenizer call — what really happens sequences = ["I've been waiting for this course my whole life.", "So have I!"] inputs = tokenizer(sequences, padding="longest", truncation=True, return_tensors="pt") # inputs["input_ids"] → tensor of token IDs (padded to same length) # inputs["attention_mask"] → 1 for real tokens, 0 for [PAD] tokens

From Hugging Face — Step-by-step decoding

This is exactly what happens under the hood when you call a tokenizer:

from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") input_str = "Hugging Face Transformers is great!" # Step 1: tokenize (returns subword strings) tokens = tokenizer.tokenize(input_str) # → ["hugging", "face", "transformers", "is", "great", "!"] # Step 2: convert tokens → integer IDs ids = tokenizer.convert_tokens_to_ids(tokens) # → [17662, 2227, 19081, 2003, 2307, 999] # Step 3: add special tokens ([CLS]=101, [SEP]=102) ids_with_special = [101] + ids + [102] # → [101, 17662, 2227, 19081, 2003, 2307, 999, 102] # Step 4: decode back to text decoded = tokenizer.decode(ids_with_special) # → "[CLS] hugging face transformers is great! [SEP]"

Facilitator note: Show this live in Google Colab or run it on a shared screen. The jump from human-readable words to a list of numbers is the "aha" moment for most non-technical learners.

Based on Hugging Face LLM Course — Chapter 6

Select a tokenizer type

Word-level

Simplest

Character-level

Most granular

BPE

GPT / Claude

WordPiece

BERT / Google

SentencePiece

LLaMA / T5

Select a type above

HF Course — BPE training algorithm (step-by-step)

BPE starts with individual characters and repeatedly merges the most frequent adjacent pair. Here's the algorithm on the corpus: "hug"(×10), "pug"(×5), "pun"(×12), "bun"(×4), "hugs"(×5)

Initial vocabulary (characters only)

bghnpsu

1

Most frequent pair:("u","g") → 20 times→New token:ugVocab size: 8

2

Most frequent pair:("u","n") → 16 times→New token:unVocab size: 9

3

Most frequent pair:("h","ug") → 15 times→New token:hugVocab size: 10

4

Most frequent pair:("p","ug") → 5 times→New token:pugVocab size: 11

This continues until the vocabulary reaches the target size (e.g. 50,000 tokens for GPT-2). The resulting merge rules are saved and applied to any new text at inference time.

Byte-level BPE (GPT-2, Claude): Instead of starting with Unicode characters, it starts with all 256 bytes. This guarantees that any character — including emojis, rare symbols, and multi-script text — can be encoded without [UNK] tokens.

Model ↔ Tokenizer reference table

Model family	Tokenizer	Vocab size	Notes
GPT-2 / GPT-3 / GPT-4	Byte-level BPE	~50K / ~100K	Uses tiktoken library; no [UNK]
Claude (Anthropic)	BPE variant	~100K	Similar to GPT-4 tokenizer
BERT / RoBERTa / DistilBERT	WordPiece	~30K	[CLS], [SEP], [MASK] special tokens
LLaMA 2 / Gemma	SentencePiece (BPE)	~32K	Uses ▁ for spaces
LLaMA 3	SentencePiece (BPE)	~128K	Expanded vocab for better multilingual coverage
T5 / mT5	SentencePiece (Unigram)	~32K	Fully language-agnostic
Mistral / Mixtral	SentencePiece (BPE)	~32K	Shares Llama 2 tokenizer

Token count comparison — same sentence, different methods

Sentence: "The quick brown fox jumps over the lazy dog."

Hugging Face — AutoTokenizer usage patterns

The AutoTokenizer class automatically selects the correct tokenizer for any model. It's the recommended starting point for all tokenization tasks.

from transformers import AutoTokenizer # Load any model's tokenizer by name tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") # Three ways to use it: # 1. Tokenize only tokens = tokenizer.tokenize("Hugging Face is great!") # → ['hugging', 'face', 'is', 'great', '!'] # 2. Tokenize + convert to IDs ids = tokenizer.encode("Hugging Face is great!") # → [101, 17662, 2227, 2003, 2307, 999, 102] (includes [CLS]=101, [SEP]=102) # 3. Full call (recommended for model input) inputs = tokenizer("Hugging Face is great!", return_tensors="pt") # Returns: input_ids, token_type_ids, attention_mask

Hugging Face — Padding & truncation

# When processing multiple sequences (batches), use padding + truncation sequences = [ "I've been waiting for a HuggingFace course my whole life.", "So have I!" ] # Pad to longest sequence in the batch inputs = tokenizer(sequences, padding="longest") # Pad to model's max length (512 for BERT) inputs = tokenizer(sequences, padding="max_length") # Truncate sequences longer than 8 tokens inputs = tokenizer(sequences, max_length=8, truncation=True) # Return PyTorch tensors inputs = tokenizer(sequences, padding=True, return_tensors="pt") # The attention_mask tells the model which tokens are real (1) vs padding (0) # inputs["attention_mask"] → [[1,1,1,...,1], [1,1,1,0,0,...,0]]

Kaggle-style — Building a custom BPE tokenizer from scratch

Training a domain-specific tokenizer for your own corpus (e.g. legal, medical, technical):

from tokenizers import Tokenizer from tokenizers.models import BPE from tokenizers.trainers import BpeTrainer from tokenizers.pre_tokenizers import Whitespace # 1. Initialise a blank BPE tokenizer tokenizer = Tokenizer(BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = Whitespace() # 2. Configure the trainer with special tokens trainer = BpeTrainer( special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], vocab_size=30000 ) # 3. Train on your files files = ["my_domain_corpus.txt"] tokenizer.train(files, trainer) # 4. Save and reload tokenizer.save("my_tokenizer.json") tokenizer = Tokenizer.from_file("my_tokenizer.json") # 5. Encode some text encoded = tokenizer.encode("Here is some domain-specific text to encode") print(encoded.tokens) # → ['Here', 'is', 'some', 'domain', '-', 'specific', ...] print(encoded.ids) # → [423, 67, 991, 2041, 15, 876, ...]

When to build a custom tokenizer: Your corpus uses domain-specific jargon (medical codes, legal terms, product SKUs) that common tokenizers fragment inefficiently. A custom tokenizer reduces token counts and improves model accuracy for your domain.

Live demo — Try it yourself

Type any text to see a simulated BPE tokenization with token IDs.

0

Tokens

0

Characters

0

Words

0

Chars/token

Simulated token IDs

Language token efficiency (BPE)

Approximate tokens to express "Good morning, how are you?" across languages:

Knowledge check — 10 questions · pass mark 80%

Tokenization Concepts for LLMs

Foundation Keywords