1. Tokenization
Definition:
Tokenization is the process of breaking down text into smaller units, called tokens, which can be words, subwords, or even individual characters. These tokens serve as the building blocks that AI models use to process and generate language.
How It Works:
A sentence like "Artificial Intelligence is powerful."
The model tokenizes it into smaller units:
Word-level tokenization:
["Artificial", "Intelligence", "is", "powerful", "."]
Subword tokenization (e.g., Byte-Pair Encoding):
["Artifi", "cial", "Intelli", "gence", "is", "power", "ful", "."]
Character-level tokenization:
["A", "r", "t", "i", "f", "i", "c", "i", "a", "l", ...]
Why It Matters:
Helps models understand and generate text efficiently.
Handles out-of-vocabulary words by breaking them into subwords.
Reduces complexity compared to processing raw text.
2. Context Window
Definition:
The context window refers to the maximum number of tokens an AI model can process at once when generating responses.…