What is Token?
The small chunk of text, often a word piece, that models read and generate.
Formal Definition
A token is the unit of text a model processes, typically a word or sub-word fragment produced by a tokenizer. Model limits, latency, and API pricing are all measured in tokens. In AI Stock Challenge, token counts govern how much market data fits in a request and factor into the cost and speed of each model's decision.
In Simple Terms
It is the building block of text for an AI. Instead of reading whole sentences at once, a model breaks everything into little pieces called tokens, and both its memory limit and its bill are counted in them.
Example
A dense block of market news might be several thousand tokens, so the benchmark condenses it to leave room for the model's reasoning and answer.