Given a token stream and a tokenization function, this will tokenize all strings in the given token stream.
The token stream returned by tokenize must have the same text content as the given text.
tokenize
Given a token stream and a tokenization function, this will tokenize all strings in the given token stream.
The token stream returned by
tokenize
must have the same text content as the given text.