Complete the code to calculate the number of tokens in a text using a simple split.
text = "Hello world! This is a test." tokens = text.[1]() print(len(tokens))
The split() method splits the text into tokens (words) separated by spaces.
Complete the code to limit the number of tokens to a maximum context window size.
text = "Hello world! This is a test." max_tokens = 5 tokens = text.split() limited_tokens = tokens[:[1]] print(limited_tokens)
We use max_tokens to slice the tokens list and keep only the allowed number of tokens.
Fix the error in the code that counts tokens but mistakenly uses a wrong method.
text = "Sample text for token count." token_count = len(text.[1]()) print(token_count)
The split() method splits the text into tokens, so counting its length gives the token count.
Complete the code to create a dictionary of tokens and their counts, filtering tokens longer than 3 characters.
text = "Hello world! This is a test." tokens = text.split() token_counts = {token: tokens.count(token) for token in tokens if len(token) [1] 3} print(token_counts)
The colon : separates keys and values in a dictionary. The greater than > filters tokens longer than 3 characters.
Fill both blanks to create a dictionary of tokens in uppercase, their counts, and filter tokens with length less than 6.
text = "Hello world! This is a test." tokens = text.split() result = { [1] : tokens.count(token) for token in tokens if len(token) [2] 6 } print(result)
We use token.upper() to convert tokens to uppercase keys, '<' to filter tokens shorter than 6, and ':' to separate keys and values in the dictionary.