this post was submitted on 18 Jun 2023
23 points (100.0% liked)

Programmer Humor

32050 readers
1663 users here now

Post funny things about programming here! (Or just rant about your favourite programming language.)

Rules:

founded 5 years ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[โ€“] clb92@kbin.social 0 points 1 year ago (1 children)

Sure it can create new words. It can't create new tokens, would be more correct, I think. But a token is just a text fragment, and, as far as I know, they can range from being several words to being single characters.

[โ€“] Dr_Cog@lemmy.ml 1 points 1 year ago

Tokens are usually never multiple words. Think of them like "information units". If you have a plural word "hats", there are two tokens: the "hat" one and the "s' that adds more information about it being plural. Combinations of words only really occur for proper names.