tokenize¶

Description¶

Splits the string into an array based on separator, which is treated as a set of characters.

The default separator is “ “.

array

This is a scalar function (calculates a single output value for a single input row.)

tokenize(value, [ separator ])

Argument	Type	Optional	Repeatable	Restrictions
value	string	no	no	none
separator	string	yes	no	constant

tokenize("a b c d")

Returns [“a”, “b”, “c”, “d”].

tokenize("[email protected]", "@.")

Returns [“hello”, “example”, “com”].