dedup¶
Type of operation: Aggregate
Description¶
dedup collapses all rows in a Table, Event, or Interval dataset with identical values in specified columns and with identical timestamp to a single row. For Interval datasets, the rows will only be collapsed if they have the same “Valid From” and the same “Valid To” time. For the remaining columns, an arbitrary value from the collapsed rows is picked while preferring non-null values.
When no column names are given, dedup collapses rows with identical values in all the columns to a single row.
Usage¶
dedup [ columnname ] ...
Argument |
Type |
Required |
Multiple |
---|---|---|---|
columnname |
expression |
Optional |
Can be multiple |
Accelerable¶
dedup is always accelerable if the input is accelerable. A dataset that only uses accelerable verbs can be accelerated, making queries on the dataset respond faster.
Examples¶
dedup vf, message
Collapse the rows with identical values in vf and message columns and with identical timestamps to a single row.
dedup
Remove duplicate rows in the input dataset
Aliases¶
distinct