Observe Performance Cookbook: Look for Hidden Columns

Problem

A Log Explorer or Worksheet view is taking longer or using more credits than expected.

Solution

Use the Table Settings tool to look for hidden columns, marked by the slash-over-eye icon. Switch to OPAL mode and use pick_col or drop_col to drop these unnecessary columns. Put these verbs as early as possible in your OPAL script.

Explanation

Hidden columns make queries slower, as the data in those columns are still fetched, even if those columns are hidden. The pick_col and drop_col verbs reduce the data volume and make downstream operations faster.

In most cases, you can prefer pick_col over drop_col, because pick_col reduces the chance of overlooking a hidden column, and also avoids re-materializing the dataset if a column is later added to an upstream or input dataset.

Conversely, use drop_col if you want new input columns to be added to the dataset automatically and automatic re-materialization is acceptable.