Data streams

Data streams are a flexible way to manage data ingestion. With a data stream, you may:

• Ingest directly to a specific source dataset, instead of the shared Observation event stream.

• Control ingest permission for a stream via one or more revokable tokens.

• Pause and restart ingest.

• Rotate a token for data stream without affecting other data sources.

• Create up to 10 data streams per workspace.

The Data Streams tab in Workspace Settings also shows a summary of current data streams:

Create a data stream

To use a data stream, first create the stream, and then create tokens for that stream. Each stream may have multiple tokens. They may be enabled, disabled, or deleted as needed.

Once you have created a data stream, send data to it in the same way you would configure other data sources. The sections below explain the process in more detail:

Create a new stream

First, create a new data stream:

1. Go to the Data Streams tab in Workspace settings and click New data stream.

2. Provide a Name and an optional Description. A good name identifies the data you expect to ingest via this stream, as it is also the name of its dataset. Create the stream in a package by prepending the package name. For example, “IoT/Raw device data”.

3. Click New data stream.

Access this stream (it’s source dataset) from the Data streams list by clicking Open dataset on the right side of the window. (Hover over the list row to show the button.) You may also find it in the list of datasets in the Explore tab, or in the More section of the left rail.

Create a token

To send data to a stream, you need a token. A data stream token is like an ingest token, but restricted to a specific stream. A stream can have multiple tokens, and any user with write access to the Workspace can create one.

To create a data stream token:

1. In the Data Streams tab, click on the stream you would like to create a token for.

2. Click New token

3. Provide a Name and an optional Description.

4. Click New token.

5. Copy your new token to a safe location, check the box to confirm, and click Go to token.

Important

Be sure to save your token. You will not be able to view it again after closing the confirmation dialog.

Ingest data

Configure your desired data source to send to Observe, using the data stream token in place of the usual ingest token. More about sending data to Observe. Data stream observations go to the event dataset associated with the token, in the same way they would previously be in the Observation event stream. This dataset is called the stream’s “source dataset.”

Manage data streams and tokens

Data streams and tokens may be enabled, disabled, or deleted. You can pause ingest from a single token or an entire stream.

Note

Only Admin and Writer members may manage data streams and tokens.

To update a stream or token, use the dropdown at the top of the page: