Datastreams

Datastreams are a flexible way to manage data ingestion. Each datastream corresponds to a dataset and is managed with unique, revokable tokens. With a Datastream, you may perform the following tasks:

  • Ingest directly to a specific source dataset, using Endpoints, Sources and Forwarders.

  • Control ingest permission for a Datastream through one or more revokable tokens.

  • Pause and restart ingest.

  • Rotate a token for Datastream without affecting other data sources.

  • Create up to 20 Datastreams per workspace. You may contact us at support@observeinc.com or talk to your Observe Data Engineer if you need more than 20 Datastreams.

The Datastreams tab in Workspace Settings also shows a summary of current datastreams:

Datastreams page, showing a list of streams and summary data for all streams. Observations per hour, number of streams, number of tokens, and last observation received.

Figure 1 - Datastream Summary page

Creating a Datastream

To use a Datastream, create the Datastream and then create tokens for that stream. Each Datastream may have multiple tokens. They may be enabled, disabled, or deleted as needed.

Once you create a Datastream, send data to it in the same way you would other data sources. The sections below explain the process in more detail:

Creating a new Datastream

First, create a new Datastream:

New datastream dialog

Figure 2 - Open a new Datastream

  1. Go to Datastreams in Workspace settings and click New datastream.

  2. Provide a Name and an optional Description. A good name identifies the data you expect to ingest into this Datastream, as it is also the name of the corresponding dataset. Create the Datastream in a package by prepending the package name. For example, “IoT/Raw device data”.

  3. Click New Datastream.

Access this stream, the source dataset, from the Datastreams list by clicking Open dataset on the right side of the window. You may also find it in the list of Datasets on the Explore tab, or in the More section of the left menu.

List of datastreams, hover over a row to access the Open dataset button for a stream.

Figure 3 - Open the dataset for a Datastream

Creating a Token

To send data to a stream, you need a token. A Datastream token is similar to an ingest token but restricted to a specific stream. A Datastream can have multiple tokens; any user with write access to the Workspace can create one.

New datastream token dialog

Figure 4 - New token for a Datastream

To create a Datastream token:

  1. In the Datastreams tab, click on the Datastream you want to create a corresponding token.

  2. Click New token.

  3. Provide a Name and an optional Description.

  4. Click New token.

  5. Copy your new token to a safe location, check the box to confirm, and click Go to token.

Important

Be sure to save your token. You cannot view it again after closing the confirmation dialog.

Ingesting Data

Configure your desired data source to send to Observe, using the datastream token. Observations go to the dataset associated with a Datastream mapped to the token. Observe calls this Dataset, the Datastream source dataset. See Ingest overview for more details on ways to ingest data.

Managing Datastreams and Tokens

Datastreams and tokens may be enabled, disabled, or deleted. You can pause ingest from a single token or an entire stream.

Note

Only Admin and Writer members may manage Datastreams and tokens.

To update a stream or token, use the dropdown at the top of the page:

datastream detail page, with Enabled/Disabled/Delete menu open

Figure 13 - Enabled/Disabled/Delete options for a datastream

Finding Tokens

Find a datastream token on your Observe instance by entering the text before the : and clicking Search.

../../_images/find-token.png

Figure 14 - Finding a Token for a Datastream

Click Find token to open the search dialogue and enter the text of the existing token before the :.

../../_images/search-token.png

Figure 15 - Searching for a Token of a Datastream

The search returns the Datastream with that token.

Configurable Data Retention

Observe retains your data for thirteen (13) months by default. You can configure the data retention period for less or more than 13 months by editing the datastream and increasing or decreasing the data retention period. Data older than the retention period automatically deletes from the Dataset.

Configuring Data Retention

Figure 1 - Configuring Data Retention

Downstream Datasets inherit the retention period from the Datastream as the minimum value for all input source Datastreams. Observe enforces data retention once a day.

As an additional benefit, shortening the data retention period can reduce your storage costs.