Datastreams

Datastreams are a flexible way to manage data ingestion. Each datastream corresponds to a dataset and is managed with unique, revokable tokens. Datastreams keep different streams of data separate.

  • Organize data by source and use case.

  • Grant and revoke permission to store data in Observe through tokens.

  • Pause and restart ingest.

Managing Datastreams

Click the Datastreams button in the left-side navigation menu to see a summary of the currently defined datastreams in your instance. This screen will show the names and important statistics of all datastreams, their status, and some overall information such as the total volume of data being ingested into Observe. Note that the health of a Datastream is shown here; if there have been data ingest errors recorded recently, you will see an Error state in the Health column. Click the Datastream name, locate the Last error column and click View to see the error that was recorded.

Hover over a datastream to get a context menu that allows you to Enable, Disable, Delete, or Open a datastream.

Enabling a datastream allows it to receive data. Disabling it blocks data ingest. Disabling a datastream stops the acceptance of new data into that datastream. Already collected data is not affected and associated tokens are not deleted, only disabled.

Click the name of a Datastream to see a listing of associated Tokens, edit the Datastream name or Data Retention Period, or open it as a Dataset.

Open the Datastream to see a Dataset page that displays the data that has been ingested into it. From here, you can create Worksheets that help you shape the data into Datasets, Monitors, and Dashboards.

Creating a Datastream

To use a Datastream, create the Datastream and then create tokens for that stream. Each Datastream may have multiple tokens. They may be enabled, disabled, or deleted as needed.

Once you create a Datastream, send data to it in the same way you would other data sources. The sections below explain the process in more detail:

  • Create up to 20 Datastreams per workspace. Contact your Observe Data Engineer if you need more than 20 Datastreams.

  1. From the Datastreams list, click Create Datastream in the top right corner.

  2. Provide a Name and an optional Description. A good name identifies the data you expect to ingest into this Datastream.

  3. Set the Datastream’s Retention Period.

  4. Click Create to finish creating the Datastream.

Data Retention

Observe retains your data for thirteen (13) months by default. You can configure the data retention period for less or more than 13 months by editing the datastream and increasing or decreasing the data retention period. Data older than the retention period automatically deletes from the Dataset.

Downstream Datasets inherit the retention period from the Datastream as the minimum value for all input source Datastreams. Observe enforces data retention once a day.

Creating a Token

To send data to a Datastream, you need a token. A Datastream can have multiple tokens.

Warning

Datastream tokens are exclusively for ingesting data into Observe. You cannot ingest data without a datastream token. If you wish to interact with the Observe REST API, you need an API token which you can read about here.

To create a Datastream token:

  1. In the Datastreams tab, click the name of the Datastream where you want to create a token.

  2. Click Create token.

  3. Provide a Name and an optional Description.

  4. Click Continue.

  5. Copy your new token to a safe location, check the box to confirm, and click Continue.

  6. Some example usage statements are provided using the new token; you can copy these and try them from a terminal if you choose, or simply click Close.

Important

Be sure to save your token. You cannot view it again after closing the confirmation dialog.

Ingesting Data

Configure your desired data source to send to Observe, using the datastream token. Observations go to the dataset associated with a Datastream mapped to the token. See Ingest overview for more details on ways to ingest data.

Managing Tokens

Tokens may be enabled, disabled, or deleted.

Enabling a token allows it to receive data. Disabling it blocks data ingest. Disabling a token stops the acceptance of new data sent with that token into that datastream. Already collected data and other tokens in this datastream are not affected by disabling a single token.

Rotate Tokens

To rotate a token safely without affecting other data sources, follow these steps:

  1. Create the new Token in the desired Datastream and test it.

  2. Disable the old Token and ensure that data continues to flow as expected.

  3. Delete the old Token when you are sure that it is no longer needed.