Ingest Routing
Learn how to configure routing to ingest your data and direct it to the desired Dataset.
Private preview feature (Invite only)This feature is enabled per tenant by Observe by Snowflake and is not generally available. To participate, contact your Observe by Snowflake representative. You may access and use this preview feature solely for internal evaluation purposes. Previews are provided as-is. Your use of this preview is subject to the Snowflake Preview Terms located at https://www.snowflake.com/en/legal.
Observe by Snowflake ingest routing allows you to specify content-based routes to direct incoming data to different Source Datasets via sets of rules matching the content of the observations.
Data flow
Flexible Observe by Snowflake data ingestion consists of three concepts:
- Self-standing ingest tokens authenticate a request and nothing more. They carry no destination. Find them under your user name in the left navigation rail, at Manage account > Ingest tokens.
- Type-specific Source Datasets store one type of data each, and work out of the box with Observe by Snowflake's explorers and Worksheets. Generic Any type Datasets can store any other telemetry, but require more data shaping. Find your Source Datasets under your user name in the left navigation rail, at Data & integrations > Source Datasets.
- Type and endpoint-specific ingest routes decide where data is stored. Find them under your user name in the left navigation rail at Data & integrations > Ingest routing.
The following diagram illustrates how Observe by Snowflake applies your routing rules to incoming data:
flowchart TD
A[Your client sends data<br/>with an ingest token]
--> B[Observe authenticates<br/>the request]
B --> C{Evaluate the rules for<br/>this data type,<br/>in priority order}
C --> D[First matching rule<br/>selects the destination]
D --> E[Drop filters run on<br/>the destination Dataset]
E --> F[Remaining data is stored<br/>in the Dataset]
NoteRouting and drop filters
Drop filters are attached to Source Datasets. Observe by Snowflake applies ingest routing first to determine the Destination Dataset, then applies any drop filters configured on that Dataset.
Choose a routing method
| Client configuration | Destination selection | What to configure |
|---|---|---|
Standalone token with X-Observe-Target-Package | A managed package route selects a compatible source dataset in that package. | Create the package's Dataset for the data type and preserve the setup header. |
| Standalone token without a package header or destination association | Enabled user-defined routes run in order, followed by the default route. | Configure the rules and default destination before sending data. |
| Token with an existing destination association | The association can take precedence over ordinary package and user-defined routes. | Preserve the existing configuration until you test a migration. |
| Legacy Datastream token | Its existing Datastream binding selects the destination. | Use the Datastream workflow; omitting a header does not migrate the token. |
For managed routes, Observe Agent package override routes precede token association routes. Token association routes precede ordinary package routes. All these managed routes precede user-defined rules, and the default route is last.
Do not use package names or content predicates as authorization boundaries. To adopt content routing, use an unassociated standalone token and omit the package header only after configuring the destination rules.
Package routing or custom routes
Package routing and custom routes both choose a Source Dataset for each observation. They differ in where that choice is made. With package routing, the client names its destination by sending the X-Observe-Target-Package header. With custom routes, rules that you maintain in Observe by Snowflake inspect each observation and pick its destination.
| Package routing | Custom routes | |
|---|---|---|
| Destination chosen by | The package header the client sends. | Rules that match observation content. |
| Where you change it | Client configuration. | Ingest routing page or routing API. |
| Rules to maintain | None. Observe by Snowflake creates and updates them. | You write, order, and enable each rule. |
| Evaluation order | Before custom rules. | After managed routes, before the default route. |
| Data types | Every type except Generic - Any Data. | Every type except Entity. |
| Secondary destination | Not supported. | Supported. |
When to use package routing
- You are following integration setup instructions. Integrations such as the Observe Agent send
X-Observe-Target-Packageso that their data lands in the Datasets their apps and dashboards read. Keep the header. - Each client sends everything to one set of datasets. For example, give each team its own package and configure that team's collectors with its package name. The destination follows the client configuration, and nobody has to maintain rules.
- You want new datasets routed without extra steps. When you create a typed source dataset in a package, Observe by Snowflake creates the package route for it. It updates the route when you rename or disable the Dataset.
A package route requires exactly one enabled Source Dataset of each data type in the package. If a package holds two Datasets of the same type, Observe by Snowflake can't tell which one should receive the data, so it creates no package route for that type.
When to use custom routes
- The destination depends on the data. Route by severity, service, namespace, environment, or any other column the data type allows. For example, send debug logs to a dataset with short retention, as shown in Configure a data ingest routing rule.
- You want to change destinations without touching clients. Rules live in Observe by Snowflake, so you can add, reorder, or disable them without redeploying collectors.
- You need a copy in a second Dataset. Only custom rules can name a secondary destination.
- You send Generic - Any Data. Package routes filter on
meta, which Generic Datasets do not expose to routing. Observe by Snowflake therefore creates no package routes for them. - Your client cannot set custom HTTP headers.
Using both
You can use both methods in one account. For example, integration clients keep their package headers, and a separate standalone token without the header carries custom data through your rules.
Keep these interactions in mind:
- Package routes claim data first. An observation that carries the package header and matches a package route never reaches your custom rules. You cannot write a custom rule that splits package data by content.
- Removing the header moves the data. Once an integration's client stops sending the header, its data no longer lands in the package datasets that the integration's apps read. Before removing the header, write rules that send the data back to those datasets, or accept that those apps stop showing it.
- Custom rules apply to all unassociated traffic of their type. A rule written for one team's logs also matches another team's logs sent without a package header, unless its filter distinguishes them. Write filters on attributes that identify the source, such as a service or cluster name.
Before sending data
Inspect the route list for each data type you will send. Confirm that the desired rule is enabled and no earlier rule claims the data. Verify the default destination even if you expect every observation to match a rule.
A standalone token does not create a destination. With content routing enabled, failure to resolve a route produces an ingestion error rather than creating a dataset automatically. For example, the response may report unable to route data. See Troubleshoot data ingestion.
Supported data types
Each data type corresponds to a source Dataset schema and to the ingest endpoints that produce compatible data. You manage a separate list of routes for each type.
Most types also restrict which columns a routing expression may reference. The restriction keeps routing decisions on stable metadata rather than on payload fields that may not exist.
| Datasource type in Observe by Snowflake | API name | Ingest endpoint | Columns you can filter on |
|---|---|---|---|
| Logs - OpenTelemetry | otellogs | /v1/otel/v1/logs | body, attributes, resource_attributes, instrumentation_scope, fields, meta |
| Metrics - OpenTelemetry | otelmetrics | /v1/otel/v1/metrics | meta, metric |
| Metrics - Prometheus | prometheus | /v1/prometheus | meta |
| Generic - Any Data | any | /v1/* | FIELDS, EXTRA, BUNDLE_TIMESTAMP, OBSERVATION_KIND, DATASTREAM_TOKEN_ID |
Referencing a column outside the allowed set is rejected when you save the rule.
A route can only target a Dataset whose type matches the route's type. For example, you can't send OTel logs to a Prometheus Dataset.
Why some are limited tometaA trace filter can only reference
meta, not span content. Filtering on span content could send some spans of a trace to one destination and the rest to another, which breaks trace reassembly and span links. Likewise, Prometheus metric data points and Prometheus metric metadata must always route to the same Datasets, or else the metric information in Observe by Snowflake will be incorrect.
Configure a data ingest routing rule
Perform the following steps to configure a new ingest routing rule. This example routes debug logs to their own Dataset. You can then keep them for a shorter period than the rest of your logs in order to save storage costs. The example assumes an OTel Logs source dataset called "Logs - OpenTelemetry".
-
Select Data & integrations > Ingest routing.
-
In the Datasource type field, select Logs - OpenTelemetry.
-
Click Add rule.
-
Enter a filter expression for the data you want to route. For this example, enter
filter fields.severity_text = "DEBUG".The table of events shows your incoming OTel logs. Check the expression against real data as you type.

-
Use the tabs to check what your expression matches:
- Routed shows the events this rule sends to its destination.
- Source shows the original set of events.
- Remaining shows the events the rule does not match.
-
Open the Routed to drop-down and choose a destination. Select an existing Dataset, or select Create new destination to make a new one.

-
Name the new Dataset, select a retention period, then click Create. A short retention period suits debug logs.

-
Save the rule.
Enable your new rule
A new rule starts disabled, so saving it does not change where your data goes. Observe by Snowflake skips a disabled rule entirely, as though it did not exist.
Use the toggle in the rule's Enabled column when you are ready for the rule to take effect. Disable a rule to stop routing through it without deleting it.
Order your routes
Observe by Snowflake evaluates the routes for a data type from top to bottom and stops at the first match. Order therefore decides which Dataset receives data that more than one rule could match.
- Put narrow rules above broad ones. A broad rule placed first claims the data that a later narrower rule would have matched.
- Drag and drop to reorder. Reordering applies to every rule of that data type at once.
- New rules are added last. A new rule sits directly above the default route until you move it.
The default route
The default route catches observations that no other rule matched. Observe by Snowflake creates it with the first rule you add, provided a Dataset of that type already exists.
The default route behaves differently from the rules you write:
- It has no filter expression. It matches anything that reached it.
- It is always evaluated last, and cannot be reordered above another rule.
- It can't be deleted or disabled. You can change its destination Dataset.
To change where unmatched data lands, edit the default route's destination.
Send data to two Datasets
A rule can name a secondary destination in addition to its primary one. Observations matching the rule are written to both Datasets.
Both destinations must be source Datasets of the rule's data type. Use a secondary destination to send a copy to a second team. You can also validate a new Dataset before cutting over to it.
NoteA secondary destination duplicates the data rather than splitting it. This adds additional payload bytes to your ingest billing.
Routes that Observe by Snowflake manages for you
Your route list may include rules that Observe by Snowflake created and that you cannot edit, delete, or reorder. Managed rows show their controls greyed out.
These managed routes exist so that content-based routing reproduces the implicit routing you already had. Observe by Snowflake manages two kinds:
- Package routes match data sent with a target package HTTP header, such as the Observe Agent. They point at the source dataset that package already used.
- Token association routes match data from a single Datastream token. They point at the source Dataset that token was associated with.
Managed routes are always evaluated before the rules you write, so existing data flows keep landing where they did before.
Observe by Snowflake also reconciles these routes continuously. It rebuilds them whenever the underlying dataset name or token association changes. Any manual change would therefore be reverted.
To send data covered by a managed route elsewhere, change the underlying source Dataset or token association.
Automation
The routing management API is beta; its schema can change. Confirm availability and compatibility before relying on it in production automation. Use a management API token on the application host, not an ingest token on the collection host.
| Operation | Method and path | Required inputs or behavior |
|---|---|---|
| Inspect routes | GET /v1/ingest/routes/{type} | Returns routes in priority order, including disabled rules. |
| Create a rule | POST /v1/ingest/routes/{type} | JSON body with pipeline and destinationId; returns the created route and its ID. |
| Enable a rule | PATCH /v1/ingest/routes/{type}/{id} | JSON body {"enabled":true}. |
Use the API name from the supported types table for {type}. destinationId must be the Source Dataset ID, not a Datastream ID or Dataset name. Create destinations through the UI workflow described in Source Datasets before supplying their IDs.
New user-defined rules start disabled. After creation, inspect the full list, confirm the default destination, and enable the intended rule before sending test data. Managed routes cannot be changed through this API. The beta API currently documents trace routing as default-route-only; do not assume every UI operation has an equivalent supported API operation.
Updated 5 days ago