Exporting Query Results in CSV or JSON Format¶
Data in Observe can be queried programmatically using the Export API, /v1/meta/export/query
. You provide an OPAL query in a JSON-encoded POST request, and the API endpoint returns data in either CSV or newline-delimited JSON.
This tutorial demonstrates how to use the cURL command line utility to perform the following tasks:
Obtain an API token to use with the Export API.
Send a correctly formatted query request using the Export API.
APIs
POST https://$CUSTOMERID.observeinc.com/v1/meta/export/query
POST https://$CUSTOMERID.observeinc.com/v1/login
API reference documentation for /v1/meta/export/query
can be found here.
Health Check¶
To determine if the export API is healthy, you can use the following endpoint:
curl https://api.observe-eng.com/v1/health
The expected response is a JSON status and message.
{"ok":true,"message":"Everything's perfectly all right now. We're fine. We're all fine here, now, thank you. How are you?"}
Obtaining a Bearer Token from your Observe Instance¶
Before you begin exporting query results, generate an access token using cURL. You can generate the token using a cURL
request with an email and password combination and the /v1/login
endpoint. API tokens generated with the /v1/login
endpoint remain valid for 31 days after creation, with a rolling 10-day extension during active use.
For example, if you have a working login for an Observe instance such as a user email, [email protected]
, and password, password123
, create a bearer token using a command similar to the following example:
curl -H 'Content-Type: application/json' \
https://123456789012.observeinc.com/v1/login \
-d '{"user_email":"[email protected]", "user_password":"hunter1", "tokenName":"my testing token"}'
In the example, replace 123456789012.observeinc.com
with the hostname of your Observe instance, and replace the email and password with your login credentials.
If successful, the response should be similar to the following key:
{"ok":true, "access_key":"1abCDE2FgHIJKLMNoPqrstuV3WXYZA4bc"}
This access key can be used for authentication in subsequent API requests, using an HTTP Authorization
header formatted like the following:
Authorization: Bearer <CUSTOMER_ID> <ACCESS_KEY>
For example, Authorization: Bearer 123456789012 1abCDE2FgHIJKLMNoPqrstuV3WXYZA4bc
Creating a cURL Request to Export Query Results¶
Now that you have an access key, use the /v1/meta/export/query
API on an endpoint to dispatch a query.
A typical request using cURL to this endpoint looks like the following example:
curl https://123456789012.observeinc.com/v1/meta/export/query?interval=1h \
-H "Authorization: Bearer 123456789012 1abCDE2FgHIJKLMNoPqrstuV3WXYZA4bc \
-H 'Content-Type: application/json'\
-H 'Accept: text/csv' \
-d '
{
"query": {
"stages":[
{
"input":[
{
"inputName": "system",
"datasetId": "41001999"
}
],
"stageID":"main",
"pipeline":"statsby count:count()"
}
]
}
}
'
If successful, the response returns CSV similar to the following example:
"count"
"12276614"
Let’s review the example cURL invocation and dive into concepts for the Export API.
Time Range
Provide the time range for the query as a parameter in the URL; in this case, interval=1h
.
https://123456789012.observeinc.com/v1/meta/export/query?interval=1h
All queries in Observe are scoped to a time range. With the Export API, you can specify one or two of interval
, startTime
, or endTime
. Specify the startTime
and endTime
in the RFC 3339 date format, for example, YYYY-MM-DDTHH:MM:SSZ
, for 2023-03-21T15:01:23Z
. interval
consists of a number of seconds, minutes, or hours using s
, m
, or h
such as 1h
or 15m
, and used as a relative offset to either startTime
or endTime
.
Authorization Header
In the Export API, you must provide an Authorization Header
with a bearer token. Using the /v1/login
endpoint, you created a bearer token.
-H 'Authorization: Bearer 123456789012 1abCDE2FgHIJKLMNoPqrstuV3WXYZA3bc' \
Results Format
Select the format of the query results returned by the Export API when specifying an Accept
header in your HTTP
request. Use either one of the following types:
text/csv
returns a comma-separated values (RFC 4180) file.application/x-ndjson
returns one JSON object per line with new line delimiters.
-H 'Accept: text/csv' \
-H 'Accept: application/x-ndjson \
Query
The query is encoded in a JSON object similar to the following example:
{
"query": {
"stages": [
{
"input": [
{
"inputName": "system",
"datasetId": "41001999"
}
],
"stageID": "main",
"pipeline": "statsby count:count()"
}
]
}
}
The full specification of this object can be found in the reference documentation here.
The two most important parts of this object to replace are the datasetId
field, which identifies the dataset you want to use as the input for your OPAL query, and the pipeline
field, the OPAL query you want to send. The stageID
and inputName
attributes can be used to reference additional stages or datasets if your query requires multiple inputs, though most queries only reference a single input.
When viewing a dataset, you can determine the datasetId
of the dataset you want to query by inspecting the browser URL in the Observe instance. For example, suppose you opened the Container Logs dataset in your Observe account. You might see a URL like the following:
https://123456789012.observeinc.com/workspace/41010101/dataset/event/Container-Logs-41001999
The trailing number in the URL indicates the datasetId
, for example, 41001999
.
The pipeline
in the query JSON object contains the OPAL query you want to dispatch.
"pipeline": "statsby count:count()"
In this example, the query returns a single record with a count of rows in the dataset for the time range given in the request URL, for example, interval=1h
. See Observe Processing and Analysis Language for full language details.
Next Steps¶
If you have questions or need assistance with the API, join the Observe Slack channel.