Troubleshooting Data Ingestion

You may encounter errors when ingesting data into Observe. Use the System datastream to help diagnose issues with incoming data. This datastream contains many different types of observations about activity in your workspace.

Ingest issues may be seen when reviewing or managing Datastreams or Tokens.

  1. From the left side navigation menu, click Datastreams and review the “Health” column of the “System” Datastream row. If it says “Error”, hover over the row to get the context menu, and click the “Open dataset” button.

You can also go to System data directly and filter for errors.

  1. From the left side navigation menu, click Datasets, type “System” in the search bar, and select the System datastream. This opens the Dataset page for the System datastream.

  2. Type ingest_error in the filter bar and select OBSERVATION_KIND to get a filter of OBSERVATION_KIND = ingest_error.

Types of Ingestion Errors

The System datastream FIELDS column contains additional information about each error. To filter by these, click the “FIELDS” column header, click Extract from JSON, select error_type, and click Apply. This will open a Worksheet with the column extracted; click the “error_type” column header and filter to the values you want to review. You can also extract more columns and save this worksheet, create a monitor or dashboard, or close the worksheet tab without saving.

The error_type may be one of the following:

  • decode_error - Observe failed to decode your observations. Check the ingest error message for details. Check that your application(s) are emitting observations in the correct encoded format.

  • empty_body - The ingest HTTP request had no content.

  • internal_auth_error - The ingest HTTP request was not authorized. Check that your applications are using the correct credentials

  • method_not_allowed - The ingest HTTP request method (POST, GET, etc.) is not supported. Unless specifically stated in the endpoint documentation, the only method supported by Observe endpoints is the POST method.

  • no_observations_found - The ingest HTTP request was valid, but no observations were found.

  • observation_too_large - A submitted observation was over the size limit. Try to reduce the size of individual observations your applications are sending.

  • path_not_found - The ingest HTTP request was directed to an unsupported path. Check the HTTP request path.

  • poller_error - The poller may not have the correct permissions or it attempted to fetch records from a non-existing endpoint.

  • read_error - Observe was unable to read the ingest HTTP request content.

  • too_many_requests - Too many requests sent at the same time.

  • unknown_observation_decode_error - An unknown error occurred during ingest. Contact your Observe Data Engineer.

  • unsupported_content_encoding - the Observe endpoint does not support the ingest HTTP request Content-Encoding field. For example, the Prometheus endpoint only supports a Content-Encoding type of snappy.

  • unsupported_media_type - The ingest HTTP request Content-Type field is not supported by the Observe endpoint. For example, sending an HTTP request with a Content-Type of application/x-protobuf to the Observe HTTP endpoint results in this error.

Resolving a decode_error

You may encounter a decode error while using the AWS Integration app.

{
		"error_type":"decode_error",
		"http_response_code":400,
		"message":"unable to decode observations",
		"source_datastream_id":0,
		"source_datastream_token_id":"",
		"source_http_request":{
		"headers":{
		  "content-encoding":"gzip",
		  "content-type":"application/json"
		},
		"id":"12a34b5c67d90e1f23456g78h9ijk0a1",
		"path":"/v1/http/aws/s3?bucket=observeo2-bucket-lce1o41vi8g&etag=%22c4842d2ed7207e8f80addcfbaa912b32%22&invokedFunctionArn=arn%3Aaws%3Alambda%3Aus-west-2%3A158067661102%3Afunction%3AObserveO2-lambda-MNEDH4OIX2HF&key=AWSLogs%2F158067661102%2FConfig%2Fus-west-2%2F2023%2F1%2F25%2FOversizedChangeNotification%2FAWS%3A%3ASSM%3A%3AManagedInstanceInventory%2Fi-0534eb8cc54ac76c7%2F158067661102_Config_us-west-2_ChangeNotification_AWS%3A%3ASSM%3A%3AManagedInstanceInventory_i-0534eb8cc54ac76c7_20230125T191305Z_1674673985309.json.gz&requestId=a0c7761b-38a0-48c4-8adf-0ecea29e297a",
		"start_time":1674674049907925064
  }
}

Consider upgrading your Lambda function to v1.0.20221214 or higher using one of three methods.

  • Using the AWS Console, perform the following step:

Update the existing CloudFormation stack withLambdaVersion=v1.0.20221214 or higher.

Lambda Version

Figure 4 - Lambda Version

  • Using the AWS CLI, perform the following step:

Update the existing CloudFormation stack with LambdaVersion=v1.0.20221214 or higher.

aws cloudformation update-stack --stack-name ObserveLambda --use-previous-template \
--capabilities CAPABILITY_NAMED_IAM \
--parameters ParameterKey=ObserveCustomer,ParameterValue="${OBSERVE_CUSTOMER?}" ParameterKey=ObserveToken,ParameterValue="${OBSERVE_TOKEN?}" ParameterKey=LambdaVersion,ParameterValue="v1.0.20221214"

  • Using Terraform, override the lambda_version parameter in the observe_collection module:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = "${OBSERVE_CUSTOMER}"
  observe_token    = "${OBSERVE_TOKEN}"
  lambda_version = "v1.0.20221214"
}

Additional Error Code Examples

If you encounter the error, method_not_allowed, use POST as the method. Observe does not support GET and DELETE. Or use a different path that supports your selected method. Using the wrong path returns a NOT FOUND error.

{
  "error_type": "method_not_allowed",
  "http_response_code": 405,
  "message": "Method GET is not allowed",
  "source_datastream_id": 0,
  "source_datastream_token_id": "",
  "source_http_request": {
    "headers": {},
    "id": "cd18bd01990b79837ee6395c0ce81165",
    "path": "/v1/http/",
    "start_time": 1671143478484717800
  }

If the error observation_too_large occurs, consider reducing a record size.

{
  "error_type": "observation_too_large",
  "http_response_code": 400,
  "message": "observation of size 5233950 over limit of 4194304",
  "source_datastream_id": 41023194,
  "source_datastream_token_id": "ds1aTyFI2f8V5IkqXiiY",
  "source_http_request": {
    "headers": {
      "content-encoding": "gzip",
      "content-type": "application/x-protobuf"
    },
    "id": "204345502211d4eea054e5d09fbc92ca",
    "path": "/v1/otel/v1/traces",
    "start_time": 1673292866920775400
  }
}

If the error unsupported_media_type occurs, double-check the media type in your ingest request to Observe.

{
  "error_type": "unsupported_media_type",
  "http_response_code": 415,
  "message": "could not parse media type ",
  "source_datastream_id": 41023194,
  "source_datastream_token_id": "ds1qXE6GevEyFadhaEXR",
  "source_http_request": {
    "headers": {
      "content-type": ""
    },
    "id": "d42b01b1413756e7e78b63780607a571",
    "path": "/v1/http/",
    "start_time": 1671143616712639200
  }
}