Troubleshooting Data Ingestion

You may encounter errors when ingesting data into Observe. Use the System datastream to help diagnose issues with incoming data. This datastream contains many different types of observations about activity in your workspace. To investigate ingest issues, open the System datastream and filter for those with OBSERVATION_KIND ingest_error or click View in the last error column for ingest tokens in the Datastream Details page.

  1. From the left menu, open the Other menu and select the System datastream.

    ../../../_images/system-datastream.png

    Figure 1 - Opening the System datastream

  2. Filter for ingest_error observations.

    ../../../_images/system-datastream-ingest_error.png

    Figure 2 - Filtering for ingest errors

../../../_images/view-last-error.png

Figure 3 - Viewing the last error in the datastream

Types of Ingestion Errors

The System datastream FIELDS column contains additional information about each error. For example, this error indicates Observe received an observation too large to ingest:

{
  "error_type": "observation_too_large",
  "http_response_code": 400,
  "message": "observation of size 5166249 over limit of 4194304",
  "source_datastream_id": 123456789,

The property, error_type, describes the type of error. This may be one of the following types:

  • decode_error - Observe failed to decode your observations. Check the ingest error message for details. Check that your application(s) are emitting observations in the correct encoded format.

  • empty_body - The ingest HTTP request had no content.

  • internal_auth_error - The ingest HTTP request was not authorized. Check that your applications are using the correct credentials

  • method_not_allowed - The ingest HTTP request method (POST, GET, etc.) is not supported. Unless specifically stated in the endpoint documentation, the only method supported by Observe endpoints is the POST method.

  • no_observations_found - The ingest HTTP request was valid, but no observations were found.

  • observation_too_large - A submitted observation was over the size limit. Try to reduce the size of individual observations your applications are sending.

  • path_not_found - The ingest HTTP request was directed to an unsupported path. Check the HTTP request path.

  • poller_error - The poller may not have the correct permissions or it attempted to fetch records from a non-existing endpoint.

  • read_error - Observe was unable to read the ingest HTTP request content.

  • too_many_requests - Too many requests sent at the same time.

  • unknown_observation_decode_error - An unknown error occurred during ingest. Contact Observe Support.

  • unsupported_content_encoding - the Observe endpoint does not support the ingest HTTP request Content-Encoding field. For example, the Prometheus endpoint only supports a Content-Encoding type of snappy.

  • unsupported_media_type - The ingest HTTP request Content-Type field is not supported by the Observe endpoint. For example, sending an HTTP request with a Content-Type of application/x-protobuf to the Observe HTTP endpoint results in this error.

Resolving a decode_error

You may encounter a decode error while using the AWS Integration app.

{
		"error_type":"decode_error",
		"http_response_code":400,
		"message":"unable to decode observations",
		"source_datastream_id":0,
		"source_datastream_token_id":"",
		"source_http_request":{
		"headers":{
		  "content-encoding":"gzip",
		  "content-type":"application/json"
		},
		"id":"12a34b5c67d90e1f23456g78h9ijk0a1",
		"path":"/v1/http/aws/s3?bucket=observeo2-bucket-lce1o41vi8g&etag=%22c4842d2ed7207e8f80addcfbaa912b32%22&invokedFunctionArn=arn%3Aaws%3Alambda%3Aus-west-2%3A158067661102%3Afunction%3AObserveO2-lambda-MNEDH4OIX2HF&key=AWSLogs%2F158067661102%2FConfig%2Fus-west-2%2F2023%2F1%2F25%2FOversizedChangeNotification%2FAWS%3A%3ASSM%3A%3AManagedInstanceInventory%2Fi-0534eb8cc54ac76c7%2F158067661102_Config_us-west-2_ChangeNotification_AWS%3A%3ASSM%3A%3AManagedInstanceInventory_i-0534eb8cc54ac76c7_20230125T191305Z_1674673985309.json.gz&requestId=a0c7761b-38a0-48c4-8adf-0ecea29e297a",
		"start_time":1674674049907925064
  }
}

Consider upgrading your Lambda function to v1.0.20221214 or higher using one of three methods.

  • Using the AWS Console, perform the following step:

Update the existing CloudFormation stack withLambdaVersion=v1.0.20221214 or higher.

../../../_images/lambda-version.png

Figure 4 - Lambda Version

  • Using the AWS CLI, perform the following step:

Update the existing CloudFormation stack with LambdaVersion=v1.0.20221214 or higher.

aws cloudformation update-stack --stack-name ObserveLambda --use-previous-template \
--capabilities CAPABILITY_NAMED_IAM \
--parameters ParameterKey=ObserveCustomer,ParameterValue="${OBSERVE_CUSTOMER?}" ParameterKey=ObserveToken,ParameterValue="${OBSERVE_TOKEN?}" ParameterKey=LambdaVersion,ParameterValue="v1.0.20221214"

  • Using Terraform, override the lambda_version parameter in the observe_collection module:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = "${OBSERVE_CUSTOMER}"
  observe_token    = "${OBSERVE_TOKEN}"
  lambda_version = "v1.0.20221214"
}

Additional Error Code Examples

If you encounter the error, method_not_allowed, use POST as the method. Observe does not support GET and DELETE. Or use a different path that supports your selected method. Using the wrong path returns a NOT FOUND error.

{
  "error_type": "method_not_allowed",
  "http_response_code": 405,
  "message": "Method GET is not allowed",
  "source_datastream_id": 0,
  "source_datastream_token_id": "",
  "source_http_request": {
    "headers": {},
    "id": "cd18bd01990b79837ee6395c0ce81165",
    "path": "/v1/http/",
    "start_time": 1671143478484717800
  }

If the error observation_too_large occurs, consider reducing a record size.

{
  "error_type": "observation_too_large",
  "http_response_code": 400,
  "message": "observation of size 5233950 over limit of 4194304",
  "source_datastream_id": 41023194,
  "source_datastream_token_id": "ds1aTyFI2f8V5IkqXiiY",
  "source_http_request": {
    "headers": {
      "content-encoding": "gzip",
      "content-type": "application/x-protobuf"
    },
    "id": "204345502211d4eea054e5d09fbc92ca",
    "path": "/v1/otel/v1/traces",
    "start_time": 1673292866920775400
  }
}

If the error unsupported_media_type occurs, double-check the media type in your ingest request to Observe.

{
  "error_type": "unsupported_media_type",
  "http_response_code": 415,
  "message": "could not parse media type ",
  "source_datastream_id": 41023194,
  "source_datastream_token_id": "ds1qXE6GevEyFadhaEXR",
  "source_http_request": {
    "headers": {
      "content-type": ""
    },
    "id": "d42b01b1413756e7e78b63780607a571",
    "path": "/v1/http/",
    "start_time": 1671143616712639200
  }
}