Google Cloud Pub/Sub

Google Cloud Pub/Sub allows you to reliably deliver streaming data from multiple sources within GCP. Observe supports ingesting data through a variant of our HTTP endpoint.

Authorization

Google Cloud Pub/Sub supports pushing data in JSON format directly over HTTP, but does not support basic authentication nor configuring authorization headers for outbound requests. Your credentials must therefore be included the URI as a base64-encoded string.

Configuration

Navigate to the GCP Cloud Pub/Sub console to create a Pub/Sub push subscription. Note that some options are required and others are optional.

  1. Select subscriptions

    GCP console, search for "pubsub" to reach the Pub/Sub page.
    • Go to the Pub/Sub page in the GCP console.

    • Select Subscriptions in the left pane, and then click CREATE SUBSCRIPTION.

  2. Configure required subscription details

    At minimim, you must configure these required options:

    • Subscription ID and Pub/Sub topic: Enter a Subscription ID and a topic. You may choose an existing topic from the menu, or enter a new one.

    • Delivery type

      Subcription configuration page, Delivery type section.

      The Delivery type Endpoint URL requires a Basic Authentication token as part of the URL. Construct this token using your Observe Customer ID and ingest token.

      • Select Push

      • Create a Basic Authorization token by Base64 encoding a string containing the customer ID and ingest token separated by a colon :.

        Bash on MacOS and Linux:

        $ echo -n "12345:my_ingest_key" | base64
        MTIzNDU6bXlfaW5nZXN0X2tleQ==
        

        PowerShell on Windows:

        > [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("12345:my_ingest_key"))
        MTIzNDU6bXlfaW5nZXN0X2tleQ==
        
      • Enter the Endpoint URL https://collect.observeinc.com/v1/pubsub, appending the Base64 token you created in the previous step.

        https://collect.observeinc.com/v1/pubsub/MTIzNDU6bXlfaW5nZXN0X2tleQ==
        
      • Leave the Enable authentication box unchecked. This option is for authentication between GCP services, not external ones.

    • Message retention duration: Select your desired retention period. To reduce storage cost, we recommend no more than 24 hours. This also determines how long an outage your subscription can recover from.

    • Acknowledgement deadline: Choose a deadline time of at least 60 seconds.

    • Retry policy: Select Retry after exponential backoff delay, with the default minimum of 10 seconds and maximum of 600 seconds.

  3. Configure optional subscription details

    You may also configure these optional settings:

    • Expiration period

    • Subscription filter

    • Dead lettering

    Message ordering is not required.

  4. When you are finished, click Create subscription.

FAQ

Retry on failure

Google Cloud Pub/Sub supports exponential backoff for retries. See the Pub/Sub Handling message failures documentation for more information.