Google Cloud Platform (GCP) Pub/Sub

Google Cloud Platform (GCP) Pub/Sub allows you to reliably deliver streaming data from multiple sources within GCP. Observe supports ingesting data through a variant of the HTTP endpoint.

Authorization

GCP Pub/Sub supports pushing data in JSON format directly over HTTP, but does not support basic authentication or configuring authorization headers for outbound requests. You must include your credentials in the URI as a base64-encoded string.

Configuration

Navigate to the GCP Cloud Pub/Sub console to create a Pub/Sub push subscription. Note that the configuration requires some parameters and others are optional.

  1. Select Subscriptions.

GCP console, search for "pubsub" to reach the Pub/Sub page.

Figure 1 - The Google Cloud Pub/Sub page

2. Go to the Pub/Sub page in the GCP console.

3. Select Subscriptions in the left pane, and then click CREATE SUBSCRIPTION.

4. Configure the required subscription details.

At a minimum, you must configure these required options:

  • Subscription ID and Pub/Sub topic - Enter a Subscription ID and a topic. You may choose an existing topic from the menu, or enter a new one.

  • Delivery type

    Subscription configuration page, Delivery type section.

    Figure 2 - Delivery Type

    The Delivery type Endpoint URL requires a Basic Authentication token as part of the URL. Construct this token using your Observe Customer ID and ingest token.

5. Select Push.

6. Create a Basic Authorization token by Base64-encoding a string containing the customer ID and ingest token separated by a colon :.

Bash on MacOS and Linux:

       $ echo -n "12345:my_ingest_key" | base64
       MTIzNDU6bXlfaW5nZXN0X2tleQ==

PowerShell on Windows:

       > [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("12345:my_ingest_key"))
       MTIzNDU6bXlfaW5nZXN0X2tleQ==

7. Enter the Endpoint URL https://pubsub.collect.observeinc.com/, appending the Base64 token you created in the previous step.

  https://pubsub.collect.observeinc.com/MTIzNDU6bXlfaW5nZXN0X2tleQ==

8. Leave the Enable authentication box unchecked. This option is for authentication between GCP services, not external ones.

9. Message retention duration - Select your desired retention period. To reduce storage costs, Observe recommends no more than 24 hours retention period. This also determines how long an outage your subscription can recover from.

10. Acknowledgement deadline - Choose a deadline time of at least 60 seconds.

11. Retry policy - Select Retry after exponential backoff delay, with the default minimum of 10 seconds and a maximum of 600 seconds.

12. Configure optional subscription details.

You may also configure these optional settings:

  • Expiration period

  • Subscription filter

  • Dead lettering

Message ordering is not required.

13. When you finish, click Create subscription.

FAQ

Retry on failure

GCP Pub/Sub supports exponential backoff for retries. See the Pub/Sub Handling message failures documentation for more information.