Google Cloud Platform (GCP)

The Observe Google Cloud Platform (GCP) app streamlines the process of collecting data from GCP. A Pub/Sub topic makes logging and asset inventory information available and monitoring APIs expose metrics from several common GCP services at once. Observe pollers then ingest the data into your Observe environment.

The GCP app works with the datasets in your workspace. After setting up your GCP project for data collection, you can install and configure the GCP app in your Observe environment. Observe provides a Terraform module for setting up your GCP project. Contact us for assistance with completing these steps.

If you currently ingest GCP data, consult with Observe to see if the GCP app could enhance your existing data collection strategy.

What data does Observe ingest?

Standard ingest sources

The GCP app automatically ingests the following types of data from a single project:

Additional ingest sources

With these previous sources configured and working, add additional services by enabling them in the GCP app interface. Details for common services may be found in the Observe documentation:

Using GCP app data

Data coming into Observe through the GCP app populates datasets such as:

  • Asset Inventory

    • Asset Inventory Records - Raw data from asset exports

    • Resource Asset Inventory Records - All cloud assets in GCP that provide the basis for resource datasets

  • Metrics

    • Metrics - Periodically, typically every 60 seconds, sample metrics from GCP

    • Distribution Metrics - Services use this value type when the individual measurements become too numerous to collect, but statistical information, such as averages or percentiles, about those measurements, can be valuable.

  • Cloud Functions - Cloud Function resource dataset

  • Cloud SQL Instance - Cloud SQL resource dataset

  • Compute Instance - Compute resource dataset

  • Projects - Project resource dataset

  • Storage Buckets - Storage resource dataset

Setup Overview

Observe provides a Terraform module that creates service accounts, log sink, and a pub/sub topic as well as the subscription needed by Observe pollers for your GCP Project. (Observe Google Collection GitHub repository). You can also follow the instructions for the GCP Console tab in the Installation Section to manually set up your project.

Observe pollers, using the service account key you created, extract asset, logging, metric, and project data and send it into your Observe account at the interval you set.

Flow of data from GCP to Observe

Figure 1 - GCP Module Flow

The GCP app shapes and accelerates the data for monitoring and troubleshooting.

Installation

Prerequisites

Before proceeding with the GCP app install, ensure you configure your GCP project using either Terraform or GCP Console

Terraform

Instructions

Terraform automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the service account key which can be exported by the Terraform module and used by both the Monitoring and Pub/Sub pollers.

You need to set up the Google Cloud SDK and run gcloud auth login to create a credentials file for Terraform to use.

The following Terraform snippet installs the GCP collection stack for the project of the Google provider.

The enable_extensions property adds cloud functions for collecting additional information not included in asset inventory, logs or meterics:

provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source            = "observeinc/collection/google"
  name              = "dev" 
  enable_extensions = true
}

output "subscription" {
  description = "The Pub/Sub subscription created by this module."
  value       = module.observe_gcp_collection.subscription
}

output "service_account_private_key" {
  description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}
To access the service account key after running Terraform, use the

terraform output -raw service_account_private_key

JSON key for GCP integrations for Observe

Figure 2 - JSON key for GCP integrations into Observe

You need this for installation of the GCP App.

Also, make note of the Subscription created. You also need this for the GCP App installation.

terraform output -json subscription | jq .name

Within the GCP Console search for and enable the following APIs:

Baseline:

Service-specific APIs:

API setup for GCP integrations for Observe

Figure 3 - API Setup for GCP integrations into Observe

Once you create these resources, you can proceed with your GCP App configuration.

Most commonly, you use the Observe Apps install page to install and configure the GCP App. However, Observe can also provide the Terraform modules and providers necessary for this task. Please contact your Observe account manager for assistance.

GCP Console

Instructions

Within the GCP Console you need to perform the following tasks:

Create Service Accounts

Create a service account with the following details:

  • Service account name: my-observe-poller-service-account

  • Service account ID: my-observe-poller-service-id

  • Service account description: My Observe Pollers

Service Account setup for GCP integrations for Observe

Figure 4 - Service Account setup for GCP integrations for Observe

Grant this service account access to the project:

  • Role: Pub/Sub Subscriber (roles/pubsub.subscriber)

  • Role: Monitoring Viewer (roles/monitoring.viewer)

  • Role: Cloud Asset Viewer (roles/cloudasset.viewer)

  • Role: Browser (roles/browser)

Service Account roles for GCP integrations for Observe

Figure 5 - Service Account Roles for GCP integrations for Observe

Generate and download a service account key for the service account you just created:

  1. Click on the KEYS tab.

  2. Click ADD KEY.

  3. Select the Key Type as JSON.

  4. Click Create.

  5. Save the downloaded key for Observe GCP App Installation.

Service Account key setup for GCP integrations for Observe

Figure 6 - Service Account Key setup for GCP integrations for Observe

JSON key for GCP integrations for Observe

Figure 7 - Service Account Key setup for GCP integrations for Observe

Creating a Pub/Sub Topic
  1. Under Topics, click CREATE a TOPIC.

  2. Configure the following parameters:

    • Enter the Topic ID my-log-sink-topic.

    • Select Add a default subscription.

    • Under Encryption, select Google-managed encryption key.

    Pubsub topic setup for GCP integrations for Observe

Figure 7 - Service Account Key setup for GCP integrations for Observe

On the Subscriptions tab, notice that GCP automatically created a Pub/Sub subscription. You need this information for GCP App setup.

Pub/Sub topic setup for GCP integrations for Observe

Figure 8 - Pub/Sub topic setup for GCP integrations for Observe

Create A Log Sink

Configure a Log Sink to publish to the Pub/Sub topic.

  1. Under Logs Router, click Create sink.

  2. Under Sink details, add the following information:

    • Enter my-observe-log-sink as the Sink name.

    • Enter For my Observe pubsub topic for the Sink description.

    Log sink setup for GCP integrations for Observe

Figure 9 - Log Sink setup for GCP integrations for Observe

  1. Under Sink destination:

    • Select Sink Service Cloud Pub/Sub topic.

    • Choose the topic, under your project, that ends with your sink name. Based on the previous step, it is my-log-sink-topic.

    Log sink topic setup for GCP integrations for Observe

    Figure 10 - Log Sink topic setup for GCP integrations for Observe

    Log sink filters for GCP integrations for Observe

    Figure 11 - Log Sink filters for GCP integrations for Observe

3. Click Create Sink.

Add cloud function(s) to collect additonal data

After you have created a pubsub topic you can use cloud functions to extend your information collection for data not automatically exported with asset inventory, metrics and logging. You will need to add cloud functions to collect compute instance group and cloud scheduler information.

We use python code based functions in this instance but you can develop further extensions using any ;anguage supported by gcp cloud functions.

1. Create a service account for cloud function execution with the following roles:

  • Storage Object Viewer

  • Pub/Sub Publisher

  • Compute Viewer

  • View Service Accounts

  • Cloud Scheduler Viewer

    Service Account for Cloud Functions

    Figure XX - Create service account

    Service Account for Cloud Functions

    Figure XX - Add roles

    Service Account for Cloud Functions

    Figure XX - Save service account

2. Create a cloud function for collecting and publishing data

Python code for functions can be found in (Observe Google Collection GitHub repository extensions src folder).

  1. Create function named “manual-list-service-groups-function” and click save

Create Cloud Functions

Figure XX - Create cloud function

  1. Set “Runtime service account” to the service account you create in step 1 and add PROJECT_ID and TOPIC_ID environment variables set to your current project and the topic created in the previous steps.

Create Cloud Functions

Figure XX - Set service account and environment variables

Copy code from main.py file in Observe Google Collection GitHub repository extensions src folder to main.py in console. Set the entry point to the list_instance_group function.

Create Cloud Functions

Figure XX - Add python code

Copy code from requirements.txt file in Observe Google Collection GitHub repository extensions src folder to requirements.txt in console.

Create Cloud Functions

Figure XX - Add python requirements

Create Cloud Functions

Figure XX - YYYYYY

Click Deploy.

Check for succesful deployment and test function

Create Cloud Functions

Figure XX - Test deployment

  1. Copy function, change entry point to list_cloud_scheduler_jobs and test

Create Cloud Functions

Figure XX - Copy Function

Create Cloud Functions

Figure XX - Rename

Create Cloud Functions

Figure XX - Change Entrypoint

Create Cloud Functions

Figure XX - Deploy and test copied function

Create cloud scheduler job

Create Cloud Functions

Figure XX - Create cloud scheduler service account

Create Cloud Functions

Figure XX - Add cloud function invoker role

Create Cloud Functions

Figure XX - Add cloud function invoker role

Create Cloud Functions

Figure XX - Create cloud scheduler job

Create Cloud Functions

Figure XX - Configure cloud scheduler job

Create Cloud Functions

Figure XX - Add OIDC token

Create Cloud Functions

Figure XX - Force job run

Repeat cloud scheduler job creation for as many functions as you want to trigger.

Enabling APIs

Within the GCP Console search for and enable the following APIs:

Baseline:

Service-specific:

API setup for GCP integrations for Observe

Figure 12 - API Setup for GCP integrations for Observe

After you create the resources and enable the APIs, you can proceed with the GCP app configuration.

GCP App

Note

Although Observe allows you to control the amount of data consumed from GCP, the consumption of assets, logs, and metrics data does incur costs. See Google Cloud’s Operations Suite Pricing for more information.

In order to proceed with the GCP app install, you need to enter the service account private key associated with a GCP service account created in Prerequisites.

As a reminder - the service account must have the following permissions:

  • Role: Pub/Sub Subscriber (roles/pubsub.subscriber)

  • Role: Monitoring Viewer (roles/monitoring.viewer)

  • Role: Cloud Asset Viewer (roles/cloudasset.viewer)

  • Role: Browser (roles/browser)

Installation

  • Go to the Apps configuration page in Observe.

GCP app for Observe

Figure 13 - Apps on Observe

  1. Select the GCP app.

  2. Click Install.

  3. Chose one of the two options:

  • Recommended - installs the recommended Observe content which can be modified later. Click Continue to proceed.

  • Manual Install - allows you to customize the Observe datastream used and app content installed. Click Continue to proceed.

List of available GCP services on Observe

Figure 14 - List of available GCP services on Observe

Creating the required connections to GCP

  1. Click on the Connections tab.

Connection for GCP integrations for Observe

Figure 15 - Creating the connection for GCP Integrations

  1. Next to Onboard asset inventory records and metrics for a GCP project, click Create Connection.

  2. For the GCP Project ID, enter your Google Project ID.

  3. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform or downloaded from the GCP Console.

    Create Poller for GCP integrations with Observe

Figure 16 - Creating the poller for GCP Integrations

  1. Next to Onboard log data using a GCP Pub/Sub Subscription, click Create Connection.

  2. For the GCP project ID, enter your Google Project ID.

  3. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform or downloaded from the GCP Console.

  4. For the GCP Pub/Sub Subscription, enter the Subscription name you noted in the Terraform or GCP Console setup.

    Pub/Sub poller for Pub/Sub subscriptions  for Observe

    Figure 17 - Creating the poller for Pub/Sub subscriptions

  5. Verify that Observe ingests the GCP data.

    P

    Figure 18 - Verify the connections