Google Cloud Platform (GCP) App

The Observe Google Cloud Platform (GCP) app streamlines the process of collecting data from GCP. A Pub/Sub topic makes logging and asset inventory information available and monitoring APIs expose metrics from several common GCP services at once. Observe pollers then ingest the data into your Observe environment.

The GCP app works with the datasets in your workspace. After setting up your GCP project for data collection, you can install and configure the GCP app in your Observe environment. Observe provides a Terraform module for setting up your GCP project. Contact us for assistance with completing these steps.

If you currently ingest GCP data, consult with Observe to see if the GCP app could enhance your existing data collection strategy.

What GCP Data Does Observe Ingest?

Standard Ingestion Sources

The GCP app automatically ingests the following types of data from a single project:

Additional Ingestion Sources

With these previous sources configured and working, add additional services by enabling them in the GCP app interface. Details for common services may be found in the Observe documentation:

Using the GCP App Data

Data coming into Observe through the GCP app populates datasets such as the following:

  • Asset Inventory

    • Asset Inventory Records - Raw data from asset exports

    • Resource Asset Inventory Records - All cloud assets in GCP that provide the basis for resource datasets

  • Metrics

    • Metrics - Periodically, typically every 60 seconds, sample metrics from GCP

    • Distribution Metrics - Services use this value type when the individual measurements become too numerous to collect, but statistical information, such as averages or percentiles, about those measurements, can be valuable.

  • Resource datasets

    • Cloud Functions

    • Cloud SQL Instance

    • Compute Instance

    • Projects

    • Storage Buckets

Setup Overview

Observe provides a Terraform module that creates service accounts, log sink, and a pub/sub topic as well as the subscription needed by Observe pollers for your GCP Project. (Observe Google Collection GitHub repository). You can also follow the instructions for the Deployment Manager tab or GCP console tab in the Installation Section to provision those resources with the Google Cloud Deployment Manager.

Observe pollers, using your created service account key, extract assets, logs, metrics, and project data and send it into your Observe account at a set interval.

Flow of data from GCP to Observe

Figure 1 - GCP Module Flow

The GCP app shapes and accelerates the data for monitoring and troubleshooting.

Installation

Prerequisites

Before proceeding with the GCP app install, ensure you configure your GCP project using either Terraform, Deployment Manager, or the GCP console.

Terraform automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the service account key, which can be exported by the Terraform module and used by both the Monitoring and Pub/Sub pollers.

Here are the steps for using Terraform:

  1. Install Terraform if needed.

  2. Set up the Google Cloud SDK and run gcloud auth login to create a credentials file for Terraform to use.

  3. Within the GCP Console search for and enable the following APIs:

    API setup for GCP integrations for Observe

    Figure 2 - API Setup for GCP integrations into Observe

4. Create a Terraform module. The following Terraform snippet installs the GCP collection stack for the project of the Google provider.

    provider "google" {
      project = "YOUR_PROJECT_ID"
      region  = "YOUR_DEFAULT_REGION"
    }

    module "observe_gcp_collection" {
      source   = "observeinc/collection/google"
      name     = "dev"
      resource = "projects/YOUR_PROJECT_ID"
    }

    output "subscription" {
      description = "The Pub/Sub subscription created by this module."
      value       = module.observe_gcp_collection.subscription
    }

    output "service_account_private_key" {
      description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
      value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
      sensitive   = true
    }

5. Run terraform apply.

6. Record information needed for installation of the GCP App.

To access the service account key after running Terraform, use the

terraform output -raw service_account_private_key

JSON key for GCP integrations for Observe

Figure 3 - JSON key for GCP integrations into Observe

Also, make a note of the created Subscription. You need this for the GCP App installation.

To access the service account key after running Terraform, use the

terraform output -raw service_account_private_key

Once you create these resources, you can proceed with your GCP App configuration.

Most commonly, you use the Observe Apps install page to install and configure the GCP App. However, Observe can also provide the Terraform modules and providers necessary for this task. Please contact your Observe account manager for assistance.

Deployment Manager automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the Service Account Key which you output from the Deployment Manager configuration and used by both the Monitoring and Pub/Sub pollers.

Use the following steps for Deployment Manager:

  1. Within the GCP Console search for and enable the following APIs:

API setup for GCP integrations for Observe

Figure 4 - API Setup for GCP integrations into Observe

2. Log into the GCP console, and provide the [PROJECT_NUMBER]@cloudservices.gserviceaccount.com service account for the Logging Configuration Writer and Security Admin roles. Deployment Manager uses this service account.

Service Account setup for GCP Deployment Manager

Figure 5 - Service Account setup for GCP integrations for Observe

Also ensure that [PROJECT_NUMBER]@cloudbuild.gserviceaccount.com service account exists with the Cloud Build Service Account role.

3. Using a terminal with gcloud configured, such as GCP Cloud Shell, use the Deployment Manager template:

    gcloud beta deployment-manager deployments create observe \
      --template https://storage.googleapis.com/observeinc/deploymentmanager-google-collection-latest.py \
      --properties "resource:'projects/$YOUR_PROJECT_ID'"

Template versions can be found by running the following command:

 gcloud storage ls gs://observeinc/

You can find the change log at CHANGELOG.

4. Using the terminal, run the following command to see the output.

 gcloud beta deployment-manager manifests describe --deployment observe

5. You need the subscription ID and private key in the finalLayout field for GCP App installation.

...
    layout: |
      resources:
      - name: deploymentmanager-google-collection-latest-py
        outputs:
        - finalValue: observe-123456
          name: project_id
          value: observe-123456
        - finalValue: observe
          name: subscription_id
          value: observe
        - finalValue: ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAidzI2MS1maW5hbC1uYXRoYW4tMjM2MTA3IiwKICAicHJpdmF0ZV9rZXlfaWQiOiAiMzFlMzNmOWU1OWJkZjIyYTlmNzU2MjE5Y2M3MWUxNWVmZGNmMzU3YyIsCiAgInByaXZhdGVfa2V5IjogIi0tLS0tQkVHSU4gUFJJVkFURSBLRVktLS0tLVxuTUlJRXZnSUJBREFOQmdrcWhraUc5dzBCQVFFRkFBU0NCS2d3Z2dTa0FnRUFBb0lCQVFEa2tRSWlRcHl3azVjTFxuV3BGQ3dka3QzMUE4azB1T0piR0NSYi9xVnV0YUFnaU9iWU5nQXpOM3BVdHZhMlIzUzVDdkk3RStqNFhyRS9BWVxuTlBVL0xYWjR3ckk1bjY4b0ZRenE3MmpsNWl5QUJOblR2K0xQcUlGMW9Wc2xDblRtaHEyR2xpaFlrdzJtWXYveVxuWVlLSW5pNm5KSTlqTXdxekZQcUh6clBCUjZxVi94Y1BsSWdjTlZBbEZYZ0d4bjlXb1J5MUhRSWQ4NWNDaGdGdlxuTXo2alg0Wm1WN1MzQzhLekRkRXYySlJLSVZLaG9oS0NKQVo2YWtRWTJLMHdNUDI2WU9YeFlYcWtaZFV2UEhNTVxuM0NhYzNmQXduQ2hjTS9VMzVVZ3RsdmVVdndYM0JNRlBTTHpTajBmNHdVc0RxZXp5K0c4cHphS0ExQm40bEo2Z1xubnRvdkNnRlZBZ01CQUFFQ2dnRUFPT1NqYkZXNlhQV1AzbUtmejhjaXdSbGFCb24vZjVXQVZ0WDN6R1B3VjBvMFxuU1FlWGNQbTdBT2dqRzh2V2lSOUZGMk5DQmptQmN4OVdYeUFuZjU5WVl2ekxraTVlVWhMVFVWTmdnQUZJU0dGc1xudXhtRFluOG1JbHFVcm1hbzdJelBlclc3dXpoM2FGZ3lMMm40TFB0NnoxMzVWdzdvZGVVZDYxb1hRbUFOWlNKT1xuWjBNYWx2N1UzUlhOa0ZqNHVYWFZVRS84OTNpdFRGY0ZKTGJEeFhHejF4WGNUckJqK2MwMllSSmRLZm9xZkRLZ1xuZXQ3c25yeXpaVEhUcVdzaDNsd0ZrTWhVcUZubEptNmNEM0tCc3l4UGpkSHF2UjJVNjNuWVd2Q3pnUHRwbjdUaVxuc3l2QXZCQWFvb0kwR0cweGJhSk5XQzhmTitBVVhjY3VMNFRSQW9MYmpRS0JnUUQ1ZEJKYldMcy8ySXF4WGdnUVREDACTED
          name: poller_private_key_base64
          value: $(ref.google_service_account_key-poller.privateKeyData)
 ...

6. Use the following command to decode the private key for GCP App installation.

echo -n <YOUR_PRIVATE_KEY> | base64 --decode

echo -n
ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAidzI2MS1maW5hbC1uYXRoYW4tMjM2MTA3IiwKICAicHJpdmF0ZV9rZXlfaWQiOiAiMzFlMzNmOWU1OWJkZjIyYTlmNzU2MjE5Y2M3MWUxNWVmZGNmMzU3YyIsCiAgInByaXZhdGVfa2V5IjogIi0tLS0tQkVHSU4gUFJJVkFURSBLRVktLS0tLVxuTUlJRXZnSUJBREFOQmdrcWhraUc5dzBCQVFFRkFBU0NCS2d3Z2dTa0FnRUFBb0lCQVFEa2tRSWlRcHl3azVjTFxuV3BGQ3dka3QzMUE4azB1T0piR0NSYi9xVnV0YUFnaU9iWU5nQXpOM3BVdHZhMlIzUzVDdkk3RStqNFhyRS9BWVxuTlBVL0xYWjR3ckk1bjY4b0ZRenE3MmpsNWl5QUJOblR2K0xQcUlGMW9Wc2xDblRtaHEyR2xpaFlrdzJtWXYveVxuWVlLSW5pNm5KSTlqTXdxekZQcUh6clBCUjZxVi94Y1BsSWdjTlZBbEZYZ0d4bjlXb1J5MUhRSWQ4NWNDaGdGdlxuTXo2alg0Wm1WN1MzQzhLekRkRXYySlJLSVZLaG9oS0NKQVo2YWtRWTJLMHdNUDI2WU9YeFlYcWtaZFV2UEhNTVxuM0NhYzNmQXduQ2hjTS9VMzVVZ3RsdmVVdndYM0JNRlBTTHpTajBmNHdVc0RxZXp5K0c4cHphS0ExQm40bEo2Z1xubnRvdkNnRlZBZ01CQUFFQ2dnRUFPT1NqYkZXNlhQV1AzbUtmejhjaXdSbGFCb24vZjVXQVZ0WDN6R1B3VjBvMFxuU1FlWGNQbTdBT2dqRzh2V2lSOUZGMk5DQmptQmN4OVdYeUFuZjU5WVl2ekxraTVlVWhMVFVWTmdnQUZJU0dGc1xudXhtRFluOG1JbHFVcm1hbzdJelBlclc3dXpoM2FGZ3lMMm40TFB0NnoxMzVWdzdvZGVVZDYxb1hRbUFOWlNKT1xuWjBNYWx2N1UzUlhOa0ZqNHVYWFZVRS84OTNpdFRGY0ZKTGJEeFhHejF4WGNUckJqK2MwMllSSmRLZm9xZkRLZ1xuZXQ3c25yeXpaVEhUcVdzaDNsd0ZrTWhVcUZubEptNmNEM0tCc3l4UGpkSHF2UjJVNjNuWVd2Q3pnUHRwbjdUaVxuc3l2QXZCQWFvb0kwR0cweGJhSk5XQzhmTitBVVhjY3VMNFRSQW9MYmpRS0JnUUQ1ZEJKYldMcy8ySXF4WGdnUVREDACTED | base64
   --decode

Within the GCP Console, you need to perform the following tasks:

Create Service Accounts

  1. Create a Service account with the following details:

    • Service account name: my-observe-poller-service-account

    • Service account ID: my-observe-poller-service-id

    • Service account description: My Observe Pollers

Service account setup for GCP integrations for Observe

Figure 6 - Service Account setup for GCP integrations for Observe

2. Grant this service account access to the project:

  • Role: Pub/Sub Subscriber (roles/pubsub.subscriber)

  • Role: Monitoring Viewer (roles/monitoring.viewer)

  • Role: Cloud Asset Viewer (roles/cloudasset.viewer)

  • Role: Browser (roles/browser)

Service Account roles for GCP integrations for Observe

Figure 7 - Service Account Roles for GCP integrations with Observe

3. Generate and download a Service account key for the Service account you just created:

  1. Click on the KEYS tab.
  2. Click ADD KEY.
  3. Select the Key type as JSON.
  4. Click Create.

4. Save the key for Observe GCP App Installation.

Service Account key setup for GCP integrations for Observe

Figure 8 - Service Account Key setup for GCP integrations for Observe

JSON key for GCP integrations for Observe

Figure 9 - Service Account Key setup for GCP integrations for Observe

Creating a Pub/Sub Topic

  1. Under Topics, click Create a topic.

  2. Configure the following parameters:

    1. Enter the Topic ID my-log-sink-topic.
    2. Select Add a default subscription.
    3. Under Encryption, select Google-managed encryption key.

3. Click CREATE TOPIC.

Pubsub topic setup for GCP integrations for Observe

Figure 10 - Service Account Key setup for GCP integrations for Observe

On the Subscriptions tab, you see that GCP automatically created a Pub/Sub subscription. You need this information for GCP App setup.

Pub/Sub topic setup for GCP integrations for Observe

Figure 11 - Pub/Sub topic setup for GCP integrations for Observe

Creating a Log Sink

Configure a Log Sink to publish to the Pub/Sub topic.

  1. Under Logs Router, click Create sink.

2. Under Sink details, add the following information:

  1. Enter my-observe-log-sink as the Sink name.
  2. Enter For my Observe pubsub topic for the Sink description.
Log sink setup for GCP integrations for Observe

Figure 12 - Log Sink setup for GCP integrations with Observe

3. Under the Sink destination, configure the following parameters:

  1. Select Sink Service Cloud Pub/Sub topic.
  2. Choose the topic under your project that ends with your Sink name. Based on the previous step, use my-log-sink-topic.

4. Click CREATE SINK.

Log sink topic setup for GCP integrations for Observe

Figure 13 - Log Sink topic setup for GCP integrations for Observe

5. Create an inclusion filter to determine which logs you want to include in the Sink.

Log sink filters for GCP integrations for Observe

Figure 14 - Log Sink filters for GCP integrations for Observe

6. Click Create Sink.

Adding GCP Cloud functions to collect additonal data

After you create a PubSub topic, you can use GCP Cloud functions to extend your information collection of data not automatically exported with asset inventory, metrics, and logging. You need to add GCP Cloud functions to collect compute instance group and cloud scheduler information.

Observe uses Python code-based functions in this instance, but you can develop further extensions using any language supported by GCP Cloud functions.

  1. Create a Service Account by entering the following parameters:

    • Pub/Sub Publisher

    • Compute Viewer

    • View Service Accounts

    • Cloud Scheduler Viewer

    • Browser

    • Cloud Asset Viewer

2. Click CREATE AND CONTINUE.

Service Account for Cloud Functions

Figure 15 - Create service account

3. Add the following roles for GCP Cloud function execution:

  • Storage Object Viewer

  • Pub/Sub Publisher

  • Compute Viewer

  • View Service Accounts

  • Cloud Scheduler Viewer

Service Account for Cloud Functions

Figure 16 - Add roles

4. Click DONE to save the Service Account.

Service Account for Cloud Functions

Figure 17 - Save service account

Creating a GCP Cloud function for collecting and publishing data.

  1. In the Basics section, add observe as the Function name.

Create Cloud Functions

Figure 18 - Creating a Cloud Function

2. Set Runtime service account to the Service Account you created in Step 1 and add the PROJECT_ID and the TOPIC_ID environment variables set to your current project and the topic created in the previous steps.

  • PARENT should be projects/<YOUR_PROJECT_ID>. Your project ID can be found in the Browser URL.

  • TOPIC_ID should be of the form projects/<YOUR_PROJECT_ID>/topics/<TOPIC_NAME>.

3. Click Next.

4. Set the Cloud Function source code. Copy the code from the main.py file in the Observe Google Collection folder to the main.py in the console. See the changelog for a list of versions. The code at the public URL is in the Cloud Storage location observeinc/google-cloud-functions-latest.

Adding python code

Figure 19 - Specify the source code

5. Set the entry point to main and the runtime to Python 3.10.

6. Click Deploy. After a few minutes, the code becomes active.

7. Locate the Trigger tab in Cloud Function, and copy the URL.

8. Navigate to Cloud Scheduler and start creating a Cloud Scheduler job. A reasonable schedule is */15  * * * *.

9. In the Configure the execution section, specify the Cloud Function trigger URL, and set the Auth header to Add OIDC token. Follow the recommendation to create a Service Account with a Cloud Functions Invoker role.

Testing the deployment

Figure 20 - Configuring Cloud Scheduler Execution

Enabling APIs

Within the GCP Console, search for and enable the following APIs:

Baseline

Service-specific

API setup for GCP integrations for Observe

Figure 21 - API Setup for GCP integrations for Observe

After you create the resources and enable the APIs, you can proceed with the GCP App configuration.

GCP App

Note

Although Observe allows you to control the amount of data consumed from GCP, the consumption of assets, logs, and metrics does incur costs. See Google Cloud’s Operations Suite Pricing for more information.

To proceed with the GCP app install, you must enter the Service Account private key associated with a GCP service account created in Prerequisites.

As a reminder - the service account must have the following permissions:

  • Role: Pub/Sub Subscriber (roles/pubsub.subscriber)

  • Role: Monitoring Viewer (roles/monitoring.viewer)

Installation

Navigate to the Apps configuration page in Observe.

GCP app for Observe

Figure 22 - Apps on Observe

  1. Select the GCP App.

  2. Click Install.

  3. Chose one of the two options:

  • Recommended - installs the recommended Observe content, which can be modified later. Click Continue to proceed.

  • Manual Install - allows you to customize the Observe datastream used and app content installed. Click Continue to proceed.

List of available GCP services on Observe

Figure 23 - List of available GCP services on Observe

Creating the required connections to GCP

  1. Click Connections.

Connection for GCP integrations for Observe

Figure 24 - Creating the connection for GCP Integrations

2. Next to Onboard metrics for a GCP project, click Create Connection.

3. For the GCP Project ID, enter your Google Project ID.

4. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform, downloaded from the GCP Console, or created using Deployment Manager.

Create Poller for GCP integrations with Observe

Figure 25 - Creating the poller for GCP Integrations

5. Next to Onboard data using a GCP Pub/Sub Subscription, click Create Connection.

6. For the GCP project ID, enter your Google Project ID.

7. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform or downloaded from the GCP Console.

8. For the GCP Pub/Sub Subscription, enter the Subscription Name you noted in the Terraform, GCP Console setup, or Deployment Manager.

Pub/Sub poller for Pub/Sub subscriptions for Observe

Figure 26 - Creating the poller for Pub/Sub subscriptions

9. Verify that Observe ingests the GCP data.

P

Figure 27 - Verify the connections

Uninstalling the GCP App

Observe Collection

To uninstall GCP collection, remove the observe-gcp-collection module by running the following in the root directory:

$ terraform destroy

The above command removes the associated Service Accounts, Pub/Sub Topics, Log Sinks, and the data collection Cloud Function. Any enabled APIs GCP Console are not affected.

Service Account Private Keys also cease to work in the GCP app connections.

If you performed the deployment using the Deployment Manager, then you can delete it using the following commands:

$ gcloud beta deployment-manager deployments delete observe

The above command removes the associated Service Accounts, Pub/Sub Topics, Log Sinks, and the data collection Cloud Function. Any enabled APIs GCP Console are not affected.

Service Account Private Keys also stop working in the GCP app connections.

To uninstall using the GCP Console, follow these steps:

Delete Service Accounts

  1. Navigate to Service Account.

  2. Under Email, select the Service Account Emails created in the installation steps.

  3. Click Delete to delete the Service Accounts.

These should be associated with the below sample descriptions:

  • For my Observe Poller

  • For data collection Cloud Function

  • For triggering the data collection Cloud Function via Scheduler

Deleting the Pub/Sub Topic

  1. Navigate to Topics

  2. Select the Topic ID created in the installation steps.

  3. Click Delete to delete the Pub/Sub Topic.

Delete the Log Sink

  1. Navigate to Log Router

  2. Select the Log Router Sink created in the installation steps for Observe Pub/Sub Topic.

  3. Click Delete to delete the Log Router Sink.

Delete GCP Cloud Function

  1. Navigate to Cloud Functions

  2. Select the Function created in the installation steps.

  3. Click Delete to delete the Function.

Delete GCP Cloud Scheduler Job

  1. Navigate to Cloud Scheduler

  2. Select the Scheduler Job created in the installation steps.

  3. Click the Delete icon to delete the Scheduler Job.

Service Account Private Keys also stop working in the GCP app connections.

Disabling APIs

Optionally, disable the following APIs GCP Console:

Baseline:

Service-specific:

GCP App

To uninstall the GCP app from your Observe workspace, follow the instructions located at Apps page.

To uninstall the GCP pollers, follow the instructions located at Github Poller Upgrade page.