Google Cloud Platform (GCP) App¶
The Observe Google Cloud Platform (GCP) app streamlines the process of collecting data from GCP. A Pub/Sub topic makes logging and asset inventory information available and monitoring APIs expose metrics from several common GCP services at once. Observe pollers then ingest the data into your Observe environment.
The GCP app works with the datasets in your workspace. After setting up your GCP project for data collection, you can install and configure the GCP app in your Observe environment. Observe provides a Terraform module for setting up your GCP project. Contact us for assistance with completing these steps.
If you currently ingest GCP data, consult with Observe to see if the GCP app could enhance your existing data collection strategy.
What GCP Data Does Observe Ingest?¶
Standard Ingestion Sources¶
The GCP app automatically ingests the following types of data from a single project:
Asset Inventory - Metadata of GCP resources
Logs - of services you use
Metrics - of services you use (if enabled)
Additional Ingestion Sources¶
With these previous sources configured and working, add additional services by enabling them in the GCP app interface. Details for common services may be found in the Observe documentation:
BigQuery - analytics data warehouse
Cloud Functions - serverless compute functions
Cloud Load Balancing - traffic load balancing
Cloud Run - serverless container platform
Cloud Storage - object storage
Cloud SQL - databases
Compute - virtual machines
Using the GCP App Data¶
Data coming into Observe through the GCP app populates datasets such as the following:
Asset Inventory
Asset Inventory Records - Raw data from asset exports
Resource Asset Inventory Records - All cloud assets in GCP that provide the basis for resource datasets
Metrics
Metrics - Periodically, typically every 60 seconds, sample metrics from GCP
Distribution Metrics - Services use this value type when the individual measurements become too numerous to collect, but statistical information, such as averages or percentiles, about those measurements, can be valuable.
Resource datasets
Cloud Functions
Cloud SQL Instance
Compute Instance
Projects
Storage Buckets
Setup Overview¶
Observe provides a Terraform module that creates service accounts, log sink, and a pub/sub topic as well as the subscription needed by Observe pollers for your GCP Project. (Observe Google Collection GitHub repository). You can also follow the instructions for the Deployment Manager tab or GCP console tab in the Installation Section to provision those resources with the Google Cloud Deployment Manager.
Observe pollers, using your created service account key, extract assets, logs, metrics, and project data and send it into your Observe account at a set interval.

Figure 1 - GCP Module Flow
The GCP app shapes and accelerates the data for monitoring and troubleshooting.
Installation¶
Prerequisites¶
Before proceeding with the GCP app install, ensure you configure your GCP project using either Terraform, Deployment Manager, or the GCP console.
Terraform automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the service account key, which can be exported by the Terraform module and used by both the Monitoring and Pub/Sub pollers.
Here are the steps for using Terraform:
Install Terraform if needed.
Set up the Google Cloud SDK and run gcloud auth login to create a credentials file for Terraform to use.
Within the GCP Console search for and enable the following APIs:
Figure 2 - API Setup for GCP integrations into Observe
4. Create a Terraform module. The following Terraform snippet installs the GCP collection stack for the project of the Google provider.
provider "google" {
project = "YOUR_PROJECT_ID"
region = "YOUR_DEFAULT_REGION"
}
module "observe_gcp_collection" {
source = "observeinc/collection/google"
name = "dev"
resource = "projects/YOUR_PROJECT_ID"
}
output "subscription" {
description = "The Pub/Sub subscription created by this module."
value = module.observe_gcp_collection.subscription
}
output "service_account_private_key" {
description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
value = base64decode(module.observe_gcp_collection.service_account_key.private_key)
sensitive = true
}
5. Run terraform apply
.
6. Record information needed for installation of the GCP App.
To access the service account key after running Terraform, use theterraform output -raw service_account_private_key

Figure 3 - JSON key for GCP integrations into Observe
Also, make a note of the created Subscription. You need this for the GCP App installation.
To access the service account key after running Terraform, use theterraform output -raw service_account_private_key
Once you create these resources, you can proceed with your GCP App configuration.
Most commonly, you use the Observe Apps install page to install and configure the GCP App. However, Observe can also provide the Terraform modules and providers necessary for this task. Please contact your Observe account manager for assistance.
Deployment Manager automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the Service Account Key which you output from the Deployment Manager configuration and used by both the Monitoring and Pub/Sub pollers.
Use the following steps for Deployment Manager:
Within the GCP Console search for and enable the following APIs:

Figure 4 - API Setup for GCP integrations into Observe
2. Log into the GCP console, and provide the [PROJECT_NUMBER]@cloudservices.gserviceaccount.com service account for the Logging Configuration Writer and Security Admin roles. Deployment Manager uses this service account.

Figure 5 - Service Account setup for GCP integrations for Observe
Also ensure that [PROJECT_NUMBER]@cloudbuild.gserviceaccount.com service account exists with the Cloud Build Service Account role.
3. Using a terminal with gcloud
configured, such as GCP Cloud Shell, use the Deployment Manager template:
gcloud beta deployment-manager deployments create observe \
--template https://storage.googleapis.com/observeinc/deploymentmanager-google-collection-latest.py \
--properties "resource:'projects/$YOUR_PROJECT_ID'"
Template versions can be found by running the following command:
gcloud storage ls gs://observeinc/
You can find the change log at CHANGELOG.
4. Using the terminal, run the following command to see the output.
gcloud beta deployment-manager manifests describe --deployment observe
5. You need the subscription ID and private key in the finalLayout
field for GCP App installation.
...
layout: |
resources:
- name: deploymentmanager-google-collection-latest-py
outputs:
- finalValue: observe-123456
name: project_id
value: observe-123456
- finalValue: observe
name: subscription_id
value: observe
- finalValue: ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAidzI2MS1maW5hbC1uYXRoYW4tMjM2MTA3IiwKICAicHJpdmF0ZV9rZXlfaWQiOiAiMzFlMzNmOWU1OWJkZjIyYTlmNzU2MjE5Y2M3MWUxNWVmZGNmMzU3YyIsCiAgInByaXZhdGVfa2V5IjogIi0tLS0tQkVHSU4gUFJJVkFURSBLRVktLS0tLVxuTUlJRXZnSUJBREFOQmdrcWhraUc5dzBCQVFFRkFBU0NCS2d3Z2dTa0FnRUFBb0lCQVFEa2tRSWlRcHl3azVjTFxuV3BGQ3dka3QzMUE4azB1T0piR0NSYi9xVnV0YUFnaU9iWU5nQXpOM3BVdHZhMlIzUzVDdkk3RStqNFhyRS9BWVxuTlBVL0xYWjR3ckk1bjY4b0ZRenE3MmpsNWl5QUJOblR2K0xQcUlGMW9Wc2xDblRtaHEyR2xpaFlrdzJtWXYveVxuWVlLSW5pNm5KSTlqTXdxekZQcUh6clBCUjZxVi94Y1BsSWdjTlZBbEZYZ0d4bjlXb1J5MUhRSWQ4NWNDaGdGdlxuTXo2alg0Wm1WN1MzQzhLekRkRXYySlJLSVZLaG9oS0NKQVo2YWtRWTJLMHdNUDI2WU9YeFlYcWtaZFV2UEhNTVxuM0NhYzNmQXduQ2hjTS9VMzVVZ3RsdmVVdndYM0JNRlBTTHpTajBmNHdVc0RxZXp5K0c4cHphS0ExQm40bEo2Z1xubnRvdkNnRlZBZ01CQUFFQ2dnRUFPT1NqYkZXNlhQV1AzbUtmejhjaXdSbGFCb24vZjVXQVZ0WDN6R1B3VjBvMFxuU1FlWGNQbTdBT2dqRzh2V2lSOUZGMk5DQmptQmN4OVdYeUFuZjU5WVl2ekxraTVlVWhMVFVWTmdnQUZJU0dGc1xudXhtRFluOG1JbHFVcm1hbzdJelBlclc3dXpoM2FGZ3lMMm40TFB0NnoxMzVWdzdvZGVVZDYxb1hRbUFOWlNKT1xuWjBNYWx2N1UzUlhOa0ZqNHVYWFZVRS84OTNpdFRGY0ZKTGJEeFhHejF4WGNUckJqK2MwMllSSmRLZm9xZkRLZ1xuZXQ3c25yeXpaVEhUcVdzaDNsd0ZrTWhVcUZubEptNmNEM0tCc3l4UGpkSHF2UjJVNjNuWVd2Q3pnUHRwbjdUaVxuc3l2QXZCQWFvb0kwR0cweGJhSk5XQzhmTitBVVhjY3VMNFRSQW9MYmpRS0JnUUQ1ZEJKYldMcy8ySXF4WGdnUVREDACTED
name: poller_private_key_base64
value: $(ref.google_service_account_key-poller.privateKeyData)
...
6. Use the following command to decode the private key for GCP App installation.
echo -n <YOUR_PRIVATE_KEY> | base64 --decode
echo -n
ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2plY3RfaWQiOiAidzI2MS1maW5hbC1uYXRoYW4tMjM2MTA3IiwKICAicHJpdmF0ZV9rZXlfaWQiOiAiMzFlMzNmOWU1OWJkZjIyYTlmNzU2MjE5Y2M3MWUxNWVmZGNmMzU3YyIsCiAgInByaXZhdGVfa2V5IjogIi0tLS0tQkVHSU4gUFJJVkFURSBLRVktLS0tLVxuTUlJRXZnSUJBREFOQmdrcWhraUc5dzBCQVFFRkFBU0NCS2d3Z2dTa0FnRUFBb0lCQVFEa2tRSWlRcHl3azVjTFxuV3BGQ3dka3QzMUE4azB1T0piR0NSYi9xVnV0YUFnaU9iWU5nQXpOM3BVdHZhMlIzUzVDdkk3RStqNFhyRS9BWVxuTlBVL0xYWjR3ckk1bjY4b0ZRenE3MmpsNWl5QUJOblR2K0xQcUlGMW9Wc2xDblRtaHEyR2xpaFlrdzJtWXYveVxuWVlLSW5pNm5KSTlqTXdxekZQcUh6clBCUjZxVi94Y1BsSWdjTlZBbEZYZ0d4bjlXb1J5MUhRSWQ4NWNDaGdGdlxuTXo2alg0Wm1WN1MzQzhLekRkRXYySlJLSVZLaG9oS0NKQVo2YWtRWTJLMHdNUDI2WU9YeFlYcWtaZFV2UEhNTVxuM0NhYzNmQXduQ2hjTS9VMzVVZ3RsdmVVdndYM0JNRlBTTHpTajBmNHdVc0RxZXp5K0c4cHphS0ExQm40bEo2Z1xubnRvdkNnRlZBZ01CQUFFQ2dnRUFPT1NqYkZXNlhQV1AzbUtmejhjaXdSbGFCb24vZjVXQVZ0WDN6R1B3VjBvMFxuU1FlWGNQbTdBT2dqRzh2V2lSOUZGMk5DQmptQmN4OVdYeUFuZjU5WVl2ekxraTVlVWhMVFVWTmdnQUZJU0dGc1xudXhtRFluOG1JbHFVcm1hbzdJelBlclc3dXpoM2FGZ3lMMm40TFB0NnoxMzVWdzdvZGVVZDYxb1hRbUFOWlNKT1xuWjBNYWx2N1UzUlhOa0ZqNHVYWFZVRS84OTNpdFRGY0ZKTGJEeFhHejF4WGNUckJqK2MwMllSSmRLZm9xZkRLZ1xuZXQ3c25yeXpaVEhUcVdzaDNsd0ZrTWhVcUZubEptNmNEM0tCc3l4UGpkSHF2UjJVNjNuWVd2Q3pnUHRwbjdUaVxuc3l2QXZCQWFvb0kwR0cweGJhSk5XQzhmTitBVVhjY3VMNFRSQW9MYmpRS0JnUUQ1ZEJKYldMcy8ySXF4WGdnUVREDACTED | base64
--decode
Within the GCP Console, you need to perform the following tasks:
Create service accounts with proper assigned IAM roles)
Create a Pub/Sub topic and subscription needed to poll a project’s data
Create a Service account with the following details:
Service account name: my-observe-poller-service-account
Service account ID: my-observe-poller-service-id
Service account description: My Observe Pollers

Figure 6 - Service Account setup for GCP integrations for Observe
2. Grant this service account access to the project:
Role: Pub/Sub Subscriber (roles/pubsub.subscriber)
Role: Monitoring Viewer (roles/monitoring.viewer)
Role: Cloud Asset Viewer (roles/cloudasset.viewer)
Role: Browser (roles/browser)

Figure 7 - Service Account Roles for GCP integrations with Observe
3. Generate and download a Service account key for the Service account you just created:
- Click on the KEYS tab.
- Click ADD KEY.
- Select the Key type as JSON.
- Click Create.
4. Save the key for Observe GCP App Installation.

Figure 8 - Service Account Key setup for GCP integrations for Observe

Figure 9 - Service Account Key setup for GCP integrations for Observe
Under Topics, click Create a topic.
Configure the following parameters:
- Enter the Topic ID my-log-sink-topic.
- Select Add a default subscription.
- Under Encryption, select Google-managed encryption key.
3. Click CREATE TOPIC.

Figure 10 - Service Account Key setup for GCP integrations for Observe
On the Subscriptions tab, you see that GCP automatically created a Pub/Sub subscription. You need this information for GCP App setup.

Figure 11 - Pub/Sub topic setup for GCP integrations for Observe
Configure a Log Sink to publish to the Pub/Sub topic.
Under Logs Router, click Create sink.
2. Under Sink details, add the following information:
- Enter my-observe-log-sink as the Sink name.
- Enter For my Observe pubsub topic for the Sink description.

Figure 12 - Log Sink setup for GCP integrations with Observe
3. Under the Sink destination, configure the following parameters:
- Select Sink Service Cloud Pub/Sub topic.
- Choose the topic under your project that ends with your Sink name. Based on the previous step, use my-log-sink-topic.
4. Click CREATE SINK.

Figure 13 - Log Sink topic setup for GCP integrations for Observe
5. Create an inclusion filter to determine which logs you want to include in the Sink.

Figure 14 - Log Sink filters for GCP integrations for Observe
6. Click Create Sink.
Observe uses Python code-based functions in this instance, but you can develop further extensions using any language supported by GCP Cloud functions.
Create a Service Account by entering the following parameters:
Pub/Sub Publisher
Compute Viewer
View Service Accounts
Cloud Scheduler Viewer
Browser
Cloud Asset Viewer
2. Click CREATE AND CONTINUE.

Figure 15 - Create service account
3. Add the following roles for GCP Cloud function execution:
Storage Object Viewer
Pub/Sub Publisher
Compute Viewer
View Service Accounts
Cloud Scheduler Viewer

Figure 16 - Add roles
4. Click DONE to save the Service Account.

Figure 17 - Save service account
In the Basics section, add observe as the Function name.

Figure 18 - Creating a Cloud Function
2. Set Runtime service account to the Service Account you created in Step 1 and add the PROJECT_ID and the TOPIC_ID environment variables set to your current project and the topic created in the previous steps.
PARENT
should beprojects/<YOUR_PROJECT_ID>
. Your project ID can be found in the Browser URL.TOPIC_ID
should be of the formprojects/<YOUR_PROJECT_ID>/topics/<TOPIC_NAME>
.
3. Click Next.
4. Set the Cloud Function source code. Copy the code from the main.py
file in the Observe Google Collection folder to the main.py
in the console. See the changelog for a list of versions.
The code at the public URL is in the Cloud Storage location observeinc/google-cloud-functions-latest
.

Figure 19 - Specify the source code
5. Set the entry point to main
and the runtime to Python 3.10
.
6. Click Deploy. After a few minutes, the code becomes active.
7. Locate the Trigger tab in Cloud Function, and copy the URL.
8. Navigate to Cloud Scheduler and start creating a Cloud Scheduler job. A reasonable schedule is */15 * * * *
.
9. In the Configure the execution section, specify the Cloud Function trigger URL, and set the Auth header to Add OIDC token. Follow the recommendation to create a Service Account with a Cloud Functions Invoker role.

Figure 20 - Configuring Cloud Scheduler Execution
Within the GCP Console, search for and enable the following APIs:
Baseline
Service-specific

Figure 21 - API Setup for GCP integrations for Observe
After you create the resources and enable the APIs, you can proceed with the GCP App configuration.
GCP App¶
Note
Although Observe allows you to control the amount of data consumed from GCP, the consumption of assets, logs, and metrics does incur costs. See Google Cloud’s Operations Suite Pricing for more information.
To proceed with the GCP app install, you must enter the Service Account private key associated with a GCP service account created in Prerequisites.
As a reminder - the service account must have the following permissions:
Role: Pub/Sub Subscriber (roles/pubsub.subscriber)
Role: Monitoring Viewer (roles/monitoring.viewer)
Installation¶
Navigate to the Apps configuration page in Observe.

Figure 22 - Apps on Observe
Select the GCP App.
Click Install.
Chose one of the two options:
Recommended - installs the recommended Observe content, which can be modified later. Click Continue to proceed.
Manual Install - allows you to customize the Observe datastream used and app content installed. Click Continue to proceed.

Figure 23 - List of available GCP services on Observe
Click Connections.

Figure 24 - Creating the connection for GCP Integrations
2. Next to Onboard metrics for a GCP project, click Create Connection.
3. For the GCP Project ID, enter your Google Project ID.
4. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform, downloaded from the GCP Console, or created using Deployment Manager.

Figure 25 - Creating the poller for GCP Integrations
5. Next to Onboard data using a GCP Pub/Sub Subscription, click Create Connection.
6. For the GCP project ID, enter your Google Project ID.
7. For the Service Account Private Key JSON, enter the entire JSON string you either generated with Terraform or downloaded from the GCP Console.
8. For the GCP Pub/Sub Subscription, enter the Subscription Name you noted in the Terraform, GCP Console setup, or Deployment Manager.

Figure 26 - Creating the poller for Pub/Sub subscriptions
9. Verify that Observe ingests the GCP data.

Figure 27 - Verify the connections
Uninstalling the GCP App¶
Observe Collection¶
To uninstall GCP collection, remove the observe-gcp-collection
module by running the following in the root directory:
$ terraform destroy
The above command removes the associated Service Accounts, Pub/Sub Topics, Log Sinks, and the data collection Cloud Function. Any enabled APIs GCP Console are not affected.
Service Account Private Keys also cease to work in the GCP app connections.
If you performed the deployment using the Deployment Manager, then you can delete it using the following commands:
$ gcloud beta deployment-manager deployments delete observe
The above command removes the associated Service Accounts, Pub/Sub Topics, Log Sinks, and the data collection Cloud Function. Any enabled APIs GCP Console are not affected.
Service Account Private Keys also stop working in the GCP app connections.
To uninstall using the GCP Console, follow these steps:
Navigate to Service Account.
Under Email, select the Service Account Emails created in the installation steps.
Click Delete to delete the Service Accounts.
These should be associated with the below sample descriptions:
For my Observe Poller
For data collection Cloud Function
For triggering the data collection Cloud Function via Scheduler
Navigate to Topics
Select the Topic ID created in the installation steps.
Click Delete to delete the Pub/Sub Topic.
Navigate to Log Router
Select the Log Router Sink created in the installation steps for Observe Pub/Sub Topic.
Click Delete to delete the Log Router Sink.
Navigate to Cloud Functions
Select the Function created in the installation steps.
Click Delete to delete the Function.
Navigate to Cloud Scheduler
Select the Scheduler Job created in the installation steps.
Click the Delete icon to delete the Scheduler Job.
Service Account Private Keys also stop working in the GCP app connections.
Optionally, disable the following APIs GCP Console:
Baseline:
Service-specific:
GCP App¶
To uninstall the GCP app from your Observe workspace, follow the instructions located at Apps page.
To uninstall the GCP pollers, follow the instructions located at Github Poller Upgrade page.