Before you install the Google Cloud Platform (GCP ) Quickstart app, configure your GCP project using either Terraform or the GCP console.

• Configure your GCP project using Terraform
• Configure your GCP project using the GCP console

Configure your GCP project using Terraform

Terraform automates the installation of the required service accounts with assigned IAM roles, Log Sinks, and Pub/Sub topics, as well as the subscription needed by the GCP application. When you finish, you need the service account key, which can be exported by the Terraform module and used by both the Monitoring and Pub/Sub pollers.

Here are the steps for using Terraform:

1. Install Terraform if needed.

📘

Note

For power users, steps 2-5 below can alternatively be performed by running each of the following Terraform example modules.

• Enable APIs
• Create Service Account

Ensure you have logged into gcloud and have set your quota and working project (Steps 7-9) before running terraform!

Terraform Reference

2. Within the GCP console search for and enable the following APIs:

   • Cloud Asset API
   • Identity and Access Management (IAM) API
   • Cloud Logging API
   • Stackdrive Monitoring API
   • Cloud Pub/Sub API
   • Cloud Resource Manager API
   • Cloud Functions API
   • Cloud Build API
   • Cloud Scheduler API
   • Cloud Storage API
   • Cloud SQL API
   • Cloud Compute API
   • Service Usage API
   • Kubernetes Engine API
   • Google Cloud Memorystore for Redis API
   • Cloud Run API
   • Cloud Tasks API

   Bash script example:

    #!/bin/bash

    # Set your Google Cloud project ID
    PROJECT_ID="YOUR_PROJECT"

    # List of APIs to enable
    APIS=(
        "cloudbuild.googleapis.com"                    # Cloud Build API
        "iam.googleapis.com"                           # IAM API (Identity and Access Management)
        "logging.googleapis.com"                       # Cloud Logging API
        "monitoring.googleapis.com"                    # Stackdriver Monitoring API
        "pubsub.googleapis.com"                        # Cloud Pub/Sub API
        "cloudresourcemanager.googleapis.com"          # Cloud Resource Manager API
        "cloudscheduler.googleapis.com"                # Cloud Scheduler API
        "storage.googleapis.com"                       # Cloud Storage API
        "sqladmin.googleapis.com"                      # Cloud SQL API
        "compute.googleapis.com"                       # Cloud Compute API
        "serviceusage.googleapis.com"                  # Service Usage API
        "container.googleapis.com"                     # Kubernetes Engine API
        "redis.googleapis.com"                         # Google Cloud Memorystore for Redis API
        "run.googleapis.com"                           # Cloud Run API
        "cloudtasks.googleapis.com"                    # Cloud Tasks API
        "cloudasset.googleapis.com"                    # Cloud Asset API
    )

    # Enable each API
    for api in "${APIS[@]}"
    do
        echo "Enabling $api ..."
        gcloud services enable "$api" --project="$PROJECT_ID"
    done

    echo "All APIs have been enabled."

    echo "Listing enabled APIs for project: $PROJECT_ID"
    gcloud services list --enabled --project="$PROJECT_ID"

2. Create a service account with the following roles:

📘

Note

The service account used here is required to run the terraform and create the collection which requires broad permissions. The service account running the collection does not have these permissions.

4. Add yourself under "Service account users role" for the service account.

5. Add Service Account Token Creator to your principal
6. Create a Terraform module. The following Terraform snippets install the GCP collection stack for the project of the Google provider.

Observe currently supports two methods for deploying the Observe GCP collection. You can choose to install the collection for a single project, or for multi-project environments, you deploy the collection to a folder and Observe collects for each project inside of that folder.

provider "google" {
  project = "YOUR_PROJECT_ID"
  region  = "YOUR_DEFAULT_REGION"
}

module "observe_gcp_collection" {
  source   = "observeinc/collection/google"
  name     = "dev"
  resource = "folders/YOUR_FOlDER_ID"
  project_id  = "YOUR_PROJECT_ID" # this is scoping project set in step 8.
}

output "subscription" {
  description = "The Pub/Sub subscription created by this module."
  value       = module.observe_gcp_collection.subscription
}

output "service_account_private_key" {
  description = "A service account key sent to the pollers for Pub/Sub and Cloud Monitoring"
  value       = base64decode(module.observe_gcp_collection.service_account_key.private_key)
  sensitive   = true
}

📘

Note

If you choose folder collection, the project you set becomes your metrics scoping project. All other projects in your folder report their metrics through this project with no added GCP charges. If you add projects to the folder, you must run terraform again to add the new projects into the metrics monitoring scope.

7. Set up the Google Cloud SDK and run gcloud auth application-default login to create a credentials file for Terraform to use:

gcloud auth application-default login

8. Set your quota project. This is the project that will be used for billing and project quotas.

gcloud auth application-default set-quota-project <MYQUOTAPROJECT>

9. Set your working project. If you are installing collection for a folder, this is the project you want to be used as the scoping project.

gcloud config set project <MYPROJECT>

10. Run export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=some-sa@scoping-project-392418.iam.gserviceaccount.com. This runs the terraform as the service account you created in Step 4. A service account is required to create the asset feed, and it cannot be done by a user.

export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=some-sa@scoping-project-392418.iam.gserviceaccount.com

📘

Note

If the service account was created via Create Service Account, service account email can be accessed via terraform output as terraform-observe-collect-sa@<your-project-id>.iam.gserviceaccount.com

11. Run terraform apply.

terraform apply

12. Record information you need to install the GCP Quickstart app.

To access the service account key after running Terraform, use the following command:

terraform output -raw service_account_private_key

Also, make a note of the created subscription name. You need this for the GCP Quickstart app installation.

terraform output subscription

In this case, capture the name property of the output.

"name" = "dev-obs"

Record the dev-obs portion for the GCP Quickstart app installation.

Once you create these resources, you can proceed with your GCP Quickstart app installation.

Most commonly, you use Observe apps to install and configure the GCP Quickstart app. However, Observe can also provide the Terraform modules and providers necessary for this task. Please contact your Observe account manager for assistance.

Configure your GCP project using the GCP console

Within the GCP Console (or gcloud CLI), you need to perform the following tasks:

• Create service accounts with properly assigned IAM roles.
• Create a Pub/Sub topic and subscription needed to poll a project's data.
• Create a log sink.
• Create an asset feed to monitor resource changes.
• Add cloud functions for additional data collection.
• Enable the proper APIs.

Create service accounts

Perform the following tasks to create the service accounts:

1. Create a service account with the following details:

• Service account name: my-observe-poller-service-account
• Service account ID: my-observe-poller-service-id
• Service account description: My Observe Pollers

2. Grant the service account access to the project. Assign the Monitoring Viewer role:

3. Generate and download a service account key for the service account you just created:
   1. Click on the KEYS tab.
   2. Click ADD KEY.
   3. Select JSON the key type.
   4. Click Create.

4. Save the key. You will need this later to install the GCP Quickstart app.

Create a Pub/Sub topic

Perform the following tasks:

1. Under Topics, click Create a topic.
2. Configure the following parameters:

• Enter the Topic ID observe-topic.
• Select Add a default subscription.
• Under Encryption select Google-managed encryption key.

4. Click Create topic.

On the Subscriptions tab, you see that GCP automatically created a Pub/Sub subscription. You need this information when you install the GCP Quickstart app.

Create a log sink

Configure a Log Sink to publish to the Pub/Sub topic.

1. Under Logs Router, click Create sink.
2. Under Sink details, add the following information:

• Enter my-observe-log-sink as the Sink name.
• Enter For my Observe pubsub topic for the Sink description.

3. Under the Sink destination, configure the following parameters:

• Select Sink Service Cloud Pub/Sub topic.
• Choose the topic under your project that ends with your Sink name. Based on the previous step, use observe-topic.

4. Click Create sink.
5. Create an inclusion filter to determine which logs you want to include in the Sink.

6. Click Create sink.

Create an asset feed to monitor resource changes

Run the following gcloud command for your project. Be sure to use the pub/sub previously created here.

gcloud asset feeds create my-asset-updates \
    --project=my-project \
    --asset-types='aiplatform.googleapis.com.*,anthos.googleapis.com.*,apigateway.googleapis.com.*,apikeys.googleapis.com.*,appengine.googleapis.com.*,apps.k8s.io.*,artifactregistry.googleapis.com.*,assuredworkloads.googleapis.com.*,batch.k8s.io.*,beyondcorp.googleapis.com.*,bigquery.googleapis.com.*,bigquerymigration.googleapis.com.*,bigtableadmin.googleapis.com.*,cloudbilling.googleapis.com.*,clouddeploy.googleapis.com.*,cloudfunctions.googleapis.com.*,cloudkms.googleapis.com.*,cloudresourcemanager.googleapis.com.*,composer.googleapis.com.*,compute.googleapis.com.*,connectors.googleapis.com.*,container.googleapis.com.*,containerregistry.googleapis.com.*,dataflow.googleapis.com.*,dataform.googleapis.com.*,datafusion.googleapis.com.*,datamigration.googleapis.com.*,dataplex.googleapis.com.*,dataproc.googleapis.com.*,datastream.googleapis.com.*,dialogflow.googleapis.com.*,dlp.googleapis.com.*,dns.googleapis.com.*,documentai.googleapis.com.*,domains.googleapis.com.*,eventarc.googleapis.com.*,extensions.k8s.io.*,file.googleapis.com.*,firestore.googleapis.com.*,gameservices.googleapis.com.*,gkebackup.googleapis.com.*,gkehub.googleapis.com.*,healthcare.googleapis.com.*,iam.googleapis.com.*,ids.googleapis.com.*,k8s.io.*,logging.googleapis.com.*,managedidentities.googleapis.com.*,memcache.googleapis.com.*,metastore.googleapis.com.*,monitoring.googleapis.com.*,networkconnectivity.googleapis.com.*,networking.k8s.io.*,networkmanagement.googleapis.com.*,networkservices.googleapis.com.*,orgpolicy.googleapis.com.*,osconfig.googleapis.com.*,privateca.googleapis.com.*,pubsub.googleapis.com.*,rbac.authorization.k8s.io.*,redis.googleapis.com.*,run.googleapis.com.*,secretmanager.googleapis.com.*,servicedirectory.googleapis.com.*,servicemanagement.googleapis.com.*,serviceusage.googleapis.com.*,spanner.googleapis.com.*,speech.googleapis.com.*,sqladmin.googleapis.com.*,storage.googleapis.com.*,tpu.googleapis.com.*,transcoder.googleapis.com.*,vpcaccess.googleapis.com.*,workflows.googleapis.com.*' \
    --content-type=resource \
    --pubsub-topic=projects/my-project/topics/observe-topic

Add GCP cloud functions

Perform the following tasks using the gcloud cli locally or using a GCP Console Cloud Shell:

1. Set Your Project and Region IDs as Environment Variables

export PROJECT_ID=<YOUR_PROJECT_ID>
export REGION_ID=<YOUR_REGION>

2. Create a Service Account for Cloud Functions:

gcloud iam service-accounts create observe-cloudfunctions --description="Used by the Observe Cloud Functions" --display-name="observe-cloudfunctions"

3. Assign Roles to the Cloud Function Service Account:

gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/compute.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/iam.serviceAccountViewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudscheduler.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudasset.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/browser
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/logging.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/storage.objectCreator
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/storage.objectViewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/storage.objectAdmin
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/storage.admin
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudfunctions.invoker
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudtasks.enqueuer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudtasks.viewer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudtasks.taskDeleter
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/iam.serviceAccountUser

4. Create a Service Account for Cloud Scheduler:

gcloud iam service-accounts create observe-scheduler --description="Allows the Cloud Scheduler job to trigger a Cloud Function" --display-name="observe-scheduler"

5. Set Up a Storage Bucket:

gsutil mb -l US gs://${PROJECT_ID}-observe
gsutil iam ch serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com:objectCreator gs://${PROJECT_ID}-observe
gsutil iam ch serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com:objectViewer gs://${PROJECT_ID}-observe

6. Set Up a Task Queue:

gcloud tasks queues create --location=${REGION_ID} observe-assets-queue

7. Deploy the observe_assets_to_gcs Cloud Function:

# Deploy observe_assets_to_gcs Cloud Function
gcloud functions deploy observe_assets_to_gcs \
--runtime=python310 \
--trigger-http \
--memory=512MB \
--allow-unauthenticated \
--service-account=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com \
--source=gs://observeinc/google-cloud-functions-v0.3.0.zip \
--entry-point=export_assets \
--set-env-vars=GCP_REGION=${REGION_ID},GCS_TO_PUBSUB_CLOUD_FUNCTION_URI=https://${REGION_ID}-${PROJECT_ID}.cloudfunctions.net/observe_gcs_to_pubsub,LOG_LEVEL=WARNING,OUTPUT_BUCKET=gs://${PROJECT_ID}-observe,PARENT=projects/${PROJECT_ID},PROJECT=${PROJECT_ID},SERVICE_ACCOUNT_EMAIL=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com,TASK_QUEUE=observe-assets-queue,TOPIC_ID=projects/${PROJECT_ID}/topics/observe-topic,VERSION=observeinc/google-cloud-functions-v0.3.0.zip \
--max-instances=100 \
--timeout=300s

8. Assign Roles to the Cloud Scheduler Service Account:

gcloud functions add-iam-policy-binding observe_assets_to_gcs --project=${PROJECT_ID} --member=serviceAccount:observe-scheduler@${PROJECT_ID}.iam.gserviceaccount.com --role=roles/cloudfunctions.invoker

9. Set Up the Cloud Scheduler:

gcloud scheduler jobs create http observe-assets-job \
--schedule="0 * * * *" \
--http-method=POST \
--uri=https://${REGION_ID}-${PROJECT_ID}.cloudfunctions.net/observe_assets_to_gcs \
--oidc-service-account-email=observe-scheduler@${PROJECT_ID}.iam.gserviceaccount.com \
--message-body='{}' \
--headers=Content-Type=application/json \
--location=${REGION_ID}

10. Deploy the observe_gcs_to_pubsub Cloud Function:

gcloud functions deploy observe_gcs_to_pubsub \
--runtime=python310 \
--trigger-http \
--memory=512MB \
--allow-unauthenticated \
--service-account=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com \
--source=gs://observeinc/google-cloud-functions-v0.3.0.zip \
--entry-point=gcs_to_pubsub \
--set-env-vars=GCP_REGION=${REGION_ID},GCS_TO_PUBSUB_CLOUD_FUNCTION_URI=not_applicable,LOG_LEVEL=WARNING,OUTPUT_BUCKET=gs://${PROJECT_ID}-observe,PARENT=projects/${PROJECT_ID},PROJECT=${PROJECT_ID},SERVICE_ACCOUNT_EMAIL=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com,TASK_QUEUE=observe-assets-queue,TOPIC_ID=projects/${PROJECT_ID}/topics/observe-topic,VERSION=observeinc/google-cloud-functions-v0.3.0.zip \
--max-instances=100 \
--timeout=300s

11. Set Service Account Permissions for Pub/Sub:

gcloud pubsub topics add-iam-policy-binding observe \
  --member=serviceAccount:observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com \
  --role=roles/pubsub.publisher \
  --project=${PROJECT_ID}

12. Deploy the observe_rest_of_assets Cloud Function:

gcloud functions deploy observe_rest_of_assets \
--runtime=python310 \
--trigger-http \
--memory=512MB \
--allow-unauthenticated \
--service-account=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com \
--source=gs://observeinc/google-cloud-functions-v0.3.0.zip \
--entry-point=rest_of_assets \
--set-env-vars=GCP_REGION=${REGION_ID},GCS_TO_PUBSUB_CLOUD_FUNCTION_URI=not_applicable,LOG_LEVEL=WARNING,OUTPUT_BUCKET=gs://${PROJECT_ID}-observe,PARENT=projects/${PROJECT_ID},PROJECT=${PROJECT_ID},SERVICE_ACCOUNT_EMAIL=observe-cloudfunctions@${PROJECT_ID}.iam.gserviceaccount.com,TASK_QUEUE=observe-assets-queue,TOPIC_ID=projects/${PROJECT_ID}/topics/observe-topic,VERSION=observeinc/google-cloud-functions-v0.3.0.zip \
--max-instances=100 \
--timeout=300s

13. Schedule the observe_rest_of_assets Cloud Function:

gcloud scheduler jobs create http observe-rest-assets-job \
--schedule="15 * * * *" \
--http-method=POST \
--uri=https://${REGION_ID}-${PROJECT_ID}.cloudfunctions.net/observe_rest_of_assets \
--oidc-service-account-email=observe-scheduler@${PROJECT_ID}.iam.gserviceaccount.com \
--message-body='{}' \
--headers=Content-Type=application/json \
--location=${REGION_ID}

Enable APIs

Within the GCP console, search for and enable the following APIs:

Baseline

• Cloud Asset API
• Identity and Access Management (IAM) API
• Cloud Logging API
• Stackdrive Monitoring API
• Cloud Pub/Sub API
• Cloud Resource Manager API
• Cloud Build API
• Cloud Scheduler API

Service-specific

• Cloud SQL API
• Computer Engine API
• Cloud Functions API
• Cloud Storage API

After you create the resources and enable the APIs, you can proceed with the GCP Quickstart app installation.