Install Observe Agent using Ansible

Install the Observe Agent

Before you begin, install Ansible on your control node.

Debian-based distributions

Install the observe-agent package. You’ll need to first add the Observe debian repository to your trusted depositories in your sources.list.d file. Create package.yaml under /etc/ansible/roles/agent/tasks/.

- name : Install Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: add observe-agent apt source
      ansible.builtin.copy:
        dest: /etc/apt/sources.list.d/observeinc.list
        mode: 0644
        content: |
          deb [trusted=yes] https://repo.observeinc.com/apt/ /
    - name: install observe-agent package
      ansible.builtin.apt:
        install_recommends: no
        name: observe-agent
        state: latest
        update_cache: yes   

Use ansible-playbook to install the Observe Agent with package.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/package.yaml

PLAY [Observe Agent Getting Started Playbook] ******************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [add observe-agent apt source] ****************************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

TASK [install observe-agent package] ***************************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

PLAY RECAP *****************************************************************************************************************************************************************************************
ansible-managed-node-1 : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ansible-managed-node-2 : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

RedHat-based distributions

Install the observe-agent package. You’ll need to first add the Observe yum repository to your trusted repositories in your yum.repos.d folder. Create package.yaml under /etc/ansible/roles/agent/tasks/.

- name : Install Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: add observe-agent yum source
      ansible.builtin.yum_repository:
        name: observe
        description: Gemfury Private Repo
        baseurl: https://yum.fury.io/observeinc/
        mode: 0644
    - name: install observe-agent package
      yum:
        name: observe-agent
        state: latest
        update_cache: yes
        disable_gpg_check: yes

Use ansible-playbook to install the Observe Agent with package.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/package.yaml

PLAY [Install Observe Agent] ************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [add observe-agent yum source] *****************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [install observe-agent package] ****************************************************************************************************************
changed: [54.202.68.91]
changed: [35.87.95.246]

PLAY RECAP ******************************************************************************************************************************************
35.87.95.246               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
54.202.68.91               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Configure the Observe Agent

Debian-based distributions

Create config.yaml under /etc/ansible/roles/agent/tasks/.

- name : Config Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: install observe-agent.yaml
      ansible.builtin.template:
        src: ../templates/observe-agent.yaml.j2
        dest: /etc/observe-agent/observe-agent.yaml
        owner: root
        group: observe-agent
        mode: 0640

Create observe-agent.yaml.j2 under /etc/ansible/roles/agent/templates. Replace ${OBSERVE_TOKEN} (your instance’s token) and ${OBSERVE_COLLECTION_ENDPOINT} (your instance’s collection endpoint) on each host. For more information on configuring the agent, see Configuration.

Note

The ${OBSERVE_COLLECTION_ENDPOINT} is composed from https://${OBSERVE_CUSTOMER_ID}.collect.${OBSERVE_INSTANCE}. For example, if you typically login to https://123456789012.observeinc.com, your ${OBSERVE_COLLECTION_ENDPOINT} is https://123456789012.collect.observeinc.com.

Note

Some Observe instances may optionally use a name instead of Customer ID; if this is the case for your instance, contact your Observe Data Engineer to discuss implementation. A stem name will work as is, but a DNS redirect name may require client configuration.

# Observe data token (ex: a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6)
token: "${OBSERVE_TOKEN}"

# Target Observe collection url (ex: https://123456789012.collect.observeinc.com/)
observe_url: "${OBSERVE_COLLECTION_ENDPOINT}"

# Debug mode - Sets agent log level to debug
debug: false

self_monitoring:
  enabled: true

host_monitoring:
  enabled: true
  logs: 
    enabled: true
    include:
      - /var/log/**/*.log
      - /var/log/syslog
  metrics:
    host: 
      enabled: true
    process:
      enabled: false

# otel_config_overrides:
#   exporters:
#     # This is a net new exporter
#     debug:
#       verbosity: detailed
#       sampling_initial: 5
#       sampling_thereafter: 200
#   service:
#     pipelines:
#       # This will override the existing metrics/host_monitoring pipeline and output to stdout debug instead
#       metrics/host_monitoring_host:
#         receivers: [hostmetrics/host-monitoring-host]
#         processors: [memory_limiter]
#         exporters: [debug]
#       # This is a net new pipeline
#       logs/new-pipeline:
#         receivers: [filelog/host_monitoring]
#         processors: [memory_limiter]
#         exporters: [debug]

Use ansible-playbook to configure the Observe Agent with config.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/config.yaml

PLAY [Config Observe Agent] ************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [install observe-agent.yaml] ******************************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

PLAY RECAP *****************************************************************************************************************************************************************************************
ansible-managed-node-1 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ansible-managed-node-2 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

RedHat-based distributions

Create config.yaml under /etc/ansible/roles/agent/tasks/.

- name : Config Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: install observe-agent.yaml
      ansible.builtin.template:
        src: ../templates/observe-agent.yaml.j2
        dest: /etc/observe-agent/observe-agent.yaml
        owner: root
        group: observe-agent
        mode: 0640

Create observe-agent.yaml.j2 under /etc/ansible/roles/agent/templates. Replace ${OBSERVE_TOKEN} (your instance’s token) and ${OBSERVE_COLLECTION_ENDPOINT} (your instance’s collection endpoint) on each host. For more information on configuring the agent, see Configuration.

# Observe data token (ex: a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6)
token: "${OBSERVE_TOKEN}"

# Target Observe collection url (ex: https://123456789012.collect.observeinc.com/)
observe_url: "${OBSERVE_COLLECTION_ENDPOINT}"

# Debug mode - Sets agent log level to debug
debug: false

self_monitoring:
  enabled: true

host_monitoring:
  enabled: true
  logs: 
    enabled: true
    include:
      - /var/log/**/*.log
      - /var/log/syslog
  metrics:
    host: 
      enabled: true
    process:
      enabled: false

# otel_config_overrides:
#   exporters:
#     # This is a net new exporter
#     debug:
#       verbosity: detailed
#       sampling_initial: 5
#       sampling_thereafter: 200
#   service:
#     pipelines:
#       # This will override the existing metrics/host_monitoring pipeline and output to stdout debug instead
#       metrics/host_monitoring_host:
#         receivers: [hostmetrics/host-monitoring-host]
#         processors: [memory_limiter]
#         exporters: [debug]
#       # This is a net new pipeline
#       logs/new-pipeline:
#         receivers: [filelog/host_monitoring]
#         processors: [memory_limiter]
#         exporters: [debug]

Use ansible-playbook to configure the Observe Agent with config.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/config.yaml

PLAY [Config Observe Agent] *************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [install observe-agent.yaml] *******************************************************************************************************************
changed: [54.202.68.91]
changed: [35.87.95.246]

PLAY RECAP ******************************************************************************************************************************************
35.87.95.246               : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
54.202.68.91               : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Start the Observe Agent

Debian-based distributions

Create enable.yaml under /etc/ansible/roles/agent/tasks/.

- name : Start Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: enable observe-agent
      ansible.builtin.systemd_service:
        enabled: true
        name: observe-agent
        state: started 

Use ansible-playbook to start the Observe Agent with enable.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/enable.yaml

PLAY [Start Observe Agent] *************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [Enable observe-agent] ************************************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

PLAY RECAP *****************************************************************************************************************************************************************************************
ansible-managed-node-1 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ansible-managed-node-2 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

RedHat-based distributions

Create enable.yaml under /etc/ansible/roles/agent/tasks/.

- name : Start Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: enable observe-agent
      systemd:
        enabled: true
        name: observe-agent
        state: started

Use ansible-playbook to start the Observe Agent with enable.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/enable.yaml

PLAY [Start Observe Agent] **************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [enable observe-agent] *************************************************************************************************************************
changed: [35.87.95.246]
changed: [54.202.68.91]

PLAY RECAP ******************************************************************************************************************************************
35.87.95.246               : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
54.202.68.91               : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Configure application instrumentation

Follow these steps to direct your application’s trace data to Observe:

1. Install the OpenTelemetry App if you have not already done so.

2. Create a token associated with OpenTelemetry App’s datastream.

3. Open the agent config file /etc/ansible/roles/agent/templates/observe-agent.yaml.j2 with superuser permissions which will allow you to edit and save changes to the file.

4. Use the otel_config_overrides section to configure an additional OTLP/HTTP Exporter. Replace <YOUR ANOTHER OBSERVE TOKEN> (your instance’s token) and <YOUR OBSERVE COLLECTION ENDPOINT> (your instance’s collection endpoint) on each host.

# this should be added to the existing observe-agent.yaml.j2
otel_config_overrides:
  exporters:
    otlphttp/observe-traces:
      # (ex: https://123456789012.collect.observeinc.com/v2/otel)
      endpoint: "<YOUR OBSERVE COLLECTION ENDPOINT>/v2/otel"
      headers:
        # (ex: Bearer a1b2c3d4e5f6g7h8i9k0:l1m2n3o4p5q6r7s8t9u0v1w2x3y4z5a6)
        authorization: "Bearer <YOUR ANOTHER OBSERVE TOKEN>"
      sending_queue:
        num_consumers: 4
        queue_size: 100
      retry_on_failure:
        enabled: true
      compression: zstd
  service:
    pipelines:
      traces/forward:
        receivers: [otlp]
        processors: [resourcedetection, resourcedetection/cloud]
        exporters: [otlphttp/observe-traces]

5. Restart the Observe Agent Follow the Upgrade the Observe Agent section to restart the Observe Agent.

6. Once the Observe Agent is up and running on a Linux host, you can configure your application running on the same Linux host to send telemetry data to the Observe Agent using one of the following addresses:

Note

When setting up the endpoint to send traces, make sure you use the path that your OTLP library requires. Some libraries need traces to go to /v1/traces, while others expect them at the root path /.

• OTLP/HTTP endpoint: http://localhost:4318

• OTLP/grpc endpoint: http://localhost:4317

7. Explore your trace data using the Trace Explorer and the Service Explorer

Uninstall the Observe Agent

Debian-based distributions

Create uninstall.yaml under /etc/ansible/roles/agent/tasks/.

- name : Uninstall Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: stop observe-agent
      ansible.builtin.systemd_service:
        name: observe-agent
        state: stopped
    - name: purge observe-agent
      ansible.builtin.apt:
        name: observe-agent
        state: absent
        purge: true 

Use ansible-playbook to start the Observe Agent with uninstall.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/uninstall.yaml

PLAY [Uninstall Observe Agent] *****************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [stop observe-agent] **********************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [purge observe-agent] *********************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

PLAY RECAP *************************************************************************************************************************************************************************
ansible-managed-node-1 : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ansible-managed-node-2 : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

RedHat-based distributions

Create uninstall.yaml under /etc/ansible/roles/agent/tasks/.

- name : Uninstall Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: stop observe-agent
      systemd:
        name: observe-agent
        state: stopped
    - name: purge observe-agent
      yum:
        name: observe-agent
        state: absent

Use ansible-playbook to start the Observe Agent with uninstall.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/uninstall.yaml

PLAY [Uninstall Observe Agent] **********************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [stop observe-agent] ***************************************************************************************************************************
ok: [54.202.68.91]
ok: [35.87.95.246]

TASK [purge observe-agent] **************************************************************************************************************************
changed: [54.202.68.91]
changed: [35.87.95.246]

PLAY RECAP ******************************************************************************************************************************************
35.87.95.246               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
54.202.68.91               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Upgrade the Observe Agent

Warning

Observe Agent v2.0.0 includes breaking changes. Learn more about these changes in Upgrade to Observe Agent v2.0.0 Observe Agent v1.0.0 includes breaking changes. Learn more about these changes in Upgrade to Observe Agent v1.0.0

Debian-based distributions

Create upgrade.yaml under /etc/ansible/roles/agent/tasks/.

- name : Upgrade Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: upgrade observe-agent package
      ansible.builtin.apt:
        install_recommends: no
        name: observe-agent
        update_cache: yes
        only_upgrade: true
    - name: restart observe-agent
      ansible.builtin.systemd_service:
        enabled: true
        state: restarted
        name: observe-agent

Use ansible-playbook to upgrade and restart the Observe Agent with upgrade.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook -i /etc/ansible/hosts/my.aws_ec2.ini --private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/upgrade.yaml

PLAY [Upgrade Observe Agent] *******************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [upgrade observe-agent package] ***********************************************************************************************************************************************
ok: [ansible-managed-node-1]
ok: [ansible-managed-node-2]

TASK [restart observe-agent] *******************************************************************************************************************************************************
changed: [ansible-managed-node-1]
changed: [ansible-managed-node-2]

PLAY RECAP *************************************************************************************************************************************************************************
ansible-managed-node-1 : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
ansible-managed-node-2 : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  

RedHat-based distributions

Create upgrade.yaml under /etc/ansible/roles/agent/tasks/.

- name : Upgrade Observe Agent
  hosts: all
  become: yes
  tasks:
    - name: upgrade observe-agent package
      yum:
        name: observe-agent
        disable_gpg_check: yes
        state: latest
        update_cache: yes
    - name: restart observe-agent
      systemd:
        enabled: true
        name: observe-agent
        state: restarted

Use ansible-playbook to upgrade and restart the Observe Agent with upgrade.yaml. Your ansible-playbook could be similar to the following one.

$ ansible-playbook –private-key observe-agent-test.pem /etc/ansible/roles/agent/tasks/upgrade.yaml

PLAY [Upgrade Observe Agent] *****************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************** ok: [35.87.95.246] ok: [54.202.68.91]

TASK [upgrade observe-agent package] ********************************************************************************************************************************************************************************************* ok: [54.202.68.91] ok: [35.87.95.246]

TASK [restart observe-agent] ***************************************************************************************************************************************************************************************************** changed: [54.202.68.91] changed: [35.87.95.246]

PLAY RECAP *********************************************************************************************************************************************************************************************************************** 35.87.95.246 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 54.202.68.91 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0