Basic Threat Intel Integration (Public Preview)

The Basic Threat Intel app ingests data from open source threat intelligence lists and infrastructure as service providers, using Observe pollers.

Observe helps you monitor the public networks that your systems contact by using resource sets that include information about known bad addresses, known infrastructure addresses, and more.

Viewing Basic Threat Intel Data in Observe

The following example provides a template to follow for instances where the source data is not provided. The Basic Threat Intel app provides pre-built configurations for some Observe apps, see Basic Threat Intel App Installation for details.

Do you have public IP Addresses?

Starting from a dataset containing IP Addresses, you can query the Resource Sets provided by the Basic Threat Intel app to determine if there are matches. For example, using AWS EC2 Cloudtrail data, and click Open Worksheet:

  1. Select the columns that you want, such as eventTime and sourceIPAddress.

    pick_col eventTime, sourceIPAddress
Using AWS EC2 Cloudtrail Event Stream to Filter IP Addresses

Figure 1 - Using AWS EC2 Cloudtrail Event Stream to Filter IP Addresses

2. Filter the IP addresses to valid IPv4 only.

  filter match_regex(sourceIPAddress, /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/, "i")
Filtering Valid IPv4 Addresses

Figure 2 - Filtering Valid IPv4 Addresses

3. Convert the sourceIPAddress to int64. You must do this because the boundaries between public, reserved, and private space predate CIDR and do not neatly align.

make_col src:ipv4_to_int64(sourceIPAddress)
Converting IPv4 Addresses to int64

Figure 3 - Converting IPv4 Addresses to int64 Format

4. Under the Inputs tab, add the Threat\_Intel\_Basic/Public IP Ranges Resource Set (leaving Data selected).

5. Join this data set in order to remove private and reserved IPs from your source data.

join on(src>@"Threat_Intel_Basic/Public IP Ranges".ipv4_range_start 
 AND src<@"Threat_Intel_Basic/Public IP Ranges".ipv4_range_end),
 public_range:@"Threat_Intel_Basic/Public IP Ranges".ipv4_range

6. Name this stage All Remote IP Addresses and publish your data set.

Are these IP addresses from IaaS Providers?

Open your All Remote IP Addresses dataset and click Create Worksheet to start a new dataset based on the first one.

AWS IP Address Ranges

  1. Change the name of Stage 1 to All Remotes.

  2. Click Link New Stage at the bottom of the All Remotes data display section.

  3. Under the Inputs tab, add the Threat_Intel_Basic/AWS IP Ranges Resource Set with selected data.

Adding Input to All Remotes Worksheet

Figure 4 - Adding Input to All Remotes Worksheet

Note

When adding Inputs to a Stage, be sure to change the Inputs Name to match the Source Name by removing any trailing underscores, dashes, letters, or numbers.

4. Join this data set in order to recognize the use of Amazon Web Services-owned IP Ranges.

join on(src>[email protected]"Threat_Intel_Basic/AWS IP Ranges".ipv4_range_start 
 AND src<[email protected]"Threat_Intel_Basic/AWS IP Ranges".ipv4_range_end),
   subnet:@"Threat_Intel_Basic/AWS IP Ranges".ip_prefix
make_col subnet_provider:"aws"
Adding AWS IP Addresses to All Remotes Worksheet

Figure 5 - Adding AWS IP Addresses to All Remotes Worksheet

5. Name the new stage AWS and save your work.

Azure IP Address Ranges

  1. Use the Manage Content list to select the All Remotes stage.

  2. Click Link New Stage at the bottom of the data display section.

  3. Under the Inputs tab, add the Threat_Intel_Basic/Azure IP Ranges Resource Set with selected data.

  4. Join this data set in order to recognize the use of Microsoft Azure-owned IP Ranges.

join on(src>[email protected]"Threat_Intel_Basic/Azure IP Ranges".ipv4_range_start 
  AND src<[email protected]"Threat_Intel_Basic/Azure IP Ranges".ipv4_range_end),
    subnet:@"Threat_Intel_Basic/Azure IP Ranges".ip_prefix
make_col subnet_provider:"azure"

5. Name the new stage Azure and save your work.

Google Cloud Provider IP Address Ranges

  1. Use the Manage Content list to select the All Remotes stage

  2. Click the Link New Stage at the bottom of the data display section.

  3. Under the Inputs tab, add the Threat_Intel_Basic/GCP IP Ranges Resource Set with data selected.

  4. Join this data set in order to recognize the use of Google Cloud Platform-owned IP Ranges.

join on(src>[email protected]"Threat_Intel_Basic/GCP IP Ranges".ipv4_range_start 
  AND src<[email protected]"Threat_Intel_Basic/GCP IP Ranges".ipv4_range_end),
    subnet:@"Threat_Intel_Basic/GCP IP Ranges".ip_prefix
make_col subnet_provider:"gcp"

5. Name the new stage GCP and save your work.

OCI IP Address Ranges

  1. Use the Manage Content list to select the All Remotes stage.

  2. Click Link New Stage at the bottom of the data display section.

  3. Under the Inputs tab, add the Threat_Intel_Basic/OCI IP Ranges Resource Set with selected data.

  4. Join this data set in order to recognize the use of Oracle Cloud Infrastructure-owned IP Ranges.

join on(src>[email protected]"Threat_Intel_Basic/OCI IP Ranges".ipv4_range_start 
  AND src<[email protected]"Threat_Intel_Basic/OCI IP Ranges".ipv4_range_end),
    subnet:@"Threat_Intel_Basic/OCI IP Ranges".ip_prefix
make_col subnet_provider:"oci"

5. Name the new stage OCI and save your work.

Joining the Resource Sets

  1. Click Link New Stage at the bottom of the OCI data display section.

  2. Under the Inputs tab, add the AWS, Azure, and GCP stages you just created.

  3. Union them together using the following OPAL:

union @AWS, @Azure, @GCP

4. Name the new stage Unioned and save your work.

5. Click Link New Stage at the bottom of the Unioned data section.

6. Under the Inputs tab, add the All Remote stage that you created in the first step.

7. Join these data sets to recover the connections that did not go to IaaS providers:

fulljoin on(sourceIPAddress[email protected]"Unioned".sourceIPAddress),
subnet:@"Unioned".subnet,
subnet_provider:@"Unioned".subnet_provider
make_col subnet:if(is_null(subnet), "unknown", subnet)
make_col subnet_provider:if(is_null(subnet_provider), "none", subnet_provider)

8. Name the new stage Connections to Cloud and Non-Cloud, save your work, and then publish as a new data set.

Are these IP Addresses on Basic Threat Intel Lists?

If you have IP addresses on the Basic Threat Intel lists, use the following steps:

  1. Open your Connections to Cloud and Non-Cloud dataset and click Create Worksheet to start a new dataset based on all public connections, decorated with the IaaS provider information where available.

  2. Change the name of Stage 1 to All Remotes with IaaS.

  3. Click Link New Stage at the bottom of the All Remotes with IaaS data display section.

  4. Under the Inputs tab, add the Threat_Intel_Basic/CI Army Block IP Resource Set with selected data.

  5. Join this data set in order to recognize the use of CI Army Block IP Ranges:

join on(sourceIPAddress[email protected]"Threat_Intel_Basic/CI Army Block IP".IP),
  CI_Army_IP:@"Threat_Intel_Basic/CI Army Block IP".IP
make_col threat_intel_provider:"CI_Army"

5. Name the new stage CI Army Matches and save your work.

6. Use the Manage Content list to select the All Remotes with IaaS stage. Click Link New Stage at the bottom of the data display section.

7.Under the Inputs tab, add the Threat_Intel_Basic/Feodo Block IP Resource Set with selected data.

8. Join this data set in order to recognize the use of Feodo Block IP Ranges:

join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Feodo Block IP".block)),
  Emerging_Threats_IP:@"Threat_Intel_Basic/Feodo Block IP".block
make_col threat_intel_provider:"Feodo"

9. Name the new stage Feodo Matches and save your work.

Basic and Emerging Threat IP Addresses

  1. Use the Manage Content list to select the All Remotes with IaaS stage. Click Link New Stage at the bottom of the data display section.

  2. Under the Inputs tab, add the Threat_Intel_Basic/Emerging Threats Block IP Resource Set with selected data.

  3. Join this data set in order to recognize the use of Emerging Threats Block IP Ranges:

join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Emerging Threats Block IP".block)),
  Emerging_Threats_IP:@"Threat_Intel_Basic/Emerging Threats Block IP".block
make_col threat_intel_provider:"Emerging_Threats"

4. Name the new stage Emerging Threats Matches and save your work.

Spamhaus Block IP Addresses

  1. Use the Manage Content list to select the All Remotes with IaaS stage.

  2. Click Link New Stage at the bottom of the data display section.

  3. Under the Inputs tab, add the Threat_Intel_Basic/Spamhaus Block IP Resource Set with selected data.

  4. Join this data set in order to recognize the use of Spamhaus Block IP Ranges

join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Spamhaus Block IP".block)),
Spamhaus_IP:@"Threat_Intel_Basic/Spamhaus Block IP".block
make_col threat_intel_provider:"Spamhaus"

5. Name the new stage Spamhaus Matches and save your work.

Joining the Resource Sets

  1. Click Link New Stage at the bottom of the Spamhaus Matches data display section.

  2. Under the Inputs tab, add the CI Army, Feodo, and Emerging Threats stages you just created.

  3. Union them together using the following OPAL:

union @"CI Army Matches", @"Feodo", @"Emerging Threats Matches"

4. Name the new stage Threatening Connections, save your work, and then publish as a new data set.

Dashboards and Monitors

To review the Dashboards associated with your work, return to the Connections to Cloud and Non-Cloud data set, click Dashboards. This displays a dashboard of network traffic split by the IaaS providers, if you have any data.

Viewing the Connections to Cloud and Non-Cloud Dashboard

Figure 6 - Viewing the Connections to Cloud and Non-Cloud Dashboard

Return to the Threatening Connections Dataset.

Resource Sets

Installing the Basic Threat Intel app provides the following Datasets to use with your data:

Resource Datasets

  • AWS IP Ranges

  • AWS IPv6 Ranges

  • Azure IP Ranges

  • Azure IPv6 Ranges

  • Bazaar Block Process

  • CI Army Block IP

  • Custom Threat IP Ranges

  • Dan’s Tor Nodes IPs

  • Dan’s Tor Nodes v6 IPs

  • Emerging Threats Block IP

  • Feodo Block IP

  • GCP IP Ranges

  • GCP IPv6 Ranges

  • OCI IP Ranges

  • Majestic Million (top 50,000)

  • Public IP Ranges

  • Spamhaus Block IP

  • URLHaus Block Domain

Unified Resource Datasets

  • Unified Hash Threatlists

  • Unified Hosts-Domains Threatlist

  • Unified IPv4 Threatlist

  • Unified URL Threatlist

Event Streams

  • Host OSQuery Host Contact with Cloud Platform Vendor IPs

  • Host OSQuery Host Contact with Threatening IPs

  • Host OSQuery Host Execution of Threatening Processes

Monitor Templates

When you install the Basic Threat Intel app, the app also installs two Monitor templates by default:

  • (TEMPLATE) Threat_Intel_Basic/Threatening Connections Count is High

  • (TEMPLATE) Threat_Intel_Basic/Threatening Executions Count is High

  • (TEMPLATE) Threat_Intel_Basic/Threatening Connections to Domains Count is High

As Observe ingests your data, these out-of-the-box datasets and boards display information about the security of your systems.

Exporting Threat Intel Data

As a security analyst, you may want to automatically export lists of threat IOCs for further processing or alerting. Observe supports this through use of an API. For instance, you might want to export a list of the threat-listed IP addresses that were in contact with your monitored hosts. A dataset with that information is created if Threat Intel Basic and Host Monitoring are both installed and the Host Monitoring integration is activated in Threat Intel Basic.

curl "https://$YOUR_HOST.observeinc.com/v1/meta/export/query?interval=4h" -H "Authorization: Bearer $YOUR_CUSTOMERID $YOUR_TOKEN" -H 'Content-Type: application/json' -H 'Accept: text/csv' -d '{"query": {"stages":[{"input":[{"inputName": "Host-OSQuery-Host-Contact-with-Threatening-IPs","datasetId": "$YOUR_DATASET_ID"}],"stageID":"main","pipeline": "statsby count:count()"}]}}'