Basic Threat Intel Integration (Public Preview)¶
The Basic Threat Intel app ingests data from open source threat intelligence lists and infrastructure as service providers, using Observe pollers.
Observe helps you monitor the public networks that your systems contact by using resource sets that include information about known bad addresses, known infrastructure addresses, and more.
Viewing Basic Threat Intel Data in Observe¶
The following example provides a template to follow for instances where the source data is not provided. The Basic Threat Intel app provides pre-built configurations for some Observe apps, see Basic Threat Intel App Installation for details.
Do you have public IP Addresses?¶
Starting from a dataset containing IP Addresses, you can query the Resource Sets provided by the Basic Threat Intel app to determine if there are matches. For example, using AWS EC2 Cloudtrail data, and click Open Worksheet:
Select the columns that you want, such as
eventTime
andsourceIPAddress
.pick_col eventTime, sourceIPAddress

Figure 1 - Using AWS EC2 Cloudtrail Event Stream to Filter IP Addresses
2. Filter the IP addresses to valid IPv4 only.
filter match_regex(sourceIPAddress, /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/, "i")

Figure 2 - Filtering Valid IPv4 Addresses
3. Convert the sourceIPAddress
to int64. You must do this because the boundaries between public, reserved, and private space predate CIDR and do not neatly align.
make_col src:ipv4_to_int64(sourceIPAddress)

Figure 3 - Converting IPv4 Addresses to int64 Format
4. Under the Inputs tab, add the Threat\_Intel\_Basic/Public IP Ranges
Resource Set (leaving Data selected).
5. Join this data set in order to remove private and reserved IPs from your source data.
join on(src>@"Threat_Intel_Basic/Public IP Ranges".ipv4_range_start
AND src<@"Threat_Intel_Basic/Public IP Ranges".ipv4_range_end),
public_range:@"Threat_Intel_Basic/Public IP Ranges".ipv4_range
6. Name this stage All Remote IP Addresses and publish your data set.
Are these IP addresses from IaaS Providers?¶
Open your All Remote IP Addresses dataset and click Create Worksheet to start a new dataset based on the first one.
AWS IP Address Ranges
Change the name of Stage 1 to All Remotes.
Click Link New Stage at the bottom of the All Remotes data display section.
Under the Inputs tab, add the Threat_Intel_Basic/AWS IP Ranges Resource Set with selected data.

Figure 4 - Adding Input to All Remotes Worksheet
Note
When adding Inputs to a Stage, be sure to change the Inputs Name to match the Source Name by removing any trailing underscores, dashes, letters, or numbers.
4. Join this data set in order to recognize the use of Amazon Web Services-owned IP Ranges.
join on(src>[email protected]"Threat_Intel_Basic/AWS IP Ranges".ipv4_range_start
AND src<[email protected]"Threat_Intel_Basic/AWS IP Ranges".ipv4_range_end),
subnet:@"Threat_Intel_Basic/AWS IP Ranges".ip_prefix
make_col subnet_provider:"aws"

Figure 5 - Adding AWS IP Addresses to All Remotes Worksheet
5. Name the new stage AWS and save your work.
Azure IP Address Ranges
Use the Manage Content list to select the All Remotes stage.
Click Link New Stage at the bottom of the data display section.
Under the Inputs tab, add the Threat_Intel_Basic/Azure IP Ranges Resource Set with selected data.
Join this data set in order to recognize the use of Microsoft Azure-owned IP Ranges.
join on(src>[email protected]"Threat_Intel_Basic/Azure IP Ranges".ipv4_range_start
AND src<[email protected]"Threat_Intel_Basic/Azure IP Ranges".ipv4_range_end),
subnet:@"Threat_Intel_Basic/Azure IP Ranges".ip_prefix
make_col subnet_provider:"azure"
5. Name the new stage Azure and save your work.
Google Cloud Provider IP Address Ranges
Use the Manage Content list to select the All Remotes stage
Click the Link New Stage at the bottom of the data display section.
Under the Inputs tab, add the Threat_Intel_Basic/GCP IP Ranges Resource Set with data selected.
Join this data set in order to recognize the use of Google Cloud Platform-owned IP Ranges.
join on(src>[email protected]"Threat_Intel_Basic/GCP IP Ranges".ipv4_range_start
AND src<[email protected]"Threat_Intel_Basic/GCP IP Ranges".ipv4_range_end),
subnet:@"Threat_Intel_Basic/GCP IP Ranges".ip_prefix
make_col subnet_provider:"gcp"
5. Name the new stage GCP and save your work.
OCI IP Address Ranges
Use the Manage Content list to select the All Remotes stage.
Click Link New Stage at the bottom of the data display section.
Under the Inputs tab, add the Threat_Intel_Basic/OCI IP Ranges Resource Set with selected data.
Join this data set in order to recognize the use of Oracle Cloud Infrastructure-owned IP Ranges.
join on(src>[email protected]"Threat_Intel_Basic/OCI IP Ranges".ipv4_range_start
AND src<[email protected]"Threat_Intel_Basic/OCI IP Ranges".ipv4_range_end),
subnet:@"Threat_Intel_Basic/OCI IP Ranges".ip_prefix
make_col subnet_provider:"oci"
5. Name the new stage OCI and save your work.
Joining the Resource Sets
Click Link New Stage at the bottom of the OCI data display section.
Under the Inputs tab, add the AWS, Azure, and GCP stages you just created.
Union them together using the following OPAL:
union @AWS, @Azure, @GCP
4. Name the new stage Unioned and save your work.
5. Click Link New Stage at the bottom of the Unioned data section.
6. Under the Inputs tab, add the All Remote stage that you created in the first step.
7. Join these data sets to recover the connections that did not go to IaaS providers:
fulljoin on(sourceIPAddress[email protected]"Unioned".sourceIPAddress),
subnet:@"Unioned".subnet,
subnet_provider:@"Unioned".subnet_provider
make_col subnet:if(is_null(subnet), "unknown", subnet)
make_col subnet_provider:if(is_null(subnet_provider), "none", subnet_provider)
8. Name the new stage Connections to Cloud and Non-Cloud, save your work, and then publish as a new data set.
Are these IP Addresses on Basic Threat Intel Lists?¶
If you have IP addresses on the Basic Threat Intel lists, use the following steps:
Open your Connections to Cloud and Non-Cloud dataset and click Create Worksheet to start a new dataset based on all public connections, decorated with the IaaS provider information where available.
Change the name of Stage 1 to All Remotes with IaaS.
Click Link New Stage at the bottom of the All Remotes with IaaS data display section.
Under the Inputs tab, add the Threat_Intel_Basic/CI Army Block IP Resource Set with selected data.
Join this data set in order to recognize the use of CI Army Block IP Ranges:
join on(sourceIPAddress[email protected]"Threat_Intel_Basic/CI Army Block IP".IP),
CI_Army_IP:@"Threat_Intel_Basic/CI Army Block IP".IP
make_col threat_intel_provider:"CI_Army"
5. Name the new stage CI Army Matches and save your work.
6. Use the Manage Content list to select the All Remotes with IaaS stage. Click Link New Stage at the bottom of the data display section.
7.Under the Inputs tab, add the Threat_Intel_Basic/Feodo Block IP Resource Set with selected data.
8. Join this data set in order to recognize the use of Feodo Block IP Ranges:
join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Feodo Block IP".block)),
Emerging_Threats_IP:@"Threat_Intel_Basic/Feodo Block IP".block
make_col threat_intel_provider:"Feodo"
9. Name the new stage Feodo Matches and save your work.
Basic and Emerging Threat IP Addresses
Use the Manage Content list to select the All Remotes with IaaS stage. Click Link New Stage at the bottom of the data display section.
Under the Inputs tab, add the Threat_Intel_Basic/Emerging Threats Block IP Resource Set with selected data.
Join this data set in order to recognize the use of Emerging Threats Block IP Ranges:
join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Emerging Threats Block IP".block)),
Emerging_Threats_IP:@"Threat_Intel_Basic/Emerging Threats Block IP".block
make_col threat_intel_provider:"Emerging_Threats"
4. Name the new stage Emerging Threats Matches and save your work.
Spamhaus Block IP Addresses
Use the Manage Content list to select the All Remotes with IaaS stage.
Click Link New Stage at the bottom of the data display section.
Under the Inputs tab, add the Threat_Intel_Basic/Spamhaus Block IP Resource Set with selected data.
Join this data set in order to recognize the use of Spamhaus Block IP Ranges
join on(ipv4_address_in_network(sourceIPAddress,@"Threat_Intel_Basic/Spamhaus Block IP".block)),
Spamhaus_IP:@"Threat_Intel_Basic/Spamhaus Block IP".block
make_col threat_intel_provider:"Spamhaus"
5. Name the new stage Spamhaus Matches and save your work.
Joining the Resource Sets
Click Link New Stage at the bottom of the Spamhaus Matches data display section.
Under the Inputs tab, add the CI Army, Feodo, and Emerging Threats stages you just created.
Union them together using the following OPAL:
union @"CI Army Matches", @"Feodo", @"Emerging Threats Matches"
4. Name the new stage Threatening Connections, save your work, and then publish as a new data set.
Dashboards and Monitors¶
To review the Dashboards associated with your work, return to the Connections to Cloud and Non-Cloud data set, click Dashboards. This displays a dashboard of network traffic split by the IaaS providers, if you have any data.

Figure 6 - Viewing the Connections to Cloud and Non-Cloud Dashboard
Return to the Threatening Connections Dataset.
Resource Sets¶
Installing the Basic Threat Intel app provides the following Datasets to use with your data:
Resource Datasets
AWS IP Ranges
AWS IPv6 Ranges
Azure IP Ranges
Azure IPv6 Ranges
Bazaar Block Process
CI Army Block IP
Custom Threat IP Ranges
Dan’s Tor Nodes IPs
Dan’s Tor Nodes v6 IPs
Emerging Threats Block IP
Feodo Block IP
GCP IP Ranges
GCP IPv6 Ranges
OCI IP Ranges
Majestic Million (top 50,000)
Public IP Ranges
Spamhaus Block IP
URLHaus Block Domain
Unified Resource Datasets
Unified Hash Threatlists
Unified Hosts-Domains Threatlist
Unified IPv4 Threatlist
Unified URL Threatlist
Event Streams
Host OSQuery Host Contact with Cloud Platform Vendor IPs
Host OSQuery Host Contact with Threatening IPs
Host OSQuery Host Execution of Threatening Processes
Monitor Templates¶
When you install the Basic Threat Intel app, the app also installs two Monitor templates by default:
(TEMPLATE) Threat_Intel_Basic/Threatening Connections Count is High
(TEMPLATE) Threat_Intel_Basic/Threatening Executions Count is High
(TEMPLATE) Threat_Intel_Basic/Threatening Connections to Domains Count is High
As Observe ingests your data, these out-of-the-box datasets and boards display information about the security of your systems.
Exporting Threat Intel Data¶
As a security analyst, you may want to automatically export lists of threat IOCs for further processing or alerting. Observe supports this through use of an API. For instance, you might want to export a list of the threat-listed IP addresses that were in contact with your monitored hosts. A dataset with that information is created if Threat Intel Basic and Host Monitoring are both installed and the Host Monitoring integration is activated in Threat Intel Basic.
Example cURL command for exporting an IP Address list from Observe:
curl "https://$YOUR_HOST.observeinc.com/v1/meta/export/query?interval=4h" -H "Authorization: Bearer $YOUR_CUSTOMERID $YOUR_TOKEN" -H 'Content-Type: application/json' -H 'Accept: text/csv' -d '{"query": {"stages":[{"input":[{"inputName": "Host-OSQuery-Host-Contact-with-Threatening-IPs","datasetId": "$YOUR_DATASET_ID"}],"stageID":"main","pipeline": "statsby count:count()"}]}}'