Algorithms and Gadgets

Overview

FA Algorithms may or may not include gadgets. Some algorithms are enabled by default. Others need to have flow exporting devices applied to them. A few algorithms need to have thresholds configured which can modified from the default.

FA Gadgets that can be added to Dashboard:

  • Flow Analytics Summary: The overall status of all algorithms and the total runtime and count of violations across all algorithms. Algorithms can be ordered alphabetically or by order of execution. The FA Configuration page can be opened with by clicking the button at the top left of this gadget.
  • Flow Reports Thresholds: Saved reports that are given a threshold to compare against every five minutes show up in this gadget.
  • Medianet Jitter Violation: Jitter values as reported by the Medianet flows that exceed the threshold defined in this algorithm. The default threshold is 80ms.
  • Network Volume: The scale of the traffic traversing through the core network. It lists the volume of unique traffic on the network for the last 5 minute vs. last 30 hours. Only include a few core routers/switches in this algorithm.
  • Analytics Violation Overview: Top Flow Analytics policy violation summary with violations counts for the Last 5 minutes, Last Hour, and Overall time.
  • Policies Violated: Last 24 hour report of top alarm policies violated with violations counts.
  • Recent Alarms/Recent Alarms by Violator: Violations listing by policy and violator, with threat heat maps included.
  • Threats: Summary report of top Flow Analytics algorithm violations.
  • Threat Index: Last 24 hour report of top violators by threat index values.
  • Top Applications: Top Applications across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Conversations: Top Conversations across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Countries: Top Countries across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Flows: Top Flow sending end systems across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Hosts: Top Hosts sending data across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Network Transports: Top Transport Layer Protocols across selected flow exporting devices. Alarms trigger for protocols that appear that haven’t been approved. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Rev 2nd lvl Domains: Top reverse 2nd level domains across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Subnet Traffic: Top IP Subnets across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.
  • Top Violators: Last 24 hour report of top alarm violators.
  • Top Well Known Ports: Top ports across selected flow exporting devices. This algorithm is on by default across all flow exporting devices that are exporting the necessary fields.

Note

Some gadgets include algorithms that should only be run against core routers/switches. Watch the Flow Analytics Summary gadget for algorithms that are taking an excessive amount of time to run. Everything needs to finish in under 5 minutes (300 seconds).

FA Algorithms that don’t include Gadgets:

Be sure to exclude certain hosts from select algorithms to avoid false positives. This can easily be done from the alarms tab as well by clicking on the host. The interface will prompt for the exclude confirmation.

  • Indicator Correlation Event: This algorithm escalates multiple Indicator of Compromise (IOC) and security events for a single host to a new alarm on the security event BB. While a single IOC may be indicative of malware, it is much more likely to be a real security concern if there are multiple indicators. By default, This algorithm correlates multiple IOCs along with any events posted to the Security Event BB and issues an alarm for any host that has three or more entries in the IOC and Security Event bulletin boards. Each of the contributing algorithms will be listed in the alarm message.

    By default, three different algorithms are required, the threshold setting for Indicator Correlation can be adjusted.

  • Breach Attempt Detection: This algorithm is examining flow behaviors that may indicate a brute force password attack on an internal IP address. This is accomplished by examining the flow, byte, and packet counts being exchanged in short-duration completed flows between one source and one destination. Specific behaviors are observed for common attack vectors such as SSH, LDAP and RDP. If the number of flows that match these characteristics exceeds the alarm threshold, an alarm will be raised. The default flow count threshold is 100. Either IP address can be excluded from triggering this alarm. This algorithm is enabled by default across all flow exporting devices that are exporting the necessary fields.

  • DDoS Detection: Identifies a Distributed Denial of Service attack targeting the protected network space. DDoS attacks are often launched by a BotNet, and “reflection attacks” are becoming more common. Scrutinizer may identify attacks against the network as “reflection attacks” if they meet the criteria.

    There are four settings that are used to adjust the sensitivity of the DDoS detection algorithm:

    • DDoS Packet Deviation (default: 10) and DDoS Bytes Deviation (default: 10) - These settings control how similar the flows associated with the attack must be. The standard deviation of the byte count and packet counts associated with the flows must be less than this setting.
    • DDoS Flows (default: 4) controls the minimum number flows used to identify attacking hosts. The sensitivity of the DDoS attack can be reduced by increasing this setting to six or higher.
    • DDoS Unique Hosts (default: 200) controls the threshold for the minimum number of hosts that have sent flows that match the other characteristics required to trigger the alarm.
  • Denied Flows Firewall: Triggers an alarm for internal IP addresses sending to external IP addresses that cause greater than the threshold of denied flows. The default threshold is set to 5 denied flows. Either the source or destination IP address can be excluded from triggering this alarm.

  • DRDoS Detection: Identifies a Distributed Reflection Denial of Service attack targeted at the protected network space. DRDoS attacks are often launched by a BotNet, and “reflection attacks” are becoming more common. Scrutinizer may identify attacks against the network as “reflection attacks” if they meet the following criteria.

    Scrutinizer detects the following ten Distributed Reflection Denial of Service (DRDoS) Attacks:

    • DNS
    • NTP
    • SNMP
    • SSDP
    • Chargen
    • NetBIOS Name Server
    • RPC Portmap
    • Sentinel
    • Quote of the Day
    • Trivial File Transfer Protocol

    There is an option to enable or disable a specific reflection attack via Admin / Settings / Flow Analytics Configuration / DRDoS Detection / Settings.

  • DNS Hits: Triggers an alarm when a host initiates an excessive number of DNS queries. This identifies hosts that perform an inordinate number of DNS lookups. To do this, set the flow threshold to a large value that reflects normal behavior on the network. The default threshold is 2500 DNS flows in five minutes. Either the source or destination IP address can be excluded from triggering this alarm.

  • FIN Scan: Alerts when a FIN scan is detected. FIN scans are often used as reconnaissance prior to an attack. They are considered to be a “stealthy scan” as they may be able to pass through firewalls, allowing an attacker to identify additional information about hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • Host Reputation: This algorithm maintains a current list of active Tor nodes that should be monitored. Some malware families use Tor for Command and Control communications. White-list users who are authorized to use Tor and regard other uses as suspicious. This algorithm will also monitor any IP address lists that can be provided as custom lists. See below.

    • Custom List: The Host Reputation algorithm also supports the use of custom lists where the user can add additional reputation lists to the system. A custom list of IP addresses can be imported into the Host Reputation algorithm. To do this, Host Reputation needs to be enabled and this list will be compared with traffic on the devices selected under “Configured Flow Analytics”.

      • To Enable Host Reputation:

        • Go to Admin > Settings > FA Configuration.
        • Expand the “Host Reputation Monitor”.
        • Make sure Disabled is not checked and exporters are being included.
      • To Create Custom Lists:

        • The IP Addresses need to be in a file with a single address on each line.

        • The name of the file will be the Threat Category Name and the Alarm Policy Name.

        • The file must have a .import file extension; for example: “CustomThreatList.import”.

        • The file must be placed in the /home/plixer/scrutinizer/files/threats/ directory.

        • Once an hour, this file will be imported into Scrutinizer and used for the next hour of processing.

        • To force a new file import to become active immediately, run:

        • After the import, the Alarms Policy can be modified to change the threat_multiplier from the default of 0.

  • ICMP Destination Unreachable: This alarm is generated when a large number of ICMP destination unreachable messages have been sent to the suspect IP address. This may happen as a result of scanning activity, misconfiguration, or network errors. ICMP Destination Unreachable is a message that comes back from a destination host or the destination host gateway to indicate that the destination is unreachable for one reason or another. The default threshold is 100 destination unreachable messages. Either the source or destination IP address can be excluded from triggering this alarm.

  • ICMP Port Unreachable: This alarm is generated when a large number of ICMP port unreachable messages have been sent to the suspect IP address. This may happen as a result of scanning activity, misconfiguration, or network errors. ICMP Port Unreachable is a message that comes back from the destination host gateway to indicate that the destination port is unavailable for the transport protocol. The default threshold is 100 port unreachable messages. Either the source or destination IP address can be excluded from triggering this alarm.

  • IP Address Violation: By default, this algorithm allows all subnets. Once subnets are defined, any flow that contains an IP address where either the source or destination IP address isn’t in an allowed subnet, an event will trigger. In other words, if in a single flow both source and destination IP addresses are outside of the allowed subnets, an event will be triggered. A common use of this algorithm is to identify unknown or unauthorized internal network addresses that are communicating with the Internet.

  • Medianet Jitter Violations: This algorithm compares the jitter values as reported by the Medianet flows to the threshold defined by the user in the Settings section of this algorithm. The default threshold is 80ms.

  • Multicast Violations: Any multicast traffic that exceeds the threshold that isn’t excluded will violate this algorithm. The default threshold is 1,000,000 and the minimum that can be set is 100,000.

  • NULL Scan: Alerts when a NULL scan is detected. NULL scans are a TCP scan with all TCP Flags cleared to zero. This scan is sometimes used as a reconnaissance tactic prior to an attack and is considered to be stealthy because often times it is able to pass through firewalls. Eluding a firewall makes it easier for an attacker to identify additional information about the hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • Odd TCP Flags Scan: Alerts when a scan is detected using unusual TCP Flag combinations. These types of scans may allow an attacker to identify additional information about hosts on the network. The default threshold is 100 unique scan (aka flows) in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • P2P Detection: Peer to Peer (P2P) traffic such as BitTorrent are identified by this algorithm. The default threshold is a P2P session involving over 100 external hosts, which will detect most P2P applications. However, there are several P2P applications that are stealthier. Experimenting with lower thresholds or periodically lowering the threshold to about 20 will allow the security admins to determine if other “low and slow” P2P traffic is on the network.

  • Persistent Flow Risk: Alerts when a persistent flow is detected. Persistent flows are a strong indicator of VPNs, proxy traffic, remote desktop technologies, or other means of covert communication. The default threshold for a flow to be considered persistent is 12 hours. In addition to the temporal threshold an optional ratio threshold is available to identify the relationship of traffic as it pertains to ingressing or egressing the network. The default PCR threshold is set to .9, identifying persistent flows where the ratio indicates more traffic is destined outside the network.

  • RST/ACK Detection: Alerts when a large number of TCP flows containing only RST and ACK flags have been detected that are sending to a single destination. These flows indicate that a connection attempt was made on the host sending the RST/ACK flow, and was rejected. This algorithm may detect other scan types used by an attacker to identify additional information about the hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • SYN Scan: Alerts when a SYN scan is detected. SYN scans are a TCP scan with the TCP SYN flag set. This scan is often used as reconnaissance prior to an attack as it is fast and somewhat stealthy. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • TCP Scan: Alerts when a possible TCP scan is detected from an exporter that does not provide TCP flag information. These types of scans may allow an attacker to identify additional information about hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

  • UDP Scan: Alerts when a possible UDP scan is detected. These types of scans may allow an attacker to identify additional information about hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

Note

If company policy allows P2P traffic on the network, then it is unwise to enable this alarm as it will often detect P2P control traffic as a UDP Scan violation.

  • XMAS Scan: Alerts when a XMAS scan is detected. XMAS scans are a TCP scan with the FIN, PSH, and URG TCP flags set. This scan is often used as reconnaissance prior to an attack. They are considered to be a “stealthy scan” as they may be able to pass through firewalls, allowing an attacker to identify additional information about hosts on the network. The default threshold is 100 unique scan flows in five minutes. Internal IP addresses that are allowed to scan the internal network, such as security team members and vulnerability scanners, should be entered into the IP exclusions list. Either the source or destination IP address can be excluded from triggering this alarm.

Note

By default, all of the scan algorithms are looking for “internal to internal” and “internal to external” scanning activity. Security admins have the option to control which scanning directions the different algorithms look for, including “external to internal”, which would normally be used to monitor public facing IP addresses listed in an IP Group. Within each of the scanning algorithms, the settings screen provides a directional control option.

FA Algorithms that Require FlowPro Defender

BotNet Detection

(Formerly named NXDomain detection)

This alarm is generated when a large number of unique DNS name lookups have failed. When a DNS lookup fails, a reply commonly known as NXDOMAIN is returned. By monitoring the number of NXDOMAINs detected as well as the DNS name looked up, behavior normally associated with a class of malware that uses Domain Generation Algorithms (DGAs) can be detected.

The default threshold is 100 unique DNS lookup failure (NXDOMAIN) messages in five minutes. Either the source or destination IP address can be excluded from triggering this alarm.

DNS Command and Control

This algorithm monitors the use of DNS TXT messages traversing the network perimeter as detected by FlowPro Defender. DNS TXT messages provide a means of sending information into and out of the protected network over DNS, even when external DNS servers. are blocked. This technique is used by malware as a method of controlling compromised assets within the network and to extract information back out. Additionally, some legitimate companies also use this method to communicate as a means to “phone home” from their applications to the developer site.

The algorithm will detect inbound, outbound, and bidirectional communications using DNS TXT messages. Thresholds may be set based either on the number of DNS TXT messages or the number of bytes observed in the DNS TXT messages within a five minute period. The default setting is for any detected traffic to alarm, and alarm aggregation defaults to 120 minutes.

To suppress alarms from authorized applications in the network, the domain generating the alarm message can be added to to the “Trusted Domain” list on FlowPro Defender. See the Trusted Domain List discussion below.

DNS Data Leak

This algorithm monitors the practice of encoding information into a DNS lookup message that has no intention of returning a valid IP address or making an actual connection to a remote device. When this happens, the local DNS server will fail to find the DNS name in its cache, and will pass the name out of the network to where it will eventually reach the authoritative server for the domain. At that point, the owner of the authoritative server can decode the information embedded in the name, and may respond with a “no existing domain” response, or return a non-routable address.

FlowPro Defender uses proprietary detection algorithms to identify suspicious DNS names that may contain encoded data, and passes this information to Scrutinizer where it is processed by the DNS Data Leak algorithm. Thresholds may be set based either on the number of suspicious DNS names or the number of bytes observed in the suspicious DNS name within a five minute period. The default setting is for any detected traffic to alarm, and alarm aggregation defaults to 120 minutes.

Domain Reputation

Domain reputation provides much more accurate alarming with a dramatic decrease in the number of false positive alarms as compared to IP based Host Reputation. The domain list is provided by Plixer and is updated each hour and currently contains several hundred thousand known bad domains.

FlowPro Defender performs the actual monitoring, and when it detects a domain with a poor reputation, it passes the information to Scrutinizer for additional processing. The default setting is for any detected traffic to alarm, and alarm aggregation defaults to disabled which means that all DNS lookups observed will result in a unique alarm.

To suppress alarms from authorized applications in the network, the domain generating the alarm message can be added to the “Trusted Domain” list on FlowPro Defender. See the Trusted Domain List discussion below.

Malware Behavior Detection

This specific alarm is correlating IP address lookups (i.e. what is my IP address) activity which is commonly performed by malware shortly after the initial compromise with the detection of the BotNet alarm or with a Domain Reputation alert. In other words, this algorithm looks for the following correlation:

  • IP address lookup combined with a Domain Reputation trigger
  • IP address lookup combined with a BotNet trigger

When either of the two events is detected, this algorithm is triggered as this behavior is a very strong indicator of a compromised asset.

Malware Domain Communications

This algorithm combines the Domain Reputation algorithm with communications detected going to the IP address that was resolved. Scrutinizer and Defender have detected the following sequence of events:

  1. Defender contains a list, updated every 10 minutes, of several hundred thousand known malware domains created by forensic analysis of the actual malware. These are very high confidence domains.
  2. Defender monitors all of the DNS resolution requests, and generates an IOC (Indicator of Compromise) alert on detection of a match with a malware domain and saves the resolved “Malware IP Address”. This only rates an “IOC” as a browser may “pre-fetch”, or resolve an address, for all of the links on a web page. Browsers like Chrome do this to make the browsing experience feel faster. However, as yet, no connection to the malware site has been made.
  3. Scrutinizer then examines all flows for any communications with the “Malware IP Address” resolved by Defender. On detection of any flows to or from that address, a connection to the malware site has been established, and a Malware Domain Communication alert is triggered.

Note

For this algorithm to work, the user must turn ON host indexing. This setting is available in Admin / Settings / System Preferences

Correlation Algorithms

These algorithms demonstrate Plixer’s cyber threat correlation capability. Correlation of multiple network behaviors over a long time period provides detection systems with more information allowing for a higher accuracy with fewer false positive alarms.

Below are the Correlation Algorithms available in Flow Analytics:

Trusted Domain List

A “trusted domain list”, often called a whitelist, is preconfigured on FlowPro Defender to suppress alarms involving specific domains. The default whitelist contains five entries. Add or remove entries as necessary to best fit the local environment.

  • mcafee.com
  • sophos.com
  • sophosxl.net
  • webcfs03.com
  • apple.com

mcafee.com suppresses DNS Data Leak alarms from McAfee AntiVirus software. McAfee encodes information from the anti-virus clients on the network into very long and complex DNS names and captures this information at their DNS server. This is exactly the type of behavior that the DNS Data Leak algorithm is looking for as this technique is also used by some forms of malware.

sophos.com and sophosxl.net are related to the Sophos Anti-virus software, and it uses multiple techniques to get information in and out of the network using DNS. In addition to using the same technique as McAfee to send information back to their servers, they also use DNS TXT messages to send information back in to the clients on the internal network. Use of DNS TXT messages to exchange information with an external host is also used by some malware families, and the DNS Command and Control algorithm will alarm on this type of activity. This will prevent Sophos from generating either DNS Data Leak or DNS Command and Control alarms.

webcfs03.com belongs to SonicWALL, and will also generate DNS Data Leak alarms.

apple.com uses DNS TXT messages to apparently exchange settings with their NTP server. This will alarm as a DNS Command and Control alarm.

It is possible to have authorized software within the local networks that abuse the DNS to bypass firewalls for data communications. If this is the case, add the domain(s) involved with the software to the Trusted Domain list as described below. Once they have been configured for the local environment, any other traffic using DNS to communicate will be worth additional investigation.

To Modify the Trusted Domain List:

  1. Log on to the FlowPro Defender
  2. Enter: “edit trusteddomains”
  3. Modify the file contents as desired
  4. enter control-x, and select “Y” to save the changes
  5. press enter to accept the file name.
  6. quit

Untrusted Domain Lists

FlowPro Defender supports both the use of a domain reputation list that is downloaded from Plixer, as well as allowing for the addition of a unique list.

Plixer Domain Reputation List

FlowPro Defender downloads a list of domains from Plixer once each hour. These are domains that have been determined to be “bad domains” with a high probability, and this list is used in the “Domain Reputation” and “Malware Behavior Detection” algorithms. Use of this list can be controlled by the FlowPro Defender:

  1. Log on to the FlowPro Defender
  2. Enter: “edit plixer.ini”
  3. To enable the list (default is enabled), set the value enableDomainReputationList=1
  4. or, to disable the list, set the value enableDomainReputationList=0
  5. enter control-x, and select “Y” to save changes
  6. quit

User Defined Domain Lists

The Plixer Domain Reputation list can be augmented by creating one or more lists that contain domains that the system should alarm on. The rules for the domain lists are:

  1. The DNS name must contain at least 2 labels, which is often called a second level domain, or 2LD for short (for example, google.com) and no more than 3 labels (maps.google.com), or a 3LD.
  2. The labels must contain between 1 and 63 characters, as is required to be a legitimate domain name.
  3. Entries that do not match these requirements will be ignored.

To create a list of domains to detect domainReputation violations:

  1. Log on to the FlowPro Defender
  2. Enter: “edit my_domain_list_name” NOTE: Do NOT enter a file extension. This will be automatically assigned.
  3. Modify the file contents as desired
  4. enter control-x, and select “Y” to save changes
  5. press enter to accept the file name.
  6. quit

To Enable a Domain List

  1. Log on to the FlowPro Defender
  2. show domainlists
  3. Enter: “enable domain_list_name”
  4. quit

To Disable a Domain List

  1. Log on to the FlowPro Defender
  2. show domainlists
  3. Enter: “disable domain_list_name”
  4. quit

Important

Hosts can easily be excluded from certain algorithms by clicking on the IP address in the Alarms Tab. This will pop up the Exclude Hosts table where the IP address can then be excluded from other algorithms as well.