Data aggregation

The system can save and roll up data in one of two ways:

  • Traditional mode (creates forensic tables). This is the default mode for version 17.x and prior.
  • SAF Mode (creates both summary tables and forensic tables). This is the default mode for version 18.x and higher. Summary tables are condensed flows that represent 100% of the traffic, but by summarizing the data as explained below, reporting is much faster.

Upgrades

When existing customers upgrade from version 17.x and prior to 18.x versions, the aggregation method will stay in Traditional (Forensic) Mode. New installs default to SAF mode. Contact technical support for instructions on how to change this to SAF mode. When running a report and switching between the two types of data, this is referred to as the “Data Mode”.

Traditional mode

Traditional mode creates forensic tables. In this mode, the collector saves 100% of all data in raw format to the 1 minute forensic tables for each router. Every hour it creates a new 1 minute interval table per router. Every 5 minutes, it creates higher intervals using the smaller intervals. This process is called “rollups”.

When the rollups occur for 5 min, 30 min, 2 hr, 12 hr, 1 day and 1 week, two tables are created:

  • Totals: The total in and out byte counts are saved per interface before the data for the forensic table is calculated. This table allows the reporting front end to display accurate total throughput per interface over time and allows the front end to operate with no dependency on SNMP yet still provide accurate total utilization reporting.
  • Forensic: All flows for the time period (e.g. 5 minutes) are aggregated together based on a tuple. Once all flows are aggregated together, the top 1000 (i.e. default) flows based on highest byte count are saved. The non top 1000 flows are dropped. Remember: the totals tables ensure a record of the total in / out utilization per interface over time.

When a report is run on an individual interface with 1 minute interval data, the totals table isn’t needed because the forensic table contains 100% of the data. When a report is run on an individual interface with no filters in 5 minute or higher intervals, both the forensic and totals tables are used in the report. When reporting, the totals tables are used to display the total in and out utilization of the interface and the top 10 from the forensic table are subtracted out from the total and added back in color.

Important

In some cases, a report that doesn’t utilize the totals tables can understate the actual utilization of the interface.

The totals tables are not used when:

  • Reporting across all interfaces of a device
  • A report is run on multiple interfaces from different devices regardless of filters
  • A non interface filter has been applied to the report (e.g. IP address)
  • When a report uses more than one template
  • Looking at 1 minute intervals in a report. One minute intervals contain 100% of all data exported for the template as no rollups have occurred. As a result, no totals tables are created for 1 minute intervals.

The totals tables are only being used when:

  • Looking at 5 minute intervals and higher
  • The Flow Templates section of the report filter indicates Available Templates.
  • Looking at a single interface without any additional filters

Note

Only the top 1000 (default) conversations are saved in the rollups by default. A conversation is defined as 1 or more flows in a time frame that match based on values in a tuple (source and destination IP address, ingress and egress interfaces, commonPort, etc.). If the collection server has the available disk space, try increasing the Maximum Conversations under Admin Tab -> Settings -> Data History to 10,000 and see if it improves the accuracy. Don’t configure it right away for the maximum. Instead try carefully increasing the number of conversations saved over a few days. Some reports may render more slowly when the maximum conversations is increased, this is the result of the tables being larger.

SAF mode (Summary and Forensic)

SAF mode creates both summary and forensic tables, but only the summary tables are rolled up. summary tables contain a subset of forensic columns and are useful for fast and long term reporting. It can act as a search index before drilling into the forensic data.

In SAF mode, the collector saves 100% of all data in aggregated format to the 1 minute summary tables for each router. By default, with summary data, flows are aggregated based on a tuple that includes the common port. Fields that prevent aggreggation such as the source and destination transport ports are dropped. Kept fields include, but are not limited to:

  • intervalTime
  • commonport
  • ingressInterface
  • egressInterface
  • sourceIpAddress
  • destinationIpAddress
  • octetDeltaCount
  • octetDeltaCount_rev
  • packetDeltaCount
  • packetDeltaCount_rev
  • flowDirection
  • applicationId
  • protocolIdentifier

The summary aggregation logic used to create the above can be modified. Contact technical support for details on how to modify these settings.

This dramatically reduces table size and maintains accuracy. Every hour the collector creates a new 1 minute interval summary table per router. Every 5 min, 30 min, 2 hr, 12 hr (no 1 day or 1 week) it creates higher intervals using the smaller intervals. This process is called “rollups”.

When the rollups occur on summary data, two tables are created in exactly the same way as outlined for traditional (forensic) Mode. The 2nd table is the totals table.

In SAF mode:

  • Nearly all reports leverage the summary tables.
  • Only vendor specific reports and a few other reports that require elements such as source and destination ports go back to the Forensic data.
  • Since forensic tables in SAF mode are not rolled up, totals tables are not created for forensic tables.

A word about sFlow

When collecting sFlow, make sure the packet samples and the interface counters are both being exported to the collector. The collector will save the packet samples to the Raw tables and the Interface counters to Totals tables even at 1 minute intervals.

Warning

If the sflow exporting device (e.g. switch) is exporting multiple templates for different flows, utilization could be overstated if the flows contain the same or nearly the same information. The front end of Scrutinizer will render reports using data from all templates with matching information. Be careful when exporting multiple templates from the same device! If this is found to be the case, use the filters to select a single template.