cflowd statistics collection

Scaling guidelines for cflowd statistics collection

The table below shows the scaling limits for an NSP Flow Collector in its ability to process cflowd flow records from the network and produce IPDR formatted files. The guidelines are divided into the NSP Flow Collector collecting in a residential/mobile deployment and the NSP Flow Collector collecting in a business deployment.

For residential and mobile deployments, the only statistics types that should be in use are Volume, Comprehensive, Unknown and corresponding Special Study types. TCP and RTP are not supported in mobile, and although the statistics types are available for Residential deployments, the use case is not, and there are no reports for this data. Additionally, even for residential, the only reports available are for Comprehensive. Comprehensive statistics have a fixed 60min aggregation interval. Due to the amount of data generated in a mobile deployment, Volume statistics require an aggregation interval of 60 minutes. As an alternative, Volume Special Study statistics on specific subscribers can be used. The only key factor of difference is whether or not additional counters are enabled for Comprehensive statistics.

Table 5-26: cflowd statistics scaling limits for residential and mobile deployments

NSP Flow Collector processing rate in flows per second

Counter selection 1

Maximum number of unique objects in memory 2

Packet loss per hour 3

100 000 FPS

Default two counters

100M Objects

<= 2%

All counters

60M Objects

<= 1%

Notes:
  1. Default: two counters. Volume: total bytes/total packets. Comp-volume: total bytes StoC/CtoS sum unknown. Only one counter exists. Vol SS: should be minuscule. All counters: Comp-volume has a total of ten counters that can be enabled.

  2. Number of aggregated output requests that are sent to the server every 60 minutes. Assumes transfer has sufficient bandwidth to complete in a timely manner.

  3. Packet loss may increase if communication between the NSP Flow Collector and target file server is interrupted.

For business deployments, in addition to the statistics types with a small number of records; Comprehensive, Volume, Unknown, and Volume Special Study, there are also statistics types with a larger number of records; TCP Performance, and RTP (Voice/Audio/Video). The main distinction is whether or not the TCP/RTP statistics types use the default enabled counters, or if all counters have been enabled. Enabling all of the TCP/RTP counters increases the amount of memory used by the NSP Flow Collector. Aside from the incoming FPS (Flows Per Second) that the NSP Flow Collector can process, the other main factor putting pressure on the NSP Flow Collector is the memory used by the number of unique objects/records (or unique routes, that is the # of output records the NSP Flow Collector produces in the IPDR files) in NSP Flow Collector memory at any one time. And finally the interval size – the smaller the aggregation interval, the greater percentage of the next interval time will overlap with the transfer time of the previous interval – during this time the NSP Flow Collector must store objects in memory from two different intervals. Comprehensive statistics types are fixed at 60 minute intervals.

A unique object/route for TCP/Volume records in the business context is:

SAP, App/AppGroup, Interval ID, Src Group ID, Source Interface ID, Dest Group ID, Dest Interface ID

A Volume record will also have a direction field. Volume records coming from the router to the NSP Flow Collector will result in two output records in the IPDR files (one for each direction). For TCP, two incoming records from the NSP Flow Collector (one for each direction) will be combined by the NSP Flow Collector into a single output TCP record in the IPDR files.

A unique object/route for COMPREHENSIVE record in the business context is:

SAP, App/AppGroup, Interval ID, Src Group ID, Source Interface ID, Dest Group ID, Dest Interface ID

and either a hostname field, or three device identification fields.

A unique object/route for RTP is defined as:

Every single flow into the NSP Flow Collector is a unique route and an equal number of flow records are produced in the IPDR file. The expected number of RTP records sent from 7750 SR Routers is expected to be a small percentage of the total flows (for example, <5% total flows TCP/VOL/RTP)

Table 5-27: cflowd statistics scaling limits for business deployments

NSP Flow Collector processing rate in flows per second

Statistic types used and counters used 1

Maximum number of unique objects in memory 2

Packet loss per hour 3

100 000 FPS

Comprehensive/Volume/ Unknown/Vol S.S Only All Counters

60M objects

<= 1%

TCP/TCP S.S Only: Default Counter

25M objects

<= 1%

TCP/TCP S.S Only: All Counters

15M objects

<= 1%

RTP Only: Default Counters

10M objects

<= 1%

RTP Only: All Counters

3M objects

<= 1%

Combined Comprehensive/Volume/ Unknown/TCP/RTP (including Special Study)

20M Comp/Volume/ Unknown + 5M TCP (All Cnt) + 0.5 RTP (All Cnt)

<= 1%

Notes:
  1. Comprehensive/Volume/ Unknown/Volume SS: All Counters RTP/TCP/TCP S.S Counter Selection Default Counters: Leaving default enabled counters on All Counters: Enabling all available counters for given stat type. There are 40-60 total counters available for TCP and RTP types.

  2. Number of aggregated output requisitions that are sent to the server every 60 seconds. Assumes transfer has sufficient bandwidth to complete in a timely manner.

  3. Packet loss may increase if communication between the NSP Flow Collector and target file server is interrupted

AA Comprehensive Raw Aggregation Limits

Flow Collector performance is impacted by large cflowd message sizes. The larger the cflowd message, the lower the Flow Collector output performance.

Table 5-28: Flow Collector scale limits by cflowd message size

cflowd message size

Maximum flows per second

less than 548 bytes

100,000

less than 864 bytes

70,000

less than 1000 bytes

50,000

Notes:
  1. The Flow Collector can handle increased flow rates with fine-tuning, MTU changes and additional resources. Please contact Nokia Support for details.