cflowd statistics collection

Scaling guidelines for cflowd statistics collection

The table below shows the scaling limits for an NSP Flow Collector in its ability to process cflowd flow records from the network and produce IPDR formatted files. The guidelines are divided into the NSP Flow Collector collecting in a residential/mobile deployment and the NSP Flow Collector collecting in a business deployment.

For residential and mobile deployments, the only statistics types that should be in use are Volume, Comprehensive, Unknown and corresponding Special Study types. TCP and RTP are not supported in mobile, and although the statistics types are available for Residential deployments, the use case is not, and there are no reports for this data. Additionally, even for residential, the only reports available are for Comprehensive. Comprehensive statistics have a fixed 60min aggregation interval. Due to the amount of data generated in a mobile deployment, Volume statistics require an aggregation interval of 60 minutes. As an alternative, Volume Special Study statistics on specific subscribers can be used. The only key factor of difference is whether or not additional counters are enabled for Comprehensive statistics.

Table 5-28: cflowd statistics scaling limits for residential and mobile deployments

NSP Flow Collector processing rate in flows per second	Counter selection¹	Maximum number of unique objects in memory²	Packet loss per hour ³
100 000 FPS	Default two counters	100M Objects	<= 2%
100 000 FPS	All counters	60M Objects	<= 1%

Notes:

Default: two counters. Volume: total bytes/total packets. Comp-volume: total bytes StoC/CtoS sum unknown. Only one counter exists. Vol SS: should be minuscule. All counters: Comp-volume has a total of ten counters that can be enabled.
Number of aggregated output requests that are sent to the server every 60 minutes. Assumes transfer has sufficient bandwidth to complete in a timely manner.
Packet loss may increase if communication between the NSP Flow Collector and target file server is interrupted.

For business deployments, in addition to the statistics types with a small number of records; Comprehensive, Volume, Unknown, and Volume Special Study, there are also statistics types with a larger number of records; TCP Performance, and RTP (Voice/Audio/Video). The main distinction is whether or not the TCP/RTP statistics types use the default enabled counters, or if all counters have been enabled. Enabling all of the TCP/RTP counters increases the amount of memory used by the NSP Flow Collector. Aside from the incoming FPS (Flows Per Second) that the NSP Flow Collector can process, the other main factor putting pressure on the NSP Flow Collector is the memory used by the number of unique objects/records (or unique routes, that is the # of output records the NSP Flow Collector produces in the IPDR files) in NSP Flow Collector memory at any one time. And finally the interval size – the smaller the aggregation interval, the greater percentage of the next interval time will overlap with the transfer time of the previous interval – during this time the NSP Flow Collector must store objects in memory from two different intervals. Comprehensive statistics types are fixed at 60 minute intervals.

A unique object/route for TCP/Volume records in the business context is:

SAP, App/AppGroup, Interval ID, Src Group ID, Source Interface ID, Dest Group ID, Dest Interface ID

A Volume record will also have a direction field. Volume records coming from the router to the NSP Flow Collector will result in two output records in the IPDR files (one for each direction). For TCP, two incoming records from the NSP Flow Collector (one for each direction) will be combined by the NSP Flow Collector into a single output TCP record in the IPDR files.

A unique object/route for COMPREHENSIVE record in the business context is:

SAP, App/AppGroup, Interval ID, Src Group ID, Source Interface ID, Dest Group ID, Dest Interface ID

and either a hostname field, or three device identification fields.

A unique object/route for RTP is defined as:

Every single flow into the NSP Flow Collector is a unique route and an equal number of flow records are produced in the IPDR file. The expected number of RTP records sent from 7750 SR Routers is expected to be a small percentage of the total flows (for example, <5% total flows TCP/VOL/RTP)

Table 5-29: cflowd statistics scaling limits for business deployments

NSP Flow Collector processing rate in flows per second	Statistic types used and counters used¹	Maximum number of unique objects in memory²	Packet loss per hour ³
100 000 FPS	Comprehensive/Volume/ Unknown/Vol S.S Only All Counters	60M objects	<= 1%
	TCP/TCP S.S Only: Default Counter	25M objects	<= 1%
	TCP/TCP S.S Only: All Counters	15M objects	<= 1%
	RTP Only: Default Counters	10M objects	<= 1%
	RTP Only: All Counters	3M objects	<= 1%
	Combined Comprehensive/Volume/ Unknown/TCP/RTP (including Special Study)	20M Comp/Volume/ Unknown + 5M TCP (All Cnt) + 0.5 RTP (All Cnt)	<= 1%

Notes:

Comprehensive/Volume/ Unknown/Volume SS: All Counters RTP/TCP/TCP S.S Counter Selection Default Counters: Leaving default enabled counters on All Counters: Enabling all available counters for given stat type. There are 40-60 total counters available for TCP and RTP types.
Number of aggregated output requisitions that are sent to the server every 60 seconds. Assumes transfer has sufficient bandwidth to complete in a timely manner.
Packet loss may increase if communication between the NSP Flow Collector and target file server is interrupted

AA Comprehensive Raw Aggregation Limits

Flow Collector performance is impacted by large cflowd message sizes. The larger the cflowd message, the lower the Flow Collector output performance.

Table 5-30: Flow Collector scale limits by cflowd message size

cflowd message size	Maximum flows per second
less than 548 bytes	100,000
less than 864 bytes	70,000
less than 1000 bytes	50,000

Notes:

The Flow Collector can handle increased flow rates with fine-tuning, MTU changes and additional resources. Please contact Nokia Support for details.