Event and accounting logs

The log files saved in local storage can be encrypted using the AES-256-CTR cipher algorithm. The encryption key is used for all local log files in the system.

Use the following command to configure the log file encryption key and enable log file encryption.

configure log encryption-key

Sending events to a console destination means the message is sent to the system console The console device can be used as an event log destination.

A session destination is a temporary log destination which directs entries to the active Telnet or SSH session for the duration of the session. When the session is terminated, for example, when the user logs out, the ‟to session” configuration is removed. Event logs configured with a session destination are stored in the configuration file but the ‟to session” part is not stored. Event logs can direct log entries to the session destination.

A CLI log is a log that outputs log events to a CLI session. The events are sent to the CLI session for the duration of that CLI session (or until an unsubscribe-from command is issued).

Use the following command to subscribe to a CLI log from within a CLI session.

tools perform log subscribe-to log-id

A memory log is a circular buffer. When the log is full, the oldest entry in the log is replaced with the new entry. When a memory log is created, the specific number of entries it can hold can be specified, otherwise it assumes a default size. An event log can send entries to a memory log destination.

Log files can be used by both event logs and accounting logs and are stored on the compact flash devices in the file system.

A log file policy is identified using a numerical ID in classic interfaces and a string name in MD interfaces, but a log file policy is generally associated with a number of individual files in the file system. A log file policy is configured with rollover, expressed in minutes, which represents the period of time an individual log file is written to before a new file is created for the relevant log file policy. The rollover time is checked only when an update to the log is performed. Therefore, complying to this rule is subject to the incoming rate of the data being logged. For example, if the rate is very low, the actual rollover time may be longer than the configured value.

The retention time for a log file policy specifies the amount of time an individual log file is retained on the system based on the creation date and time of the file. The system continuously checks for log files with expired retention periods once every hour and deletes as many files as possible during a 10-second interval.

When a log file policy is created, only the compact flash device for the log files is specified. Log files are created in specific subdirectories with standardized names depending on the type of information stored in the log file.

Event log files are always created in the \log directory on the specified compact flash device. The naming convention for event log files is:

log eeff-timestamp

Accounting log files are created in the \act-collect directory on a compact flash device (specifically cf1 or cf2). The naming convention for accounting log files is nearly the same as for log files except the prefix act is used instead of the prefix log. The naming convention for accounting logs is:

act aaff-timestamp.xml.gz

Accounting logs are .xml files created in a compressed format and have a .gz extension.

The \act-collect directory is where active accounting logs are written. When an accounting log is rolled over, the active file is closed and archived in the \act directory before a new active accounting log file created in \act-collect.

When creating a new log file on a compact flash disk card, the system checks the amount of free disk space and that amount must be greater than or equal to the lesser of 5.2 MB or 10% of the compact flash disk capacity.

An event log can be configured to send events to SNMP trap receivers by specifying an SNMP trap group destination.

An SNMP trap group can have multiple trap targets. Each trap target can have different operational values.

A trap destination has the following properties:

• The IP address of the trap receiver.

• The UDP port used to send the SNMP trap.

• SNMP version (v1, v2c, or v3) used to format the SNMP notification.

• SNMP community name for SNMPv1 and SNMPv2c receivers.

• Security name and level for SNMPv3 trap receivers.

For SNMP traps that are sent out-of-band through the Management Ethernet port on the SF/CPM, the source IP address of the trap is the IP interface address defined on the Management Ethernet port. For SNMP traps that are sent in-band, the source IP address of the trap is the system IP address of the router.

Each trap target destination of a trap group receives the identical sequence of events as defined by the log ID and the associated sources and log filter applied. For the list of options that can be sent in SNMP notifications, please see the SR OS MIBs (and RFC 3416, section 4.2.6).

A NETCONF log is a log that outputs log events to a NETCONF session as notifications. A NETCONF client can subscribe to a NETCONF log using the configured netconf-stream stream-name for the log in a subscription request. See NETCONF notifications for more details.

In Event logging block diagram, the event sources are the main categories of events that feed the log manager.

• security

  The security event source is all events that affect attempts to breach system security such as failed login attempts, attempts to access MIB tables to which the user is not granted access or attempts to enter a branch of the CLI to which access has not been granted. Security events are generated by the SECURITY application and the authenticationFailure event in the SNMP application.

• change

  The change activity event source is all events that directly affect the configuration or operation of the node. Change events are generated by the USER application. The Change event stream also includes the tmnxConfigModify (#2006), tmnxConfigCreate (#2007), tmnxConfigDelete (#2008) and tmnxStateChange (#2009) change events from the SYSTEM application, as well as the various xxxConfigChange events from the MGMT_CORE application.

• debug

  The debug event source is the debugging configuration that has been enabled on the system. Debug events are generated when debug is enabled for various protocols under the debug branch of the CLI (for example, debug system ntp).

• main

  The main event source receives events from all other applications within the router.

Use the following command to show information about all applications.

show log applications

The following example shows partial output of the show log applications command. Examples of applications within the system include IP, MPLS, OSPF, CLI, services, and so on.

==================================
Log Event Application Names
==================================
Application Name
----------------------------------
...
BGP
CCAG
CFLOWD
CHASSIS
...
MPLS
MSDP
NTP
...
USER
VRRP
VRTR
==================================

Event control pre-processes the events generated by applications before the event is passed into the main event stream. Event control assigns a severity to application events and can either forward the event to the main event source or suppress the event. Suppressed events are counted in event control, but these events do not generate log entries as it never reaches the log manager.

Simple event throttling is another method of event control and is configured similarly to the generation and suppression options. See Simple logger event throttling.

Events are assigned a default severity level in the system, but the application event severities can be changed by the user.

Application events contain an event number and description that describes why the event is generated. The event number is unique within an application, but the number can be duplicated in other applications.

Use the following command to display log event information.

show log event-control

The following example, generated by querying event control for application generated events, displays a partial list of event numbers and names.

=======================================================================
Log Events
=======================================================================
Application
 ID#    Event Name                       P   g/s     Logged     Dropped
-----------------------------------------------------------------------

show
BGP:
   2001 bgpEstablished                   MI  gen          1           0
   2002 bgpBackwardTransition            WA  gen          7           0
   2003 tBgpMaxPrefix90                  WA  gen          0           0
...
CCAG:
CFLOWD:
   2001 cflowdCreated                    MI  gen          1           0
   2002 cflowdCreateFailure              MA  gen          0           0
   2003 cflowdDeleted                    MI  gen          0           0
...
CHASSIS:
   2001 cardFailure                      MA  gen          0           0
   2002 cardInserted                     MI  gen          4           0
   2003 cardRemoved                      MI  gen          0           0
...

,,,
DEBUG:
L  2001 traceEvent                       MI  gen          0           0
DOT1X:
FILTER:
   2001 filterPBRPacketsDropped          MI  gen          0           0
IGMP:
   2001 vRtrIgmpIfRxQueryVerMismatch     WA  gen          0           0
   2002 vRtrIgmpIfCModeRxQueryMismatch   WA  gen          0           0
IGMP_SNOOPING:
IP:
L  2001 clearRTMError                    MI  gen          0           0
L  2002 ipEtherBroadcast                 MI  gen          0           0
L  2003 ipDuplicateAddress               MI  gen          0           0
...
ISIS:
   2001 vRtrIsisDatabaseOverload         WA  gen          0           0

Events that are forwarded by event control are sent to the log manager. The log manager manages the event logs in the system and the relationships between the log sources, event logs and log destinations, and log filter policies.

An event log has the following properties:

• a unique log ID

  The log ID is a short, numeric identifier for the event log. A maximum of 30 logs can be configured at a time.

• one or more log sources

  The source stream or streams to be sent to log destinations can be specified. The source must be identified before the destination can be specified. The events can be from the main event stream, events in the security event stream, or events in the user activity stream.

• one event log destination

  A log can only have a single destination (for example, syslog or memory).

• an optional event filter policy

  An event filter policy defines whether to forward or drop an event or trap-based on match criteria.

The log manager uses event filter policies to allow fine control over which events are forwarded or dropped based on various criteria. Like other filter policies in the SR OS, filter policies have a default action. The default actions are either:

• Forward

• Drop

Filter policies also include a number of filter policy entries that are identified with an entry ID and define specific match criteria and a forward or drop action for the match criteria.

Each entry contains a combination of matching criteria that define the application, message, event number, router, severity, and subject conditions. The entry’s action determines how the packets should be treated if they have met the match criteria.

Entries are evaluated in order from the lowest to the highest entry ID. The first matching event is subject to the forward or drop action for that entry.

Valid operators are displayed in Valid filter policy operators:

A match criteria entry can include combinations of:

• Equal to or not equal to a specific system application.

• Equal to or not equal to an event message string or regular expression match.

• Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal to an event number within the application.

• Equal to, not equal to, less than, less than or equal to, greater than or greater than or equal to a severity level.

• Equal to or not equal to a router name string or regular expression match.

• Equal to or not equal to an event subject string or regular expression match.

Log entries that are forwarded to a destination are formatted in a way appropriate for the specific destination whether it be recorded to a file or sent as an SNMP trap, but log event entries have common elements or properties. All application generated events have the following properties:

• A time stamp in UTC or local time.

• The generating application.

• A unique event ID within the application.

• A router name identifying the router instance that generated the event.

• A subject identifying the affected object.

• A short text description.

The general format for an event in an event log with either a memory, console or file destination is as follows.

nnnn <time> TZONE <severity>: <application> #<event-id> <vrtr-name> <subject> 
<message>

The following is an event log example:

252 2013/05/07 16:21:00.761 UTC WARNING: SNMP #2005 Base my-interface-abc
"Interface my-interface-abc is operational"

The specific elements that compose the general format are described in Log entry field descriptions.

configure log log-id x time-format

Simple event throttling provides a mechanism to protect event receivers from being overloaded when a scenario causes many events to be generated in a very short period of time. A throttling rate, # events/# seconds, can be configured. Specific event types can be configured to be throttled. When the throttling event limit is exceeded in a throttling interval, any further events of that type cause the dropped events counter to be incremented.

Use the commands in the following context to display dropped event counts.

show log event-control

Events are dropped before being sent to one of the logger event collector tasks. There is no record of the details of the dropped events and therefore no way to retrieve event history data lost by this throttling method.

A particular event type can be generated by multiple managed objects within the system. At the point this throttling method is applied the logger application has no information about the managed object that generated the event and cannot distinguish between events generated by object ‟A” from events generated by object ‟B”. If the events have the same event-id, they are throttled regardless of the managed object that generated them. It also does not know which events may eventually be logged to destination log-id <n> from events that are logged to destination log-id <m>.

Throttle rate applies commonly to all event types. It is not configurable for a specific event-type.

A timer task checks for events dropped by throttling when the throttle interval expires. If any events have been dropped, a TIMETRA-SYSTEM-MIB::tmnxTrapDropped notification is sent.

Log 99 is a pre-configured memory-based log which logs events from the main event source (not security, debug, and so on). Log 99 exists by default.

The following example displays the log 99 configuration.

[ex:/configure log]
A:admin@node-2# info
    log-id "99" {
        admin-state enable
        description "Default system log"
        source {
            main true
        }
        destination {
            memory {
                max-entries 500
            }
        }
    }
    snmp-trap-group "7" {
    } 

A:node-2>config>log# info detail
#------------------------------------------
echo "Log Configuration "
#------------------------------------------
...
        snmp-trap-group 7
        exit
...
        log-id 99
            description "Default system log"
            no filter
            from main
            to memory 500
            no shutdown
        exit
----------------------------------------------

The Event Handling System (EHS) is a framework that allows operator-defined behavior to be configured on the router. EHS adds user-controlled programmatic exception handling by allowing the execution of either a CLI script or a Python 3 application when a log event (the ‟trigger”) is detected. Various fields in the log event provide regexp style expression matching, which allows flexibility for the trigger definition.

EHS handler objects are used to tie together the following:

• trigger events (typically log events that match a configurable criteria)

• a set of actions to perform (enabled using CLI scripts and Python applications)

EHS, along with CRON, may execute SR OS CLI scripts or Python 3 applications to perform operator-defined functions as a result of receiving a trigger event. The Python programming language provides an extensive framework for automation activities for triggered or scheduled events, including model-driven transactional configuration and state manipulation. See the Python chapter for more information.

The use of Python applications from EHS is supported only in model-driven configuration mode.

EHS object relationships (classic CLI) shows the relationships among the different configurable objects used by EHS (and CRON).

Log events can initiate from within a VPRN, instead of from the base router instance or the CPM management router instance. For example, a syslog collector may be reachable through a VPRN interface. Use the following command to configure event logs for a VPRN service.

configure service vprn log

By default, the event-source streams for VPRN event logs contain only events that are associated with that specific VPRN. It is also possible to configure a system-wide set of log events. This can be useful, for example, when a VPRN is being used as a management VPRN. Use the following command to send a VPRN event log for the entire system-wide set of log events (VPRN and non-VPRN).

configure log services-all-events

This section describes the syslog-specific aspects of Python processing. For an introduction to Python, see the 7450 ESS, 7750 SR, and VSR Triple Play Service Delivery Architecture Guide, "Python script support for ESM".

When an event is dispatched to the log manager in SR OS, the log manager asynchronously passes the event context data and variables (varbinds in Python 2 and event parameters in Python 3) to the Python engine; that is, the logger task is not waiting for feedback from Python.

Varbinds or event parameters are variable bindings that represent the variable number of values that are included in the event. Each varbind in Python 2 consists of a triplet (OID, type, value).

Along with other system-level variables, the Python engine constructs a syslog message and sends it to the syslog destination when the Python engine successfully concludes. During this process, the operator can modify the format of the syslog message or leave it intact, as if it was generated by the syslog process within the log manager.

The tasks of the Python engine in a syslog context are as follows:

• assemble custom syslog messages (including PRI, HEADER and MSG fields) based on the received event context data, varbinds and event parameters specific to the event, system-level data, and the configuration parameters (syslog server IP address, syslog facility, log-prefix, and the destination UDP port)

• reformat timestamps in a syslog message

• modify attributes in the message and reformats the message

• send the original or modified message to the syslog server

• drop the message

Python retrieves event-related variables from the log manager, as opposed to retrieving pre-assembled syslog messages. This eliminates the need for string parsing of the syslog message to manipulate it constituent parts increasing the speed of Python processing.

To further improve processing performance, Nokia recommends performing string manipulation via the Python native string method, when possible.

A Python task assembles syslog messages based on the context information received from the logger and sends them to the syslog server independent of the logger. If the Python task is congested because of a high volume of received data, the backpressure should be sent to the ISA so that the ISA stops allocating NAT resources. This behavior matches the current behavior in which NAT resources allocation is blocked if that logger is congested.

An accounting policy must define a record name and collection interval. Only one record name can be configured per accounting policy. Also, a record name can only be used in one accounting policy.

The record name, sub-record types, and default collection period for service and network accounting policies are shown in Accounting record name and collection periods. Policer stats field descriptions (fields per policer stat-mode are provided in the stat-mode command descriptions in the 7450 ESS, 7750 SR, 7950 XRS, and VSR Quality of Service Guide), Queue group record types, and Queue group record type fields provide field descriptions.

When creating accounting policies, one service accounting policy and one network accounting policy can be defined as default. If statistics collection is enabled on a SAP or network port and no accounting policy is applied, then the respective default policy is used. If no default policy is defined, then no statistics are collected unless a specifically defined accounting policy is applied.

Each accounting record name is composed of one or more sub-records which is in turn composed of multiple fields.

See the Application Assurance Statistics Fields Generated per Record table in the 7450 ESS, 7750 SR, and VSR Multiservice ISA and ESA Guide for fields names for Application Assurance records.

The availability of the records listed in Accounting record name details depends on the specific platform functionality and user configuration.

Policer stats field descriptions, Queue group record types, and Queue group record type fields provide field descriptions.

When a policy has been created and applied to a service or network port, the accounting file is stored on the compact flash in a compressed XML file format. The router creates two directories on the compact flash to store the files. The following output displays a directory named act-collect that holds accounting files that are open and actively collecting statistics. The directory named act stores the files that have been closed and are awaiting retrieval.

ALA-1>file cf1:\# dir act*
12/19/2006 06:08a      <DIR>          act-collect
12/19/2006 06:08a      <DIR>          act

ALA-1>file cf1:\act-collect\ # dir
Directory of cf1:\act-collect#

12/23/2006 01:46a      <DIR>          .
12/23/2006 12:47a      <DIR>          ..
12/23/2006 01:46a                 112 act1111-20031223-014658.xml.gz
12/23/2006 01:38a                 197 act1212-20031223-013800.xml.gz

Accounting files always have the prefix "act" followed by the accounting policy ID, log ID, and timestamp. The accounting log file naming and log file policy properties like rollover and retention are described in more detail in Log files.

The router has ample resources to support large scale accounting policy deployments. When preparing for an accounting policy deployment, verify that data collection, file rollover, and file retention intervals are properly tuned for the amount of statistics to be collected.

If the accounting policy collection interval is too brief there may be insufficient time to store the data from all the services within the specified interval. If that is the case, some records may be lost or incomplete. Interval time, record types, and number of services using an accounting policy are all factors that should be considered when implementing accounting policies.

The rollover and retention intervals on the log files and the frequency of file retrieval must also be considered when designing accounting policy deployments. The amount of data stored depends on the type of record collected, the number of services that are collecting statistics, and the collection interval that is used. For example, with a 1Gb CF and using the default collection interval, the system is expected to hold 48 hours’ worth of billing information.

SR OS on the 7750 SR platform has support for volume accounting and time-based accounting concepts, and provides an extra level of intelligence at the network element level to provide service models such as ‟prepaid access” in a scalable manner. This means that the network element gathers and stores per-subscriber accounting information and compares it with ‟pre-defined” quotas. When a quota is exceeded, the pre-defined action (such as re-direction to a web portal or disconnect) is applied.

Custom records can be used to decrease accounting messaging overhead as follows:

• User configurable records

• Changed statistics only

• Configurable accounting records

• Significant change only reporting

This feature allows the operator to report on protocol/application/app-group volume usage per forwarding class by adding a bitmap information representing the observed FC in the XML accounting files. In case the accounted object is deleted or changed, the latest information is written in the XML file with a ‟final” tag indication in the record header.

Configure logging to save information in a log file or direct the messages to other devices. Logging does the following:

• Provides you with logging information for monitoring and troubleshooting.

• Allows the selection of the types of logging information to be recorded.

• Allows the assignment of a severity to the log messages.

• Allows the selection of source and target of logging information.

Logs can be configured in the following contexts:

• log file

  Log files can contain log event message streams or accounting/billing information. Log file policies are used to direct events, alarms, traps, and debug information to a file on local storage devices (for example, cf2:).

• SNMP trap groups

  SNMP trap groups contain an IP address and community names which identify targets to send traps following specified events.

• syslog

  Information can be sent to a syslog host that is capable of receiving selected syslog messages from a network element.

• event control

  Configures a particular event or all events associated with an application to be generated or suppressed.

• event filters

  An event filter defines whether to forward or drop an event or trap based on match criteria.

• accounting policies

  An accounting policy defines the accounting records that will be created. Accounting policies can be applied to one or more service access points (SAPs).

• event logs

  An event log defines the types of events to be delivered to its associated destination.

• event throttling rate

  Defines the rate of throttling events.

The most basic log configuration must have the following:

• log ID or accounting policy ID

• log source

• log destination

The following example displays a log configuration for the 7750 SR.

[ex:/configure log]
A:admin@node-2# info
    log-events {
        bgp event sendNotification {
            severity critical
            throttle false
        }
    }
    file "1" {
        description "This is a test file-id."
        compact-flash-location {
            primary cf1
        }
    }
    file "2" {
        description "This is a test log."
        compact-flash-location {
            primary cf1
        }
    }
    log-id "2" {
        source {
            main true
        }
        destination {
            file "2"
        }
    }
    snmp-trap-group "7" {
        trap-target "testTarget" {
            address 11.22.33.44
            version snmpv2c
            notify-community "public"
        }
    }

A:node-2>config>log# info
#------------------------------------------
echo "Log Configuration "
#------------------------------------------
        event-control "bgp" 2005 generate critical
        file-id 1
            description "This is a test file-id."
            location cf1:
        exit
        file-id 2
            description "This is a test log."
            location cf1:
        exit
        snmp-trap-group 7
            trap-target "testTarget" address 11.22.33.44 "snmpv2c" notify-community "public"
        exit
        log-id 2
            from main
            to file 2
        exit
---------------------------------------------- 

The following sections describe basic system tasks that must be performed.