Application Assurance — Local URL List Filtering

This chapter provides information about the Application Assurance local URL list filtering.

Topics in this chapter include:

Applicability

This chapter is applicable to all SR OS systems supporting Application Assurance and was initially based on SR OS Release 13.0.R2. The chapter has been updated with CLI and functionality changes introduced in SR OS Release 19.10.R1, but the CLI in the sections URL-list update and Cron job is based on SR OS Release 16.0.

There are no specific prerequisites for this feature.

Overview

The local URL-list filtering capability provided by Application Assurance offers the following:

configuration of a list of URLs which are globally allowed (allow-list)
configuration of a list of restricted URLs subscribers should be prevented from accessing (deny-list)

The lists are stored locally in the SR OS system compact flash (CF).

This chapter provides a guide for configuring deny-lists.

Deny-lists assist service providers to comply with regulatory requirements for network-wide URL filtering policies, such as:

Court-ordered URL takedown
Child protection
Government-mandated URL takedown list

The SR OS uses the Application Assurance capabilities to extract the URL from a subscriber HTTP/HTTPS request and compare it to the list of URLs contained in the local file. If a match occurs, the subscriber request is redirected to a preconfigured web server landing page, typically describing why the access to this resource was denied.

The system supports both unencrypted and OpenSSL Triple Data Encryption Standard (3DES) encrypted file formats to protect the content of the list.

HTTP/HTTPS filtering

Each HTTP request within a TCP flow is analyzed and filtered. For HTTPS traffic, the system extracts the domain name information contained in the Transport Layer Security (TLS) Server Name Indication (SNI).

Setup details

The setup consists of the following elements, as shown in Local URL-list filtering setup:

SR-series + ISA-AA
Web server (redirect landing page)
FTP server (source for the URL-list file)
Subscriber (desktop/laptop/tablet/smart phone)
Internet access
Optional: AAA for subscriber authentication and policy modification

Figure 1. Local URL-list filtering setup

This chapter is written in the context of a residential or WiFi deployment. However, local URL-list filtering is also applicable to business VPN services.

Configuration

To configure the system for local URL-list filtering, the operator needs to:

Create a URL-list policy referencing a valid URL-list file located on the system compact flash
Create a URL-filter policy for local filtering by referencing the URL-list policy previously created
Create an App-QoS-Policy (AQP) to apply this url-filter policy
Optionally configure a cron job to automatically fetch a new list file and upgrade the URL list.

URL-list policy and URL-filter policy

In the following example, two dedicated URL-list and URL-filter policies show URL filtering based on a plain text file and an encrypted file:

configure
    application-assurance group 1 
        url-list "denylist1-encrypted" create
            description "Demo URL Filtering List - Encrypted File"
            decrypt-key "ON3HU2GFPHmpOHwWbSGw/zdM4iuxzySpqS7pw/u3qIcuG4mABmrhc." 
                                                                           hash2
            file "cf3:\aa-url-list\url-list1.encrypted"
            no shutdown
        exit
        url-filter "local-filter-list1-encrypted" create
            local-filtering
                deny-list "denylist1-encrypted"
                default-action allow
                http-redirect "redirect-denylist"
            exit
            no shutdown
        exit

configure
    application-assurance group 1 
        url-list "denylist1-plaintext" create
            description "Demo URL Filtering List - Plaintext File"
            file "cf3:\aa-url-list\url-list1-plaintext.txt"
            no shutdown
        exit
        url-filter "local-filter-list1-txt" create
            local-filtering
                deny-list "denylist1-plaintext"
                default-action allow
                http-redirect "redirect-denylist"
            exit
            no shutdown
        exit

In the preceding example, both URL-filter policies are defined using default-action allow. The default action is used in case the file could not be loaded by the system, either at boot time or the first time the URL-list file was configured in the system. Possible causes are, for example:

File corrupted, compact flash corrupted
Incorrect file encryption format or password
Wrong URL format in the file
Too many URLs in the file

Operators should always use default-action allow when configuring the URL-filter policy associated with a URL-list file because the file or the CF may be corrupted, in which case the system logs an error and a trap is raised.

Note that if a valid URL-list file was previously in use, and an invalid file is uploaded and the URL-list policy upgraded using this file, then the system will continue using the previous list.

HTTP-redirect policy

Both URL-filter policies defined in the preceding example refer to the following http-redirect policy; subscribers accessing a URL from the URL-list file are redirected to the following landing page:

configure
    application-assurance group 1 
        http-redirect "redirect-denylist" create
            description "Redirect for Local List URL Filtering"
            template 5
            tcp-client-reset
            redirect-url "http://172.16.70.100/Redirect/redirect-denylist.html?
                                                             Request edURL=$URL"
            no shutdown
        exit

URL-list file

File format

A URL-list file may contain either hostnames or URLs.

To create a URL-list containing hostnames, set expression-match in the url-list configuration:

config>app-assure>group# url-list url-list-name [create]
                description <description-string>
                no description
                decrypt-key key | hash-key | hash2-key [hash | hash2]
                no decrypt-key
                file file-url
                no file
                size url-list-size
                expression-match
                [no] shutdown

A URL-list with hostnames only (using expression-match) may contain the following wildcards:

Head anchors character set [^ *]
Tail anchors character set [$ *]
Mid expression character set [\d \l \. \**]
Hex escaped characters [\x00 - \xFF]

Note that when expression-match is enabled, the list should contain hostnames only (with optional wildcards).

When configuring a URL-list with expression-match disabled (default), the system supports the following format for the URLs contained in the URL-list file:

URLs without the HTTP keyword. For example:
```
        www.domain.com/path
```
URLs with the HTTP keyword. For example:
```
       http://www.domain.com/path
```

In all cases, the following is supported:

Comment lines starting with the number sign character (#). For example:
```
       # This is a comment line
```
Printable ASCII characters. URLs using non-printable ASCII characters are percent-encoded by the web browser automatically and, therefore, need to be percent-encoded in the URL-list file.

File encryption

OpenSSL triple DES -nosalt is the supported encryption format. Files can be encrypted offline on a server using the following command:

openssl des3 -nosalt -in <input.txt> -out <output.enc>

List upgrade

The URL-list file can be upgraded using the admin command:

A:BNG# admin application-assurance group 1 url-list "denylist1-plaintext" upgrade

The upgrade result is logged in the system log-id 99:

A:BNG# show log log-id 99
===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents  [size=500   next event=72  (not wrapped)]
71 2015/07/07 13:09:25.01 EST MINOR: APPLICATION_ASSURANCE #4446 Base url-list success
"URL list "denylist1-plaintext" in ISA-AA group 1 has been updated. There are 3 entries in the URL list."

App-profiles and app-service-options

Application Assurance policies can be selectively applied to specific AA subscribers by modifying the app-profile assigned to the subscriber or using Application Service Option (ASO) override. See 7450 ESS, 7750 SR, and VSR Multiservice ISA and ESA Guide for more information about modifying the app-profile or ASO assigned to AA subscribers (RADIUS, Gx, Override).

In this example, the following ASO configuration is used:

configure
    application-assurance group 1:1 policy 
        app-service-options
            characteristic "local-list-filtering" create
                value "no"
                value "yes-encrypted"
                value "yes-plaintext"
                default-value "no"
            exit
        exit
        app-profile "1-1/Default" create
            divert
        exit

The ASO characteristic local-list-filtering value of yes-encrypted and yes-plaintext enable the AQP entry 210 and 220 in the example:

configure
    application-assurance group 1:1 policy app-qos-policy
        entry 210 create
            match
                characteristic "local-list-filtering" eq "yes-encrypted"
            exit
            action
                url-filter "local-filter-list1-encrypted"
            exit
            no shutdown
        exit
        entry 220 create
            match
                characteristic "local-list-filtering" eq "yes-plaintext"
            exit
            action
                url-filter "local-filter-list1-txt"
            exit
            no shutdown
        exit

If the url-filter policy needs to be applied to 100% of the subscribers in the network, it is also possible to remove the ASO match criteria.

URL-list update

The system supports a flexible mechanism to upgrade the URL list automatically, using either cron or the NSP, to comply with the regulatory requirements for list upgrade frequency.

To configure a crontab job to periodically upgrade the URL list, the operator needs to:

Generate the file to be periodically executed and store it to compact flash. As an example, create a file with filename "fetch.txt" with the following content:
```
file
copy ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt . force
exit
admin application-assurance group 1 url-list "test" upgrade
```
The preceding commands will fetch a file from an ftp server and store it in compact flash. Assuming the operator has configured a local url-list "test" containing the file "list.txt", the url list will be upgraded.

Configure the script policy:

>config>system>script-control# info
----------------------------------------------
            script "bring"
                location "cf3:/fetch.txt"
                no shutdown
            exit
            script-policy "test"
                results "cf3:/results.txt"
                script "bring"
                no shutdown
            exit

Configure the crontab job:

>config>system>cron# info
----------------------------------------------
        schedule "bring_list"
            interval 60
            script-policy "test"
            no shutdown
        exit
----------------------------------------------

The preceding configuration will execute the commands stored in the file "fetch.txt" every 60 seconds. A value of 60 seconds was chosen for the test. In a real deployment, a list would be typically updated every 12 – 24 hours. A log file (results "results.txt") will also be created.

The end result will be that the system will automatically fetch from an ftp server a new list file, store it to compact flash and upgrade the URL list.

Show commands

url-list

The status of the URL list can be shown in the CLI. The url-list show command provides basic admin and operational status, as well as the number of URLs in the list. The command also provides reasons for any possible issue related to loading the list, as well as the last successfully deployed file and the last upgrade attempt. Therefore, the operator can determine whether the latest version of the file is currently in use or if an error occurred when trying to upgrade the list.

Show command output:

Label                    Description
Admin                    Status [Up | Down] - Administrative status of the url-list
Oper Status              [Up | Down] - Operational status of the url-list
Oper Flags               [admin-down | file-does-not-exist |invalid-file-format |
                         too-many-urls| switch-over-error]
File Deployed to ISA      [Yes | No] - This flag describes if the file located in 
                          the compact flash is the one deployed in the ISA, in the 
                          event the file is overwritten and before the admin upgrade
                          command is used this flag will display ‟No”.
Upgrade Statistics
Last Success              Last time the list was successfully upgraded
File Name                 File name for the last successful upgrade
URL Entries               Number of URLs loaded at the last success
Blank/Comment Lines       Number of blank or commented out lines
Last Attempt              Last time the operator tried to upgrade the list
Result                    Success | Failure. Result of the last upgrade
File Name                 File name for the last upgrade attempt


*A:Dut-C# show application-assurance group 1 url-list "Deny List1"
======================================================================
Application Assurance Group 1 url-list "Deny List1"
======================================================================
Description : (Not Specified)
Size : standard
Host Expressions : disabled
Admin Status : Up
Oper Status : Up
Oper Flags : <none>
File deployed to ISAs : Yes
-------------------------------------------------------------------------------
Upgrade Statistics
-------------------------------------------------------------------------------
Last Success : 11/02/2020 15:07:40
Deployed
File Name : cf1:/host.txt
URL Entries : 1 ( 0.01% full)
URL Characters : 6 (~0.00% full)
URL Host Expr Entries: 1 ( 0.01% full)
Blank/Comment Lines : 1
Last Attempt : 11/02/2020 15:07:40
Result : Success
File Name : cf1:/host.txt
======================================================================

url-filter

The url-filter show command provides its operational and admin status, as well as actions taken, such as the number of redirects. With URL list filtering, using a default-action set to allow, the only counters increasing are allow, redirect, and default.

*A:Dut-C>config>app-assure>group>url-filter# show application-assurance group 1 url-filter "Url Filter1"
===============================================================================
Application Assurance Group 1 URL Filter "Url Filter1"
===============================================================================
Description            : (Not Specified)
Admin Status           : Up
Oper Status            : Up
Oper Flags             : <none>
HTTP Request Filtering : all
AQP Referenced         : No
-------------------------------------------------------------------------------
URL Stats Summary
-------------------------------------------------------------------------------
Total Requests  : 0                     Default Action  : 0
Requests Allowed: 0                     Reqs Block/Redir: 0
-------------------------------------------------------------------------------
Local Filter
-------------------------------------------------------------------------------
deny-list              : Deny List1
  Admin Status         : Up
  Oper Status          : Up
  Oper Flags           : <none>
  Number of URLs       : 1
  Default Action       : block-all
  HTTP Redirect        : (Not Specified)
  URL-List Lookups     : 0
    Match              : 0
    Miss               : 0
    Default Action     : 0
===============================================================================

http-redirect

The http-redirect show command provides more information about how the traffic was blocked; for example, it differentiates TCP client reset used for HTTPS from regular redirect used for HTTP traffic.

A:BNG# show application-assurance group 1 http-redirect "redirect-denylist"
===============================================================================
Application Assurance Group 1 HTTP Redirect redirect-denylist
===============================================================================
Description        : Redirect for Local List URL Filtering
Template           : 5
                   : Redirect supporting macro substitution using HTTP 302
Redirect URL       : http://172.16.70.100/Redirect/redirect-denylist.
                     html?RequestedURL=$URL
Admin Status       : Up
AQP Ref            : No
-------------------------------------------------------------------------------
Summary Statistics
-------------------------------------------------------------------------------
Grp:Part             Redirects           Client Resets               Redirects
                          Sent                    Sent                Not Sent
-------------------------------------------------------------------------------
1:1                          2                      1                        0
-------------------------------------------------------------------------------
Total                        2                      1                        0
-------------------------------------------------------------------------------
===============================================================================

Cron job

This chapter provides details on how to verify that the cron job executes successfully. The operator can see the location and execution interval of the file which is executed periodically, the time it was last executed and any possible errors which may have occurred during execution using the following command:

 # show system cron schedule "bring_list"
===============================================================================
CRON Schedule Information
===============================================================================
Schedule                     : bring_list
Schedule owner               : TiMOS CLI
Description                  : none
Administrative status        : enabled
Operational status           : enabled
Script Policy                : test
Script Policy Owner          : TiMOS CLI
Script                       : bring
Script Owner                 : TiMOS CLI
Script source location       : cf3:/fetch.txt
Script results location      : cf3:/results.txt
Schedule type                : periodic
Interval                     : 0d 00:01:00 (60 seconds)
Repeat count                 : infinite
Next scheduled run           : 0d 00:00:51
End time                     : none
Weekday                      : none
Month                        : none
Day of month                 : none
Hour                         : none
Minute                       : none
Number of schedule runs      : 12
Last schedule run            : 2019/09/13 11:28:25 EEST
Number of schedule failures  : 0
Last schedule failure        : no error
Last failure time            : never

===============================================================================

The following command provides more information on the execution cycle:

*A:4LS_CloudMG# show system script-control script-policy "test"

===============================================================================
Script-policy Information
===============================================================================
Script-policy                : test
Script-policy Owner          : TiMOS CLI
Administrative status        : enabled
Operational status           : enabled
Script                       : bring
Script owner                 : TiMOS CLI
Script source location       : cf3:/fetch.txt
Script results location      : cf3:/results.txt
Max running allowed          : 1
Max completed run histories  : 1
Max lifetime allowed         : 0d 01:00:00 (3600 seconds)
Completed run histories      : 1
Executing run histories      : 0
Initializing run histories   : 0
Max time run history saved   : 0d 01:00:00 (3600 seconds)
Script start error           : N/A
Last change                  : 2019/09/13 10:31:20 EEST
Max row expire time          : never
Last application             : cron
Last auth. user account      : not-specified

===============================================================================
Script Run History Status Information
-------------------------------------------------------------------------------
Script Run #25
-------------------------------------------------------------------------------
Start time    : 2019/09/13 11:29:25 EEST
End time      : 2019/09/13 11:29:26 EEST
Elapsed time  : 0d 00:00:01             Lifetime      : 0d 00:00:00
State         : terminated              Run exit code : noError
Result time   : 2019/09/13 11:29:26 EEST
Keep history  : 0d 00:59:41
Error time    : never
Results file  : cf3:/results.txt_20190913-082924-UTC.646420.out
Run exit      : Success
Error         : N/A
Application   : cron                    Auth. user ac*: not-specified
* indicates that the corresponding row element may have been truncated.
===============================================================================

Every time a crontab job is executed, a log file is generated. The filename will be "results.txt_<timestamp>.out", where:

results.txt: configured in the script-policy
<timestamp>:file timestamp. As an example: 20190913-082324-UTC.646436.

An example log file will be: results.txt_20190913-082324-UTC.646436.out and the log file contents is as follows:

Pre-processing configuration file (V0v0)...
Completed processing 4 lines in 0.0 seconds
*A:4LS_CloudMG# file
*A:4LS_CloudMG# copy ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt . force
Copying file ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt ... OK
1 file copied.
*A:4LS_CloudMG# exit
*A:4LS_CloudMG# admin application-assurance group 1 url-list "test" upgrade
Executed 4 lines in 0.7 seconds from file cf3:\fetch.txt

Finally, using the following command, the operator may check when the list was last upgraded and verify that the cron job runs as intended:

# show application-assurance group 1 url-list "test"

===============================================================================
Application Assurance Group 1 url-list "test"
===============================================================================
Description              : (Not Specified)
Size                     : standard
Admin Status             : Up
Oper Status              : Up
Oper Flags               : <none>
File deployed to ISAs    : Yes

-------------------------------------------------------------------------------
Upgrade Statistics
-------------------------------------------------------------------------------
Last Success             : 09/13/2019 11:49:57
  Deployed
    File Name            : cf3:\list.txt
    URL Entries          : 8 ( 0.05% full)
    URL Characters       : 103 (~0.00% full)
    Blank/Comment Lines  : 0
Last Attempt             : 09/13/2019 11:49:57
  Result                 : Success
    File Name            : cf3:\list.txt
===============================================================================

Conclusion

This chapter, intended for Application Assurance (AA) network architects and engineers, provides two examples for deploying URL-list filtering, upgrading the list and displaying its statistics, as well as configuring a cron job so that the system will periodically fetch a new URL list file and upgrade the list automatically.