Application Assurance — Local URL List Filtering
This chapter provides information about the Application Assurance local URL list filtering.
Topics in this chapter include:
Applicability
This chapter is applicable to all 7x50 systems supporting Application Assurance and was initially based on SR OS Release 13.0.R2. The chapter has been updated with CLI and functionality changes introduced in SR OS Release 19.10.R1, but the CLI in the sections URL-list update and Cron job is based on SR OS Release 16.0.
There are no specific prerequisites for this feature.
Overview
The local URL-list filtering capability provided by Application Assurance offers the following:
configuration of a list of URLs which are globally allowed (allow-list)
configuration of a list of restricted URLs subscribers should be prevented from accessing (deny-list)
The lists are stored locally in the 7x50 system compact flash (CF).
This chapter provides a guide for configuring deny-lists.
Deny-lists assist service providers to comply with regulatory requirements for network-wide URL filtering policies, such as:
Court-ordered URL takedown
Child protection
Government-mandated URL takedown list
The 7x50 uses the Application Assurance capabilities to extract the URL from a subscriber HTTP/HTTPS request and compare it to the list of URLs contained in the local file. If a match occurs, the subscriber request is redirected to a preconfigured web server landing page, typically describing why the access to this resource was denied.
The system supports both unencrypted and OpenSSL Triple Data Encryption Standard (3DES) encrypted file formats to protect the content of the list.
HTTP/HTTPS filtering
Each HTTP request within a TCP flow is analyzed and filtered. For HTTPS traffic, the system extracts the domain name information contained in the Transport Layer Security (TLS) Server Name Indication (SNI).
Setup details
The setup consists of the following elements, as shown in Local URL-list filtering setup:
7x50 + ISA-AA
Web server (redirect landing page)
FTP server (source for the URL-list file)
Subscriber (desktop/laptop/tablet/smart phone)
Internet access
Optional: AAA for subscriber authentication and policy modification
This chapter is written in the context of a residential or WiFi deployment. However, local URL-list filtering is also applicable to business VPN services.
Configuration
To configure the system for local URL-list filtering, the operator needs to:
Create a URL-list policy referencing a valid URL-list file located on the system compact flash
Create a URL-filter policy for local filtering by referencing the URL-list policy previously created
Create an App-QoS-Policy (AQP) to apply this url-filter policy
Optionally configure a cron job to automatically fetch a new list file and upgrade the URL list.
URL-list policy and URL-filter policy
In the following example, two dedicated URL-list and URL-filter policies show URL filtering based on a plain text file and an encrypted file:
configure
application-assurance group 1
url-list "denylist1-encrypted" create
description "Demo URL Filtering List - Encrypted File"
decrypt-key "ON3HU2GFPHmpOHwWbSGw/zdM4iuxzySpqS7pw/u3qIcuG4mABmrhc."
hash2
file "cf3:\aa-url-list\url-list1.encrypted"
no shutdown
exit
url-filter "local-filter-list1-encrypted" create
local-filtering
deny-list "denylist1-encrypted"
default-action allow
http-redirect "redirect-denylist"
exit
no shutdown
exit
configure
application-assurance group 1
url-list "denylist1-plaintext" create
description "Demo URL Filtering List - Plaintext File"
file "cf3:\aa-url-list\url-list1-plaintext.txt"
no shutdown
exit
url-filter "local-filter-list1-txt" create
local-filtering
deny-list "denylist1-plaintext"
default-action allow
http-redirect "redirect-denylist"
exit
no shutdown
exit
In the preceding example, both URL-filter policies are defined using default-action allow. The default action is used in case the file could not be loaded by the system, either at boot time or the first time the URL-list file was configured in the system. Possible causes are, for example:
File corrupted, compact flash corrupted
Incorrect file encryption format or password
Wrong URL format in the file
Too many URLs in the file
Operators should always use default-action allow when configuring the URL-filter policy associated with a URL-list file because the file or the CF may be corrupted, in which case the system logs an error and a trap is raised.
Note that if a valid URL-list file was previously in use, and an invalid file is uploaded and the URL-list policy upgraded using this file, then the system will continue using the previous list.
HTTP-redirect policy
Both URL-filter policies defined in the preceding example refer to the following http-redirect policy; subscribers accessing a URL from the URL-list file are redirected to the following landing page:
configure
application-assurance group 1
http-redirect "redirect-denylist" create
description "Redirect for Local List URL Filtering"
template 5
tcp-client-reset
redirect-url "http://172.16.70.100/Redirect/redirect-denylist.html?
Request edURL=$URL"
no shutdown
exit
URL-list file
File format
A URL-list file may contain either hostnames or URLs.
To create a URL-list containing hostnames, set expression-match in the url-list configuration:
config>app-assure>group# url-list url-list-name [create]
description <description-string>
no description
decrypt-key key | hash-key | hash2-key [hash | hash2]
no decrypt-key
file file-url
no file
size url-list-size
expression-match
[no] shutdown
A URL-list with hostnames only (using expression-match) may contain the following wildcards:
Head anchors character set [^ *]
Tail anchors character set [$ *]
Mid expression character set [\d \l \. \**]
Hex escaped characters [\x00 - \xFF]
Note that when expression-match is enabled, the list should contain hostnames only (with optional wildcards).
When configuring a URL-list with expression-match disabled (default), the system supports the following format for the URLs contained in the URL-list file:
URLs without the HTTP keyword. For example:
www.domain.com/path
URLs with the HTTP keyword. For example:
http://www.domain.com/path
In all cases, the following is supported:
Comment lines starting with the number sign character (#). For example:
# This is a comment line
Printable ASCII characters. URLs using non-printable ASCII characters are percent-encoded by the web browser automatically and, therefore, need to be percent-encoded in the URL-list file.
File encryption
OpenSSL triple DES -nosalt is the supported encryption format. Files can be encrypted offline on a server using the following command:
openssl des3 -nosalt -in <input.txt> -out <output.enc>
List upgrade
The URL-list file can be upgraded using the admin command:
A:BNG# admin application-assurance group 1 url-list "denylist1-plaintext" upgrade
The upgrade result is logged in the system log-id 99:
A:BNG# show log log-id 99
===============================================================================
Event Log 99
===============================================================================
Description : Default System Log
Memory Log contents [size=500 next event=72 (not wrapped)]
71 2015/07/07 13:09:25.01 EST MINOR: APPLICATION_ASSURANCE #4446 Base url-list success
"URL list "denylist1-plaintext" in ISA-AA group 1 has been updated. There are 3 entries in the URL list."
App-profiles and app-service-options
Application Assurance policies can be selectively applied to specific AA subscribers by modifying the app-profile assigned to the subscriber or using Application Service Option (ASO) override. See SR OS User Guide for more information about modifying the app-profile or ASO assigned to AA subscribers (RADIUS, Gx, Override).
In this example, the following ASO configuration is used:
configure
application-assurance group 1:1 policy
app-service-options
characteristic "local-list-filtering" create
value "no"
value "yes-encrypted"
value "yes-plaintext"
default-value "no"
exit
exit
app-profile "1-1/Default" create
divert
exit
The ASO characteristic local-list-filtering value of yes-encrypted and yes-plaintext enable the AQP entry 210 and 220 in the example:
configure
application-assurance group 1:1 policy app-qos-policy
entry 210 create
match
characteristic "local-list-filtering" eq "yes-encrypted"
exit
action
url-filter "local-filter-list1-encrypted"
exit
no shutdown
exit
entry 220 create
match
characteristic "local-list-filtering" eq "yes-plaintext"
exit
action
url-filter "local-filter-list1-txt"
exit
no shutdown
exit
If the url-filter policy needs to be applied to 100% of the subscribers in the network, it is also possible to remove the ASO match criteria.
URL-list update
The system supports a flexible mechanism to upgrade the URL list automatically, using either cron or the NSP, to comply with the regulatory requirements for list upgrade frequency.
To configure a crontab job to periodically upgrade the URL list, the operator needs to:
Generate the file to be periodically executed and store it to compact flash. As an example, create a file with filename "fetch.txt" with the following content:
file copy ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt . force exit admin application-assurance group 1 url-list "test" upgrade
The preceding commands will fetch a file from an ftp server and store it in compact flash. Assuming the operator has configured a local url-list "test" containing the file "list.txt", the url list will be upgraded.
Configure the script policy:
>config>system>script-control# info ---------------------------------------------- script "bring" location "cf3:/fetch.txt" no shutdown exit script-policy "test" results "cf3:/results.txt" script "bring" no shutdown exit
Configure the crontab job:
>config>system>cron# info ---------------------------------------------- schedule "bring_list" interval 60 script-policy "test" no shutdown exit ----------------------------------------------
The preceding configuration will execute the commands stored in the file "fetch.txt" every 60 seconds. A value of 60 seconds was chosen for the test. In a real deployment, a list would be typically updated every 12 – 24 hours. A log file (results "results.txt") will also be created.
The end result will be that the system will automatically fetch from an ftp server a new list file, store it to compact flash and upgrade the URL list.
Show commands
url-list
The status of the URL list can be shown in the CLI. The url-list show command provides basic admin and operational status, as well as the number of URLs in the list. The command also provides reasons for any possible issue related to loading the list, as well as the last successfully deployed file and the last upgrade attempt. Therefore, the operator can determine whether the latest version of the file is currently in use or if an error occurred when trying to upgrade the list.
Show command output:
Label Description
Admin Status [Up | Down] - Administrative status of the url-list
Oper Status [Up | Down] - Operational status of the url-list
Oper Flags [admin-down | file-does-not-exist |invalid-file-format |
too-many-urls| switch-over-error]
File Deployed to ISA [Yes | No] - This flag describes if the file located in
the compact flash is the one deployed in the ISA, in the
event the file is overwritten and before the admin upgrade
command is used this flag will display ‟No”.
Upgrade Statistics
Last Success Last time the list was successfully upgraded
File Name File name for the last successful upgrade
URL Entries Number of URLs loaded at the last success
Blank/Comment Lines Number of blank or commented out lines
Last Attempt Last time the operator tried to upgrade the list
Result Success | Failure. Result of the last upgrade
File Name File name for the last upgrade attempt
*A:Dut-C# show application-assurance group 1 url-list "Deny List1"
======================================================================
Application Assurance Group 1 url-list "Deny List1"
======================================================================
Description : (Not Specified)
Size : standard
Host Expressions : disabled
Admin Status : Up
Oper Status : Up
Oper Flags : <none>
File deployed to ISAs : Yes
-------------------------------------------------------------------------------
Upgrade Statistics
-------------------------------------------------------------------------------
Last Success : 11/02/2020 15:07:40
Deployed
File Name : cf1:/host.txt
URL Entries : 1 ( 0.01% full)
URL Characters : 6 (~0.00% full)
URL Host Expr Entries: 1 ( 0.01% full)
Blank/Comment Lines : 1
Last Attempt : 11/02/2020 15:07:40
Result : Success
File Name : cf1:/host.txt
======================================================================
url-filter
The url-filter show command provides its operational and admin status, as well as actions taken, such as the number of redirects. With URL list filtering, using a default-action set to allow, the only counters increasing are allow, redirect, and default.
*A:Dut-C>config>app-assure>group>url-filter# show application-assurance group 1 url-filter "Url Filter1"
===============================================================================
Application Assurance Group 1 URL Filter "Url Filter1"
===============================================================================
Description : (Not Specified)
Admin Status : Up
Oper Status : Up
Oper Flags : <none>
HTTP Request Filtering : all
AQP Referenced : No
-------------------------------------------------------------------------------
URL Stats Summary
-------------------------------------------------------------------------------
Total Requests : 0 Default Action : 0
Requests Allowed: 0 Reqs Block/Redir: 0
-------------------------------------------------------------------------------
Local Filter
-------------------------------------------------------------------------------
deny-list : Deny List1
Admin Status : Up
Oper Status : Up
Oper Flags : <none>
Number of URLs : 1
Default Action : block-all
HTTP Redirect : (Not Specified)
URL-List Lookups : 0
Match : 0
Miss : 0
Default Action : 0
===============================================================================
http-redirect
The http-redirect show command provides more information about how the traffic was blocked; for example, it differentiates TCP client reset used for HTTPS from regular redirect used for HTTP traffic.
A:BNG# show application-assurance group 1 http-redirect "redirect-denylist"
===============================================================================
Application Assurance Group 1 HTTP Redirect redirect-denylist
===============================================================================
Description : Redirect for Local List URL Filtering
Template : 5
: Redirect supporting macro substitution using HTTP 302
Redirect URL : http://172.16.70.100/Redirect/redirect-denylist.
html?RequestedURL=$URL
Admin Status : Up
AQP Ref : No
-------------------------------------------------------------------------------
Summary Statistics
-------------------------------------------------------------------------------
Grp:Part Redirects Client Resets Redirects
Sent Sent Not Sent
-------------------------------------------------------------------------------
1:1 2 1 0
-------------------------------------------------------------------------------
Total 2 1 0
-------------------------------------------------------------------------------
===============================================================================
Cron job
This chapter provides details on how to verify that the cron job executes successfully. The operator can see the location and execution interval of the file which is executed periodically, the time it was last executed and any possible errors which may have occurred during execution using the following command:
# show system cron schedule "bring_list"
===============================================================================
CRON Schedule Information
===============================================================================
Schedule : bring_list
Schedule owner : TiMOS CLI
Description : none
Administrative status : enabled
Operational status : enabled
Script Policy : test
Script Policy Owner : TiMOS CLI
Script : bring
Script Owner : TiMOS CLI
Script source location : cf3:/fetch.txt
Script results location : cf3:/results.txt
Schedule type : periodic
Interval : 0d 00:01:00 (60 seconds)
Repeat count : infinite
Next scheduled run : 0d 00:00:51
End time : none
Weekday : none
Month : none
Day of month : none
Hour : none
Minute : none
Number of schedule runs : 12
Last schedule run : 2019/09/13 11:28:25 EEST
Number of schedule failures : 0
Last schedule failure : no error
Last failure time : never
===============================================================================
The following command provides more information on the execution cycle:
*A:4LS_CloudMG# show system script-control script-policy "test"
===============================================================================
Script-policy Information
===============================================================================
Script-policy : test
Script-policy Owner : TiMOS CLI
Administrative status : enabled
Operational status : enabled
Script : bring
Script owner : TiMOS CLI
Script source location : cf3:/fetch.txt
Script results location : cf3:/results.txt
Max running allowed : 1
Max completed run histories : 1
Max lifetime allowed : 0d 01:00:00 (3600 seconds)
Completed run histories : 1
Executing run histories : 0
Initializing run histories : 0
Max time run history saved : 0d 01:00:00 (3600 seconds)
Script start error : N/A
Last change : 2019/09/13 10:31:20 EEST
Max row expire time : never
Last application : cron
Last auth. user account : not-specified
===============================================================================
Script Run History Status Information
-------------------------------------------------------------------------------
Script Run #25
-------------------------------------------------------------------------------
Start time : 2019/09/13 11:29:25 EEST
End time : 2019/09/13 11:29:26 EEST
Elapsed time : 0d 00:00:01 Lifetime : 0d 00:00:00
State : terminated Run exit code : noError
Result time : 2019/09/13 11:29:26 EEST
Keep history : 0d 00:59:41
Error time : never
Results file : cf3:/results.txt_20190913-082924-UTC.646420.out
Run exit : Success
Error : N/A
Application : cron Auth. user ac*: not-specified
* indicates that the corresponding row element may have been truncated.
===============================================================================
Every time a crontab job is executed, a log file is generated. The filename will be "results.txt_<timestamp>.out", where:
results.txt: configured in the script-policy
<timestamp>:file timestamp. As an example: 20190913-082324-UTC.646436.
An example log file will be: results.txt_20190913-082324-UTC.646436.out and the log file contents is as follows:
Pre-processing configuration file (V0v0)...
Completed processing 4 lines in 0.0 seconds
*A:4LS_CloudMG# file
*A:4LS_CloudMG# copy ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt . force
Copying file ftp://user:pwd@192.168.1.99/home/cmg/test/list.txt ... OK
1 file copied.
*A:4LS_CloudMG# exit
*A:4LS_CloudMG# admin application-assurance group 1 url-list "test" upgrade
Executed 4 lines in 0.7 seconds from file cf3:\fetch.txt
Finally, using the following command, the operator may check when the list was last upgraded and verify that the cron job runs as intended:
# show application-assurance group 1 url-list "test"
===============================================================================
Application Assurance Group 1 url-list "test"
===============================================================================
Description : (Not Specified)
Size : standard
Admin Status : Up
Oper Status : Up
Oper Flags : <none>
File deployed to ISAs : Yes
-------------------------------------------------------------------------------
Upgrade Statistics
-------------------------------------------------------------------------------
Last Success : 09/13/2019 11:49:57
Deployed
File Name : cf3:\list.txt
URL Entries : 8 ( 0.05% full)
URL Characters : 103 (~0.00% full)
Blank/Comment Lines : 0
Last Attempt : 09/13/2019 11:49:57
Result : Success
File Name : cf3:\list.txt
===============================================================================
Conclusion
This chapter, intended for Application Assurance (AA) network architects and engineers, provides two examples for deploying URL-list filtering, upgrading the list and displaying its statistics, as well as configuring a cron job so that the system will periodically fetch a new URL list file and upgrade the list automatically.