BGP Convergence — Delayed Route Advertisement
This chapter describes BGP Convergence — Delayed Route Advertisement.
Topics in this chapter include:
Applicability
The information and MD-CLI configuration in this chapter are based on SR OS Release 20.7.R2. BGP Delayed Route Advertisement is supported in SR OS Release 19.7.R1 and later.
Overview
When the BGP process on a router is starting up or restarting, BGP convergence is finished after the restarting router completes the following actions:
-
Reestablish the sessions with configured and discovered BGP neighbors.
-
Relearn BGP routes advertised by the direct BGP neighbors (their best paths plus potentially some additional paths).
-
Advertise to its direct neighbors the locally originated BGP routes plus the received routes from its set of best paths.
By default, the preceding steps are executed in parallel. After the first BGP session is reestablished, the restarting router starts advertising its own best paths to the BGP neighbor, even though it is still learning BGP routes and rebuilding its RIB-IN database. When more BGP sessions come up and more routes are learned, it is possible that routes previously considered best are no longer best, leading to multiple route advertisements for the same prefix, as shown in Default SR OS behavior when the BGP process restarts.
Multiple route advertisements increase the processing workload on the restarting router and on its BGP neighbors. This lengthens the overall convergence time and it can cause short-term inefficiencies in traffic forwarding.
The BGP delayed route advertisement feature provides the following two convergence timers to offer the operator more control on the BGP convergence process when BGP is starting up or restarting: min-wait-to-advertise and max-wait-to-advertise. This feature applies to IPv4 unicast and IPv6 unicast routes of the base router BGP instance and VPRN BGP instances. BGP convergence tuning allows different timers in the base router and the VPRNs. Also, the max-wait-to-advertise timer can be different for IPv4 and IPv6 address families. BGP convergence tuning with delayed route advertisement shows the BGP convergence tuning.
When a BGP peer has advertised all its routes for a specific address family, it sends an End of RIB (EOR) marker for each address family; for example, peer 192.0.2.4 sent the following EOR for IPv4:
171 2020/10/06 13:53:07.312 UTC MINOR: DEBUG #2001 Base Peer 1: 192.0.2.4
"Peer 1: 192.0.2.4: UPDATE
Peer 1: 192.0.2.4 - Received BGP UPDATE:
Withdrawn Length = 0
Total Path Attr Length = 0
End-of-Rib marker (IPV4)
"
The restarting node will never advertise routes before the min-wait-to-advertise timer has expired. In BGP convergence tuning with delayed route advertisement, no routes were received at that time, but it is possible. Each peer advertises its routes followed by an EOR message per address family. When the restarting node receives all the expected EOR messages (and after the min-wait-to-advertise timer expires), it starts advertising its best routes. However, if the max-wait-to-advertise timer for the address family expires before all expected EORs have been received, it also starts advertising its best routes.
The timer values must be chosen well, because it is possible that the convergence degrades instead of improves with unsuitable timer values. The timer values depend on the BGP topology (number of peers, number of prefixes per peer, and BGP activeness of the peers). Timer values can be optimized by trial and error, and may have to be reviewed in case of network changes.
BGP convergence timers shows that the min-wait-to-advertise timer starts when the BGP process starts up or restarts, whereas the max-wait-to-advertise timer starts when the first peer (dynamic or configured) is established. It also shows that BGP convergence tuning does not apply to a new peer (peer-X) that is established after the min-wait-to-advertise timer has expired.
The BGP convergence process can be monitored with the show router bgp convergence command. BGP convergence states shows the different BGP convergence states.
The BGP convergence states are:
-
Waiting: when BGP convergence timers are configured and no peer has reconnected yet.
-
Started: when the first peer (dynamic or configured) is established.
-
Partial: when the first EOR is received from a neighbor for a specific address family.
-
Converged: when the last EOR for an address family is received. If that occurs before the max-time-to-advertise timer expires, the restarting node starts advertising its RIB-OUT.
-
Timeout: when the max-wait-to-advertise timer expires before the last EOR for an address family is received. The restarting node advertises its RIB-OUT when the timer expires.
When the feature is implemented, BGP maintains information about the convergence process associated with the last startup.
Configuration
The following example shows the principles of SR OS BGP convergence, whereas real-life examples have much larger numbers of BGP sessions and routes. Example topology shows the example topology with one node in Autonomous System (AS) 64501 and three nodes in AS 64500. On all four nodes, VPRN 1 is configured in AS 64496.
Example topology only shows the IPv4 addresses, but all interfaces also have IPv6 addresses.
The initial configuration on the nodes includes:
-
Cards, MDAs, ports
-
Router interfaces, with IPv4 and IPv6 addresses
-
SR-ISIS in the base router on the three nodes in AS 64500
-
IS-IS in VPRN 1 on all four nodes in AS 64496
In the base router, an eBGP session is established between PE-1 in AS 64501 and PE-2 in AS 64500. For the iBGP sessions in AS 64500, PE-2 acts as a Route Reflector (RR). The BGP configuration in the base router on PE-2 is as follows:
# on PE-2:
configure {
router "Base" {
autonomous-system 64500
bgp {
split-horizon true
group "eBGP" {
peer-as 64501
local-address "int-PE-2-PE-1"
local-as {
as-number 64500
}
import {
policy ["1:1"]
}
}
group "iBGP-IPv4" {
peer-as 64500
family {
ipv4 true
}
cluster {
cluster-id 192.0.2.2
}
}
group "iBGP-IPv6" {
peer-as 64500
family {
ipv6 true
}
cluster {
cluster-id 192.0.2.2
}
}
neighbor "192.0.2.3" {
group "iBGP-IPv4"
next-hop-self true
export {
policy ["export-10.21"]
}
}
neighbor "192.0.2.4" {
group "iBGP-IPv4"
next-hop-self true
export {
policy ["export-10.21"]
}
}
neighbor "192.168.12.1" {
group "eBGP"
next-hop-self true
family {
ipv4 true
}
export {
policy ["export-10.21" "1:1"]
}
}
neighbor "2001:db8::2:3" {
group "iBGP-IPv6"
next-hop-self true
export {
policy ["export-10:21"]
}
}
neighbor "2001:db8::2:4" {
group "iBGP-IPv6"
next-hop-self true
export {
policy ["export-10:21"]
}
}
neighbor "2001:db8::12:1" {
group "eBGP"
next-hop-self true
family {
ipv6 true
}
export {
policy ["export-10:21" "1:1"]
}
}
}
The policies are the following:
# on PE-2:
configure {
policy-options {
community "1:1" {
member "1:1" { }
}
prefix-list "10.21.0.0/16" {
prefix 10.21.0.0/16 type longer {
}
}
prefix-list "_::10:21_" {
prefix 2001:db8::10:21:0:0/120 type longer {
}
}
policy-statement "1:1" {
entry 10 {
from {
community {
name "1:1"
}
}
action {
action-type accept
}
}
}
policy-statement "export-10.21" {
entry 10 {
from {
prefix-list ["10.21.0.0/16"]
}
action {
action-type accept
community {
add ["1:1"]
}
}
}
}
policy-statement "export-10:21" {
entry 10 {
from {
prefix-list ["_::10:21_"]
}
action {
action-type accept
community {
add ["1:1"]
}
}
}
}
The BGP configuration in the base router is similar on the other PEs, with similar export policies.
BGP is also configured in VPRN 1, with similar export policies. On RR PE-2, the BGP configuration in VPRN 1 is as follows:
# on PE-2
configure {
service {
vprn "VPRN 1" {
autonomous-system 64496
bgp {
router-id 172.31.0.2
split-horizon true
group "iBGP-VPRN1" {
peer-as 64496
cluster {
cluster-id 172.31.0.2
}
}
neighbor "172.31.0.1" {
group "iBGP-VPRN1"
local-address 172.31.0.2
family {
ipv4 true
}
export {
policy ["export-10.1"]
}
}
neighbor "172.31.0.3" {
group "iBGP-VPRN1"
local-address 172.31.0.2
family {
ipv4 true
}
export {
policy ["export-10.1"]
}
}
neighbor "172.31.0.4" {
group "iBGP-VPRN1"
local-address 172.31.0.2
family {
ipv4 true
}
export {
policy ["export-10.1"]
}
}
neighbor "2001:db8::31:0:1" {
group "iBGP-VPRN1"
family {
ipv6 true
}
export {
policy ["export-10:1"]
}
}
neighbor "2001:db8::31:0:3" {
group "iBGP-VPRN1"
family {
ipv6 true
}
export {
policy ["export-10:1"]
}
}
neighbor "2001:db8::31:0:4" {
group "iBGP-VPRN1"
family {
ipv6 true
}
export {
policy ["export-10:1"]
}
}
}
The configuration is similar on the other nodes.
The following BGP summary on PE-2 shows the different sessions where PE-2 receives one IPv4 or IPv6 route per neighbor and advertises three IPv4 or IPv6 routes per neighbor, both in the base router (Def. Instance) and in VPRN 1 (Svc: 1):
[]
A:admin@PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 8 0 00h01m56s 1/1/3 (IPv4)
11 0
192.0.2.4
Def. Instance 64500 8 0 00h01m43s 1/1/3 (IPv4)
11 0
192.168.12.1
Def. Instance 64501 9 0 00h02m20s 1/1/3 (IPv4)
11 0
2001:db8::2:3
Def. Instance 64500 8 0 00h01m56s 1/1/3 (IPv6)
11 0
2001:db8::2:4
Def. Instance 64500 8 0 00h01m43s 1/1/3 (IPv6)
11 0
2001:db8::12:1
Def. Instance 64501 9 0 00h02m14s 1/1/3 (IPv6)
11 0
172.31.0.1
Svc: 1 64496 9 0 00h02m01s 1/1/3 (IPv4)
12 0
172.31.0.3
Svc: 1 64496 9 0 00h02m02s 1/1/3 (IPv4)
12 0
172.31.0.4
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv4)
10 0
2001:db8::31:0:1
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
2001:db8::31:0:3
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
2001:db8::31:0:4
Svc: 1 64496 8 0 00h01m50s 1/1/3 (IPv6)
10 0
-------------------------------------------------------------------------------
By default, BGP does not delay route advertisement. The following show command on PE-2 shows that no min-wait-to-advertise timer and no max-wait-to-advertise timer is configured (the default value is 0). The number of established peers is 3 for IPv4 and IPv6 in the base router.
[]
A:admin@PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 0
Established peers at min wait timer expiry : N/A
Current established peers : 3
First session established time : N/A
Last session established time : N/A
Max Wait advertise timer : 0
Converged peers : N/A
Converged state : N/A
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 0
Established peers at min wait timer expiry : N/A
Current established peers : 3
First session established time : N/A
Last session established time : N/A
Max Wait advertise timer : 0
Converged peers : N/A
Converged state : N/A
Converged time : N/A
===============================================================================
A similar command can be launched for VPRN 1: show router 1 bgp convergence. The output is similar, but not shown here.
On PE-2, BGP delayed route advertisement is configured with min-wait-to-advertise equal to 20 seconds in the base router and min-wait-to-advertise equal to 60 seconds in VPRN 1. For both cases, the max-wait-to-advertise is three times as long as the min-wait-to-advertise, but it is possible to have different max-wait-to-advertise timers for IPv4 and IPv6.
In this example, BGP delayed route advertisement is only configured on PE-2, while the other nodes keep advertising their routes immediately after the BGP session is reestablished. PE-2 will accept these routes, but it will only advertise them after receiving the last expected EOR for IPv4 or IPv6 (for the base router or VPRN 1) and min-wait-to-advertise timer expires. If the max-wait-to-advertise timer expires before the last expected EOR is received for IPv4 or IPv6, PE-2 will start advertising the received routes.
# on PE-2:
configure {
router "Base" {
bgp {
convergence {
min-wait-to-advertise 20
family ipv4 {
max-wait-to-advertise 30
}
family ipv6 {
max-wait-to-advertise 30
}
}
}
}
service {
vprn "VPRN 1" {
bgp {
convergence {
min-wait-to-advertise 60
family ipv4 {
max-wait-to-advertise 180
}
family ipv6 {
max-wait-to-advertise 180
}
}
}
}
With this configuration, the BGP converged state on PE-2 changes to "waiting", because no BGP sessions are reestablished yet, so no BGP convergence tuning has taken place.
[]
A:admin@PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m00s
Last session established time : 00h00m00s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : waiting
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m00s
Last session established time : 00h00m00s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : waiting
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
The following clear command on PE-2 causes BGP to restart:
# on PE-2:
clear router bgp protocol
clear router 1 bgp protocol
When BGP restarts, the converged state remains "waiting" until the first peer is established.
With the first peer established, the converged state changes to "started", as follows:
[]
A:admin@PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m01s
Max Wait advertise timer : 30
Converged peers : 0
Converged state : started
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m01s
Max Wait advertise timer : 30
Converged peers : 0
Converged state : started
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
After a few seconds, PE-2 receives IPv4 and IPv6 routes from PE-3 and PE-4, both in the base router and VPRN 1, as follows:
[]
A:admin@PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv4)
4 0
192.0.2.4
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv4)
4 0
192.168.12.1
Def. Instance 64501 3 0 00h00m00s 0/0/0 (IPv4)
4 0
2001:db8::2:3
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::2:4
Def. Instance 64500 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::12:1
Def. Instance 64501 3 0 00h00m00s 0/0/0 (IPv6)
4 0
172.31.0.1
Svc: 1 64496 3 0 00h00m00s 0/0/0 (IPv4)
3 0
172.31.0.3
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv4)
4 0
172.31.0.4
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv4)
4 0
2001:db8::31:0:1
Svc: 1 64496 3 0 00h00m00s 0/0/0 (IPv6)
3 0
2001:db8::31:0:3
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv6)
4 0
2001:db8::31:0:4
Svc: 1 64496 5 0 00h00m01s 1/1/0 (IPv6)
4 0
-------------------------------------------------------------------------------
PE-2 accepts the received routes, but does not advertise the routes because the min-wait-to-advertise timer has not expired yet, and PE-2 only received IPv4 and IPv6 routes and EORs from PE-3 and PE-4, not from PE-1, so the converged state is "partial", as follows:
[]
A:admin@PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m02s
Max Wait advertise timer : 30
Converged peers : 2
Converged state : partial
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m02s
Max Wait advertise timer : 30
Converged peers : 2
Converged state : partial
Converged time : N/A
===============================================================================
The show router 1 bgp convergence command shows a similar output for VPRN 1, but is not shown here.
After a few seconds, all IPv4 and IPv6 routes have been received in the base router. PE-2 has received an EOR message from each neighbor in the base router. The following BGP summary shows that PE-2 has received and advertised all IPv4 and IPv6 routes in the base router, whereas it only received IPv4 and IPv6 routes from two neighbors in VPRN 1, not yet from PE-1:
[]
A:admin@PE-2# show router bgp summary all
===============================================================================
BGP Summary
===============================================================================
Legend : D - Dynamic Neighbor
===============================================================================
Neighbor
Description
ServiceId AS PktRcvd InQ Up/Down State|Rcv/Act/Sent (Addr Family)
PktSent OutQ
-------------------------------------------------------------------------------
192.0.2.3
Def. Instance 64500 5 0 00h00m12s 1/1/3 (IPv4)
7 0
192.0.2.4
Def. Instance 64500 5 0 00h00m11s 1/1/3 (IPv4)
7 0
192.168.12.1
Def. Instance 64501 5 0 00h00m14s 1/1/3 (IPv4)
5 0
2001:db8::2:3
Def. Instance 64500 5 0 00h00m12s 1/1/3 (IPv6)
7 0
2001:db8::2:4
Def. Instance 64500 5 0 00h00m11s 1/1/3 (IPv6)
7 0
2001:db8::12:1
Def. Instance 64501 5 0 00h00m13s 1/1/3 (IPv6)
5 0
172.31.0.1
Svc: 1 64496 4 0 00h00m13s 0/0/0 (IPv4)
4 0
172.31.0.3
Svc: 1 64496 5 0 00h00m12s 1/1/0 (IPv4)
4 0
172.31.0.4
Svc: 1 64496 5 0 00h00m11s 1/1/0 (IPv4)
4 0
2001:db8::31:0:1
Svc: 1 64496 4 0 00h00m13s 0/0/0 (IPv6)
4 0
2001:db8::31:0:3
Svc: 1 64496 5 0 00h00m12s 1/1/0 (IPv6)
4 0
2001:db8::31:0:4
Svc: 1 64496 5 0 00h00m11s 1/1/0 (IPv6)
4 0
-------------------------------------------------------------------------------
As a result of this, BGP is in the "converged" state in the base router, both for IPv4 and IPv6, as follows:
[]
A:admin@PE-2# show router bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : converged
Converged time : 00h00m20s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 20
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 30
Converged peers : 3
Converged state : converged
Converged time : 00h00m20s
===============================================================================
The converged time is only applicable in the "converged" state and is measured relative to BGP instance restart at time T=0.
BGP is still in the "partial" state within the VPRN 1 context, both for IPv4 and IPv6, as follows:
[]
A:admin@PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : partial
Converged time : N/A
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 0
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : partial
Converged time : N/A
===============================================================================
After a while, PE-1 also advertises its routes for VPRN 1, followed by EORs for IPv4 and IPv6. BGP converges for VPRN 1, as follows:
[]
A:admin@PE-2# show router 1 bgp convergence
===============================================================================
BGP IPv4 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : converged
Converged time : 00h01m00s
===============================================================================
===============================================================================
BGP IPv6 Convergence
===============================================================================
Min wait advertise timer : 60
Established peers at min wait timer expiry : 3
Current established peers : 3
First session established time : 00h00m01s
Last session established time : 00h00m03s
Max Wait advertise timer : 180
Converged peers : 3
Converged state : converged
Converged time : 00h01m00s
===============================================================================
Conclusion
With BGP convergence tuning (by means of delaying route advertisements using two timers), less path churn and fewer advertisements can result in faster convergence. BGP convergence is mainly important in scaled environments (high number of BGP sessions and routes). As a result, the advertised paths are more optimal. The BGP convergence process can be monitored using a show command.