MC-LAG lab – basic L2 connectivity
My topology for MC-LAG practice:
I already configured IP fabric between vQFX and vMX nodes using EBGP (to save space, I show the configuration for only one device – the rest are configured the same way):
alex@vMX1> show configuration protocols bgp group underlay
type external;
export direct;
local-as 65001;
multipath multiple-as;
neighbor 192.168.0.1 { ###vQFX1
peer-as 65011;
}
neighbor 192.168.0.3 { ###vQFX2
peer-as 65022;
}
alex@vMX1> show configuration policy-options policy-statement direct
term lo {
from {
protocol direct;
interface [ lo0.0 irb.100 irb.200 ];
}
then accept;
}
type external;
export direct;
local-as 65001;
multipath multiple-as;
neighbor 192.168.0.1 { ###vQFX1
peer-as 65011;
}
neighbor 192.168.0.3 { ###vQFX2
peer-as 65022;
}
alex@vMX1> show configuration policy-options policy-statement direct
term lo {
from {
protocol direct;
interface [ lo0.0 irb.100 irb.200 ];
}
then accept;
}
so every router has IP connectivity for each other loopback address:
alex@vMX1> show route table inet.0
inet.0: 15 destinations, 17 routes (15 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[Direct/0] 01:55:51
> via lo0.0
2.2.2.2/32 *[BGP/170] 00:17:19, localpref 100
AS path: 65011 65002 I, validation-state: unverified
> to 192.168.0.1 via ge-0/0/1.0
to 192.168.0.3 via ge-0/0/2.0
[BGP/170] 00:17:19, localpref 100
AS path: 65022 65002 I, validation-state: unverified
> to 192.168.0.3 via ge-0/0/2.0
11.11.11.11/32 *[BGP/170] 00:41:53, localpref 100
AS path: 65011 I, validation-state: unverified
> to 192.168.0.1 via ge-0/0/1.0
22.22.22.22/32 *[BGP/170] 00:17:33, localpref 100
AS path: 65022 I, validation-state: unverified
> to 192.168.0.3 via ge-0/0/2.0
inet.0: 15 destinations, 17 routes (15 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32 *[Direct/0] 01:55:51
> via lo0.0
2.2.2.2/32 *[BGP/170] 00:17:19, localpref 100
AS path: 65011 65002 I, validation-state: unverified
> to 192.168.0.1 via ge-0/0/1.0
to 192.168.0.3 via ge-0/0/2.0
[BGP/170] 00:17:19, localpref 100
AS path: 65022 65002 I, validation-state: unverified
> to 192.168.0.3 via ge-0/0/2.0
11.11.11.11/32 *[BGP/170] 00:41:53, localpref 100
AS path: 65011 I, validation-state: unverified
> to 192.168.0.1 via ge-0/0/1.0
22.22.22.22/32 *[BGP/170] 00:17:33, localpref 100
AS path: 65022 I, validation-state: unverified
> to 192.168.0.3 via ge-0/0/2.0
This connectivity will be used later for vQFX ICCP session.
There are 2 pairs of MC-LAG devices – 2 vQFX nodes on the left and 2 vMX nodes on the right. vMX v14 emulate CE devices and connect to the corresponding MC-LAG pair with simple LACP LAG:
alex@MX2> show configuration interfaces
ge-0/0/0 {
description vQFX1;
gigether-options {
802.3ad ae0;
}
}
ge-0/0/1 {
description vQFX2;
gigether-options {
802.3ad ae0;
}
}
ae0 {
description to_MC-LAG_vQFX;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
}
}
unit 100 {
vlan-id 100;
family inet {
address 172.16.0.22/24;
}
family inet6 {
address 2001:dead:beef:100::22/64;
}
}
}
lo0 {
unit 0 {
family inet {
address 222.222.222.222/32;
}
}
}
alex@MX2> show configuration routing-options
rib inet6.0 {
static {
route ::/0 next-hop 2001:dead:beef:100::a;
}
}
static {
route 0.0.0.0/0 next-hop 172.16.0.254;
}
router-id 222.222.222.222;
alex@MX2> show configuration logical-systems
second {
interfaces {
ae0 {
unit 200 {
vlan-id 200;
family inet {
address 172.16.1.22/24;
}
family inet6 {
address 2001:dead:beef:200::22/64;
}
}
}
}
routing-options {
rib inet6.0 {
static {
route ::/0 next-hop 2001:dead:beef:200::a;
}
}
static {
route 0.0.0.0/0 next-hop 172.16.1.254;
}
}
}
ge-0/0/0 {
description vQFX1;
gigether-options {
802.3ad ae0;
}
}
ge-0/0/1 {
description vQFX2;
gigether-options {
802.3ad ae0;
}
}
ae0 {
description to_MC-LAG_vQFX;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
}
}
unit 100 {
vlan-id 100;
family inet {
address 172.16.0.22/24;
}
family inet6 {
address 2001:dead:beef:100::22/64;
}
}
}
lo0 {
unit 0 {
family inet {
address 222.222.222.222/32;
}
}
}
alex@MX2> show configuration routing-options
rib inet6.0 {
static {
route ::/0 next-hop 2001:dead:beef:100::a;
}
}
static {
route 0.0.0.0/0 next-hop 172.16.0.254;
}
router-id 222.222.222.222;
alex@MX2> show configuration logical-systems
second {
interfaces {
ae0 {
unit 200 {
vlan-id 200;
family inet {
address 172.16.1.22/24;
}
family inet6 {
address 2001:dead:beef:200::22/64;
}
}
}
}
routing-options {
rib inet6.0 {
static {
route ::/0 next-hop 2001:dead:beef:200::a;
}
}
static {
route 0.0.0.0/0 next-hop 172.16.1.254;
}
}
}
I created 2 logical interfaces on the CE aggregate interface, and placed one of them in logical system – this will allow for more convenient testing of IRB functionality. I plan to test both IPv4 and IPv6 forwarding, so there is configuration for both address families.
So let’s start MC-LAG configuration. First of all, don’t forget switch-options config:
switch-options {
service-id 1;
}
service-id 1;
}
This ID should be identical on both MC-LAG boxes. Little knob that can cause a lot of headache.
Next is the InterChassis Link (ICL) configuration.
QFX style:
xe-0/0/3 {
description ICL;
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members all;
}
}
}
}
description ICL;
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members all;
}
}
}
}
And MX style:
ge-0/0/3 {
description vMX2;
gigether-options {
802.3ad ae0;
}
}
ae0 {
description ICL;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
}
}
unit 0 {
family bridge {
interface-mode trunk;
vlan-id-list 1-999;
}
}
unit 1 {
vlan-id 1000;
family inet {
address 192.168.12.0/31;
}
}
}
description vMX2;
gigether-options {
802.3ad ae0;
}
}
ae0 {
description ICL;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
}
}
unit 0 {
family bridge {
interface-mode trunk;
vlan-id-list 1-999;
}
}
unit 1 {
vlan-id 1000;
family inet {
address 192.168.12.0/31;
}
}
}
I configured single ICL interface on the QFX and aggregate interface on the MX just to show you the difference. It’s recommended to use at least two interfaces for the ICL link, so just use aggregate interface.
Next step is the InterChassis Control Protocol (ICCP) configuration. You need just plain IP connectivity between MC-LAG boxes to establish ICCP session. You can use existing IP connectivity through IP fabric (I use that for QFX MC-LAG), or you can create L3 link via ICL and use that connection (I show that for MX MC-LAG).
So there is vQFX ICCP configuration using loopback interface addresses:
protocols {
iccp {
local-ip-addr 11.11.11.11;
peer 22.22.22.22 {
redundancy-group-id-list 1;
liveness-detection {
minimum-interval 10000;
multiplier 3;
}
}
}
}
iccp {
local-ip-addr 11.11.11.11;
peer 22.22.22.22 {
redundancy-group-id-list 1;
liveness-detection {
minimum-interval 10000;
multiplier 3;
}
}
}
}
And this is vMX ICCP config using ICL L3 link:
protocols {
iccp {
local-ip-addr 192.168.12.0;
peer 192.168.12.1 {
redundancy-group-id-list 1;
liveness-detection {
minimum-interval 10000;
multiplier 3;
}
}
}
}
iccp {
local-ip-addr 192.168.12.0;
peer 192.168.12.1 {
redundancy-group-id-list 1;
liveness-detection {
minimum-interval 10000;
multiplier 3;
}
}
}
}
Final step is the actual MC-LAG interface configuration.
QFX style:
interfaces {
xe-0/0/2 {
description MCLAG;
ether-options {
802.3ad ae0;
}
}
ae0 {
description MCLAG;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:11:11:11;
admin-key 1;
}
mc-ae {
mc-ae-id 1;
chassis-id 0;
mode active-active;
status-control active;
init-delay-time 15;
}
}
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ v100 v200 ];
}
}
}
}
}
xe-0/0/2 {
description MCLAG;
ether-options {
802.3ad ae0;
}
}
ae0 {
description MCLAG;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:11:11:11;
admin-key 1;
}
mc-ae {
mc-ae-id 1;
chassis-id 0;
mode active-active;
status-control active;
init-delay-time 15;
}
}
unit 0 {
family ethernet-switching {
interface-mode trunk;
vlan {
members [ v100 v200 ];
}
}
}
}
}
And MX style:
interfaces {
ge-0/0/0 {
description MX1;
gigether-options {
802.3ad ae1;
}
}
ae1 {
flexible-vlan-tagging;
multi-chassis-protection 192.168.12.1 {
interface ae0;
}
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:00:00:01;
admin-key 1;
}
mc-ae {
mc-ae-id 1;
redundancy-group 1;
chassis-id 0;
mode active-active;
status-control active;
}
}
unit 0 {
family bridge {
interface-mode trunk;
vlan-id-list [ 100 200 ];
}
}
}
}
ge-0/0/0 {
description MX1;
gigether-options {
802.3ad ae1;
}
}
ae1 {
flexible-vlan-tagging;
multi-chassis-protection 192.168.12.1 {
interface ae0;
}
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
active;
periodic slow;
system-id 00:00:00:00:00:01;
admin-key 1;
}
mc-ae {
mc-ae-id 1;
redundancy-group 1;
chassis-id 0;
mode active-active;
status-control active;
}
}
unit 0 {
family bridge {
interface-mode trunk;
vlan-id-list [ 100 200 ];
}
}
}
}
I showed config for primary boxes in MC-LAG Active-Active pairs – for standby boxes just configure chassis-id 1 and status-control standby, the rest of config is completely the same.
One important thing to notice here is the lack of multi-chassis-protection in QFX interface config – this is because QFX doesn’t support this option at the interface level – you need to configure this option under global multi-chassis stanza:
multi-chassis {
multi-chassis-protection 22.22.22.22 {
interface xe-0/0/3;
}
}
multi-chassis-protection 22.22.22.22 {
interface xe-0/0/3;
}
}
Also notice the lack of redundancy-group ID configuration for QFX – this config knob doesn’t exists on this platform, so I think this just means that all MC-LAG interfaces participate in one default redundancy group.
And finally let’s configure vlans/bridge-domains and basic IRB interfaces to test connectivity.
QFX:
interfaces {
irb {
unit 100 {
family inet {
address 172.16.0.254/24;
}
family inet6 {
address 2001:dead:beef:100::a/64;
}
}
unit 200 {
family inet {
address 172.16.1.254/24;
}
family inet6 {
address 2001:dead:beef:200::a/64;
}
}
}
}
vlans {
v100 {
vlan-id 100;
l3-interface irb.100;
mcae-mac-synchronize;
}
v200 {
vlan-id 200;
l3-interface irb.200;
mcae-mac-synchronize;
}
}
irb {
unit 100 {
family inet {
address 172.16.0.254/24;
}
family inet6 {
address 2001:dead:beef:100::a/64;
}
}
unit 200 {
family inet {
address 172.16.1.254/24;
}
family inet6 {
address 2001:dead:beef:200::a/64;
}
}
}
}
vlans {
v100 {
vlan-id 100;
l3-interface irb.100;
mcae-mac-synchronize;
}
v200 {
vlan-id 200;
l3-interface irb.200;
mcae-mac-synchronize;
}
}
MX:
interfaces {
irb {
unit 100 {
family inet {
address 172.16.111.254/24;
}
family inet6 {
address 2001:dead:beef:111::a/64;
}
}
unit 200 {
family inet {
address 172.16.112.254/24;
}
family inet6 {
address 2001:dead:beef:112::a/64;
}
}
}
}
bridge-domains {
bd100 {
vlan-id 100;
mcae-mac-synchronize;
routing-interface irb.100;
}
bd200 {
vlan-id 200;
mcae-mac-synchronize;
routing-interface irb.200;
}
}
irb {
unit 100 {
family inet {
address 172.16.111.254/24;
}
family inet6 {
address 2001:dead:beef:111::a/64;
}
}
unit 200 {
family inet {
address 172.16.112.254/24;
}
family inet6 {
address 2001:dead:beef:112::a/64;
}
}
}
}
bridge-domains {
bd100 {
vlan-id 100;
mcae-mac-synchronize;
routing-interface irb.100;
}
bd200 {
vlan-id 200;
mcae-mac-synchronize;
routing-interface irb.200;
}
}
For simplified Layer 3 gateway services, where Layer 3 routing protocols are not run on the MC-LAG peers, you simply configure the same Layer 3 gateway IP address on both MC-LAG peers and enable IRB MAC address synchronization. MAC address synchronization requires you to configure the same IP address on the IRB interface in the VLAN on both MC-LAG peers.
So we are finally done with configuration, now it’s time to check that all works as expected.
First I show you all commands for MX MC-LAG, and later repeat all of this for QFX pair of devices.
ICCP status:
alex@vMX1> show iccp
Redundancy Group Information for peer 192.168.12.1
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
alex@vMX2> show iccp
Redundancy Group Information for peer 192.168.12.0
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
Client Application: lacpd
Redundancy Group IDs Joined: 1
alex@vMX1> show bfd session extensive
Detect Transmit
Address State Interface Time Interval Multiplier
192.168.12.1 Up 30.000 10.000 3
Client ICCP realm 192.168.12.1, TX interval 10.000, RX interval 10.000
alex@vMX2> show bfd session extensive
Detect Transmit
Address State Interface Time Interval Multiplier
192.168.12.0 Up 30.000 10.000 3
Client ICCP realm 192.168.12.0, TX interval 10.000, RX interval 10.000
Redundancy Group Information for peer 192.168.12.1
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
alex@vMX2> show iccp
Redundancy Group Information for peer 192.168.12.0
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
Client Application: lacpd
Redundancy Group IDs Joined: 1
alex@vMX1> show bfd session extensive
Detect Transmit
Address State Interface Time Interval Multiplier
192.168.12.1 Up 30.000 10.000 3
Client ICCP realm 192.168.12.1, TX interval 10.000, RX interval 10.000
alex@vMX2> show bfd session extensive
Detect Transmit
Address State Interface Time Interval Multiplier
192.168.12.0 Up 30.000 10.000 3
Client ICCP realm 192.168.12.0, TX interval 10.000, RX interval 10.000
ICCP sessions established, BFD sessions are Up and clearly shows ICCP as a client protocol (don’t mind huge BFD timers – it’s just to reduce the load on my PC).
MC-AE interface status:
alex@vMX1> show interfaces mc-ae extensive
Member Link : ae1
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 192.168.12.1 ae0.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : active
Chassis ID : 0
LACP Configuration
System ID : 00:00:00:00:00:01
Admin Key : 1
alex@vMX2> show interfaces mc-ae extensive
Member Link : ae1
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 192.168.12.0 ae0.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : standby
Chassis ID : 1
LACP Configuration
System ID : 00:00:00:00:00:01
Admin Key : 1
Member Link : ae1
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 192.168.12.1 ae0.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : active
Chassis ID : 0
LACP Configuration
System ID : 00:00:00:00:00:01
Admin Key : 1
alex@vMX2> show interfaces mc-ae extensive
Member Link : ae1
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 192.168.12.0 ae0.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : standby
Chassis ID : 1
LACP Configuration
System ID : 00:00:00:00:00:01
Admin Key : 1
Both MC-LAG interfaces are Up and Active. Note ICL protection status, and also LACP configuration in extensive output.
LACP interfaces status:
alex@vMX1> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/3 Current Slow periodic Collecting distributing
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
alex@vMX2> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/3 Current Slow periodic Collecting distributing
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/3 Current Slow periodic Collecting distributing
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
alex@vMX2> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/3 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/3 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/3 Current Slow periodic Collecting distributing
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
LACP status on the CE device:
alex@MX1> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
ge-0/0/1 Current Slow periodic Collecting distributing
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
ge-0/0/1 Current Slow periodic Collecting distributing
Both interfaces on CE are Up and forwarding traffic.
Let’s ping gateway addresses from CE:
alex@MX1> ping 172.16.111.254 count 3
PING 172.16.111.254 (172.16.111.254): 56 data bytes
64 bytes from 172.16.111.254: icmp_seq=0 ttl=64 time=10.082 ms
64 bytes from 172.16.111.254: icmp_seq=1 ttl=64 time=2.723 ms
64 bytes from 172.16.111.254: icmp_seq=2 ttl=64 time=36.738 ms
--- 172.16.111.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.723/16.514/36.738/14.612 ms
alex@MX1> ping 172.16.112.254 count 3 logical-system second
PING 172.16.112.254 (172.16.112.254): 56 data bytes
64 bytes from 172.16.112.254: icmp_seq=0 ttl=64 time=10.150 ms
64 bytes from 172.16.112.254: icmp_seq=1 ttl=64 time=6.360 ms
64 bytes from 172.16.112.254: icmp_seq=2 ttl=64 time=23.189 ms
--- 172.16.112.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 6.360/13.233/23.189/7.208 ms
alex@MX1> ping 2001:dead:beef:111::a count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:111::a
16 bytes from 2001:dead:beef:111::a, icmp_seq=0 hlim=64 time=13.770 ms
16 bytes from 2001:dead:beef:111::a, icmp_seq=1 hlim=64 time=2.581 ms
16 bytes from 2001:dead:beef:111::a, icmp_seq=2 hlim=64 time=4.750 ms
--- 2001:dead:beef:111::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 2.581/7.034/13.770/4.845 ms
alex@MX1> ping 2001:dead:beef:112::a count 3 logical-system second
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::a
16 bytes from 2001:dead:beef:112::a, icmp_seq=0 hlim=64 time=60.422 ms
16 bytes from 2001:dead:beef:112::a, icmp_seq=1 hlim=64 time=60.089 ms
16 bytes from 2001:dead:beef:112::a, icmp_seq=2 hlim=64 time=2.263 ms
--- 2001:dead:beef:112::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 2.263/40.925/60.422/27.338 ms
PING 172.16.111.254 (172.16.111.254): 56 data bytes
64 bytes from 172.16.111.254: icmp_seq=0 ttl=64 time=10.082 ms
64 bytes from 172.16.111.254: icmp_seq=1 ttl=64 time=2.723 ms
64 bytes from 172.16.111.254: icmp_seq=2 ttl=64 time=36.738 ms
--- 172.16.111.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.723/16.514/36.738/14.612 ms
alex@MX1> ping 172.16.112.254 count 3 logical-system second
PING 172.16.112.254 (172.16.112.254): 56 data bytes
64 bytes from 172.16.112.254: icmp_seq=0 ttl=64 time=10.150 ms
64 bytes from 172.16.112.254: icmp_seq=1 ttl=64 time=6.360 ms
64 bytes from 172.16.112.254: icmp_seq=2 ttl=64 time=23.189 ms
--- 172.16.112.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 6.360/13.233/23.189/7.208 ms
alex@MX1> ping 2001:dead:beef:111::a count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:111::a
16 bytes from 2001:dead:beef:111::a, icmp_seq=0 hlim=64 time=13.770 ms
16 bytes from 2001:dead:beef:111::a, icmp_seq=1 hlim=64 time=2.581 ms
16 bytes from 2001:dead:beef:111::a, icmp_seq=2 hlim=64 time=4.750 ms
--- 2001:dead:beef:111::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 2.581/7.034/13.770/4.845 ms
alex@MX1> ping 2001:dead:beef:112::a count 3 logical-system second
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::a
16 bytes from 2001:dead:beef:112::a, icmp_seq=0 hlim=64 time=60.422 ms
16 bytes from 2001:dead:beef:112::a, icmp_seq=1 hlim=64 time=60.089 ms
16 bytes from 2001:dead:beef:112::a, icmp_seq=2 hlim=64 time=2.263 ms
--- 2001:dead:beef:112::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 2.263/40.925/60.422/27.338 ms
CE can ping IP addresses of MX’s IRB in both Vlans, IPv4 and IPv6.
And show MAC table on MC-LAG devices:
alex@vMX1> show bridge mac-table
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd100, VLAN : 100
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DLR ae1.0
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd200, VLAN : 200
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DLR ae1.0
alex@vMX2> show bridge mac-table
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd100, VLAN : 100
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DL ae1.0
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd200, VLAN : 200
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DL ae1.0
alex@vMX1> show l2-learning redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 192.168.12.1 100 00:05:86:71:8c:c0 1 0 1 0 Installed
1 192.168.12.1 200 00:05:86:71:8c:c0 1 0 1 0 Installed
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd100, VLAN : 100
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DLR ae1.0
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd200, VLAN : 200
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DLR ae1.0
alex@vMX2> show bridge mac-table
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd100, VLAN : 100
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DL ae1.0
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
O -OVSDB MAC, SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC, P -Pinned MAC)
Routing instance : default-switch
Bridging domain : bd200, VLAN : 200
MAC MAC Logical NH MAC
address flags interface Index property
00:05:86:71:8c:c0 DL ae1.0
alex@vMX1> show l2-learning redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 192.168.12.1 100 00:05:86:71:8c:c0 1 0 1 0 Installed
1 192.168.12.1 200 00:05:86:71:8c:c0 1 0 1 0 Installed
vMX2 has locally learned MAC addresses of CE in both bridge-domains. vMX1 learned this MACs remotely, but because it has local interfaces in the same LAG, it shows DLR flags for this addresses.
And finally let’s ping between CE interfaces in different Vlans to check IRB functionality:
alex@MX1> ping 172.16.112.11 source 172.16.111.11 count 3
PING 172.16.112.11 (172.16.112.11): 56 data bytes
64 bytes from 172.16.112.11: icmp_seq=0 ttl=63 time=45.707 ms
64 bytes from 172.16.112.11: icmp_seq=1 ttl=63 time=5.573 ms
64 bytes from 172.16.112.11: icmp_seq=2 ttl=63 time=14.868 ms
--- 172.16.112.11 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 5.573/22.049/45.707/17.153 ms
alex@MX1> ...::11 source 2001:dead:beef:111::11 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::11
16 bytes from 2001:dead:beef:112::11, icmp_seq=0 hlim=63 time=198.725 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=1 hlim=63 time=365.604 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=2 hlim=63 time=14.449 ms
--- 2001:dead:beef:112::11 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 14.449/192.926/365.604/143.417 ms
alex@MX1> ...::11 source 2001:dead:beef:111::11 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::11
16 bytes from 2001:dead:beef:112::11, icmp_seq=0 hlim=63 time=198.725 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=1 hlim=63 time=365.604 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=2 hlim=63 time=14.449 ms
--- 2001:dead:beef:112::11 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 14.449/192.926/365.604/143.417 ms
alex@MX1> traceroute 172.16.112.11
traceroute to 172.16.112.11 (172.16.112.11), 30 hops max, 40 byte packets
1 172.16.111.254 (172.16.111.254) 3270.836 ms 3693.345 ms 338.158 ms
2 172.16.112.11 (172.16.112.11) 41.231 ms 63.477 ms 37.072 ms
alex@MX1> traceroute 2001:dead:beef:112::11
traceroute6 to 2001:dead:beef:112::11 (2001:dead:beef:112::11) from 2001:dead:beef:111::11, 64 hops max, 12 byte packets
1 2001:dead:beef:111::a (2001:dead:beef:111::a) 2797.591 ms 885.998 ms 8.219 ms
2 2001:dead:beef:112::11 (2001:dead:beef:112::11) 35.385 ms 105.196 ms 56.986 ms
PING 172.16.112.11 (172.16.112.11): 56 data bytes
64 bytes from 172.16.112.11: icmp_seq=0 ttl=63 time=45.707 ms
64 bytes from 172.16.112.11: icmp_seq=1 ttl=63 time=5.573 ms
64 bytes from 172.16.112.11: icmp_seq=2 ttl=63 time=14.868 ms
--- 172.16.112.11 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 5.573/22.049/45.707/17.153 ms
alex@MX1> ...::11 source 2001:dead:beef:111::11 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::11
16 bytes from 2001:dead:beef:112::11, icmp_seq=0 hlim=63 time=198.725 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=1 hlim=63 time=365.604 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=2 hlim=63 time=14.449 ms
--- 2001:dead:beef:112::11 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 14.449/192.926/365.604/143.417 ms
alex@MX1> ...::11 source 2001:dead:beef:111::11 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:111::11 --> 2001:dead:beef:112::11
16 bytes from 2001:dead:beef:112::11, icmp_seq=0 hlim=63 time=198.725 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=1 hlim=63 time=365.604 ms
16 bytes from 2001:dead:beef:112::11, icmp_seq=2 hlim=63 time=14.449 ms
--- 2001:dead:beef:112::11 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 14.449/192.926/365.604/143.417 ms
alex@MX1> traceroute 172.16.112.11
traceroute to 172.16.112.11 (172.16.112.11), 30 hops max, 40 byte packets
1 172.16.111.254 (172.16.111.254) 3270.836 ms 3693.345 ms 338.158 ms
2 172.16.112.11 (172.16.112.11) 41.231 ms 63.477 ms 37.072 ms
alex@MX1> traceroute 2001:dead:beef:112::11
traceroute6 to 2001:dead:beef:112::11 (2001:dead:beef:112::11) from 2001:dead:beef:111::11, 64 hops max, 12 byte packets
1 2001:dead:beef:111::a (2001:dead:beef:111::a) 2797.591 ms 885.998 ms 8.219 ms
2 2001:dead:beef:112::11 (2001:dead:beef:112::11) 35.385 ms 105.196 ms 56.986 ms
Everything works just fine, CE interfaces in different Vlans have reachability to each other. Traceroute output clearly shows that traffic goes via IRB interfaces on MX.
Below are the show commands output from QFX devices for your reference.
ICCP status:
alex@vQFX1> show iccp
Redundancy Group Information for peer 22.22.22.22
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: MCSNOOPD
Redundancy Group IDs Joined: None
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
alex@vQFX2> show iccp
Redundancy Group Information for peer 11.11.11.11
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: MCSNOOPD
Redundancy Group IDs Joined: None
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
Redundancy Group Information for peer 22.22.22.22
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: MCSNOOPD
Redundancy Group IDs Joined: None
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
alex@vQFX2> show iccp
Redundancy Group Information for peer 11.11.11.11
TCP Connection : Established
Liveliness Detection : Up
Redundancy Group ID Status
1 Up
Client Application: lacpd
Redundancy Group IDs Joined: 1
Client Application: MCSNOOPD
Redundancy Group IDs Joined: None
Client Application: l2ald_iccpd_client
Redundancy Group IDs Joined: 1
MC-AE interface status:
alex@vQFX1> show interfaces mc-ae extensive
Member Link : ae0
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae0.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 22.22.22.22 xe-0/0/3.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : active
Chassis ID : 0
LACP Configuration
System ID : 00:00:00:11:11:11
Admin Key : 1
alex@vQFX2> show interfaces mc-ae extensive
Member Link : ae0
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae0.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 11.11.11.11 xe-0/0/3.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : standby
Chassis ID : 1
LACP Configuration
System ID : 00:00:00:11:11:11
Admin Key : 1
Member Link : ae0
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae0.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 22.22.22.22 xe-0/0/3.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : active
Chassis ID : 0
LACP Configuration
System ID : 00:00:00:11:11:11
Admin Key : 1
alex@vQFX2> show interfaces mc-ae extensive
Member Link : ae0
Current State Machine State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae0.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 11.11.11.11 xe-0/0/3.0 up
MCAE Configuration
Redundancy Group : 1
MCAE ID : 1
MCAE Mode : active_active
Status Control : standby
Chassis ID : 1
LACP Configuration
System ID : 00:00:00:11:11:11
Admin Key : 1
LACP interfaces status:
alex@vQFX1> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
xe-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/2 Current Slow periodic Collecting distributing
alex@vQFX2> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
xe-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/2 Current Slow periodic Collecting distributing
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
xe-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/2 Current Slow periodic Collecting distributing
alex@vQFX2> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/2 Actor No No Yes Yes Yes Yes Slow Active
xe-0/0/2 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/2 Current Slow periodic Collecting distributing
LACP status on the CE device:
alex@MX2> show lacp interfaces
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
ge-0/0/1 Current Slow periodic Collecting distributing
Aggregated interface: ae0
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
ge-0/0/0 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/0 Partner No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Actor No No Yes Yes Yes Yes Slow Active
ge-0/0/1 Partner No No Yes Yes Yes Yes Slow Active
LACP protocol: Receive State Transmit State Mux State
ge-0/0/0 Current Slow periodic Collecting distributing
ge-0/0/1 Current Slow periodic Collecting distributing
Ping gateway addresses from CE:
alex@MX2> ping 172.16.0.254 count 3
PING 172.16.0.254 (172.16.0.254): 56 data bytes
64 bytes from 172.16.0.254: icmp_seq=0 ttl=64 time=348.026 ms
64 bytes from 172.16.0.254: icmp_seq=1 ttl=64 time=371.542 ms
64 bytes from 172.16.0.254: icmp_seq=2 ttl=64 time=545.497 ms
--- 172.16.0.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 348.026/421.688/545.497/88.071 ms
alex@MX2> ping 2001:dead:beef:100::a count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:100::22 --> 2001:dead:beef:100::a
16 bytes from 2001:dead:beef:100::a, icmp_seq=0 hlim=64 time=226.809 ms
16 bytes from 2001:dead:beef:100::a, icmp_seq=1 hlim=64 time=220.764 ms
16 bytes from 2001:dead:beef:100::a, icmp_seq=2 hlim=64 time=181.753 ms
--- 2001:dead:beef:100::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 181.753/209.775/226.809/19.968 ms
alex@MX2> ping 172.16.1.254 count 3 logical-system second
PING 172.16.1.254 (172.16.1.254): 56 data bytes
64 bytes from 172.16.1.254: icmp_seq=0 ttl=64 time=95.561 ms
64 bytes from 172.16.1.254: icmp_seq=1 ttl=64 time=1018.245 ms
64 bytes from 172.16.1.254: icmp_seq=2 ttl=64 time=110.563 ms
--- 172.16.1.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 95.561/408.123/1018.245/431.465 ms
alex@MX2> ping 2001:dead:beef:200::a count 3 logical-system second
PING6(56=40+8+8 bytes) 2001:dead:beef:200::22 --> 2001:dead:beef:200::a
16 bytes from 2001:dead:beef:200::a, icmp_seq=0 hlim=64 time=201.916 ms
16 bytes from 2001:dead:beef:200::a, icmp_seq=1 hlim=64 time=211.778 ms
16 bytes from 2001:dead:beef:200::a, icmp_seq=2 hlim=64 time=448.803 ms
--- 2001:dead:beef:200::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 201.916/287.499/448.803/114.130 ms
PING 172.16.0.254 (172.16.0.254): 56 data bytes
64 bytes from 172.16.0.254: icmp_seq=0 ttl=64 time=348.026 ms
64 bytes from 172.16.0.254: icmp_seq=1 ttl=64 time=371.542 ms
64 bytes from 172.16.0.254: icmp_seq=2 ttl=64 time=545.497 ms
--- 172.16.0.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 348.026/421.688/545.497/88.071 ms
alex@MX2> ping 2001:dead:beef:100::a count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:100::22 --> 2001:dead:beef:100::a
16 bytes from 2001:dead:beef:100::a, icmp_seq=0 hlim=64 time=226.809 ms
16 bytes from 2001:dead:beef:100::a, icmp_seq=1 hlim=64 time=220.764 ms
16 bytes from 2001:dead:beef:100::a, icmp_seq=2 hlim=64 time=181.753 ms
--- 2001:dead:beef:100::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 181.753/209.775/226.809/19.968 ms
alex@MX2> ping 172.16.1.254 count 3 logical-system second
PING 172.16.1.254 (172.16.1.254): 56 data bytes
64 bytes from 172.16.1.254: icmp_seq=0 ttl=64 time=95.561 ms
64 bytes from 172.16.1.254: icmp_seq=1 ttl=64 time=1018.245 ms
64 bytes from 172.16.1.254: icmp_seq=2 ttl=64 time=110.563 ms
--- 172.16.1.254 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 95.561/408.123/1018.245/431.465 ms
alex@MX2> ping 2001:dead:beef:200::a count 3 logical-system second
PING6(56=40+8+8 bytes) 2001:dead:beef:200::22 --> 2001:dead:beef:200::a
16 bytes from 2001:dead:beef:200::a, icmp_seq=0 hlim=64 time=201.916 ms
16 bytes from 2001:dead:beef:200::a, icmp_seq=1 hlim=64 time=211.778 ms
16 bytes from 2001:dead:beef:200::a, icmp_seq=2 hlim=64 time=448.803 ms
--- 2001:dead:beef:200::a ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 201.916/287.499/448.803/114.130 ms
MAC table on MC-LAG devices:
alex@vQFX1> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical NH RTR
name address flags interface Index ID
v100 00:05:86:71:c6:c0 DLR - ae0.0 0 0
v200 00:05:86:71:c6:c0 DLR - ae0.0 0 0
alex@vQFX2> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical NH RTR
name address flags interface Index ID
v100 00:05:86:71:c6:c0 DLR - ae0.0 0 0
v200 00:05:86:71:c6:c0 DLR - ae0.0 0 0
alex@vQFX1> show ethernet-switching redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 22.22.22.22 100 00:05:86:71:c6:c0 1 0 1 0 Installed
1 22.22.22.22 200 00:05:86:71:c6:c0 1 0 1 0 Installed
alex@vQFX2> show ethernet-switching redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 11.11.11.11 100 00:05:86:71:c6:c0 1 0 1 0 Installed
1 11.11.11.11 200 00:05:86:71:c6:c0 1 0 1 0 Installed
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical NH RTR
name address flags interface Index ID
v100 00:05:86:71:c6:c0 DLR - ae0.0 0 0
v200 00:05:86:71:c6:c0 DLR - ae0.0 0 0
alex@vQFX2> show ethernet-switching table
MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static, C - Control MAC
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 2 entries, 2 learned
Routing instance : default-switch
Vlan MAC MAC Age Logical NH RTR
name address flags interface Index ID
v100 00:05:86:71:c6:c0 DLR - ae0.0 0 0
v200 00:05:86:71:c6:c0 DLR - ae0.0 0 0
alex@vQFX1> show ethernet-switching redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 22.22.22.22 100 00:05:86:71:c6:c0 1 0 1 0 Installed
1 22.22.22.22 200 00:05:86:71:c6:c0 1 0 1 0 Installed
alex@vQFX2> show ethernet-switching redundancy-groups remote-macs
Redundancy Group ID : 1 Flags : Local Connect,Remote Connect
Service-id Peer-Addr VLAN MAC MCAE-ID Subunit Opcode Flags Status
1 11.11.11.11 100 00:05:86:71:c6:c0 1 0 1 0 Installed
1 11.11.11.11 200 00:05:86:71:c6:c0 1 0 1 0 Installed
Ping between CE interfaces in different Vlans:
alex@MX2> ping 172.16.1.22 source 172.16.0.22 count 3
PING 172.16.1.22 (172.16.1.22): 56 data bytes
64 bytes from 172.16.1.22: icmp_seq=0 ttl=63 time=35.906 ms
64 bytes from 172.16.1.22: icmp_seq=1 ttl=63 time=74.570 ms
64 bytes from 172.16.1.22: icmp_seq=2 ttl=63 time=61.460 ms
--- 172.16.1.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 35.906/57.312/74.570/16.055 ms
alex@MX2> ...::22 source 2001:dead:beef:100::22 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:100::22 --> 2001:dead:beef:200::22
16 bytes from 2001:dead:beef:200::22, icmp_seq=0 hlim=63 time=332.601 ms
16 bytes from 2001:dead:beef:200::22, icmp_seq=1 hlim=63 time=325.890 ms
16 bytes from 2001:dead:beef:200::22, icmp_seq=2 hlim=63 time=341.242 ms
--- 2001:dead:beef:200::22 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 325.890/333.244/341.242/6.284 ms
alex@MX2> traceroute 172.16.1.22 source 172.16.0.22
traceroute to 172.16.1.22 (172.16.1.22) from 172.16.0.22, 30 hops max, 40 byte packets
1 172.16.0.254 (172.16.0.254) 498.274 ms 69.079 ms 71.222 ms
2 172.16.1.22 (172.16.1.22) 52.327 ms 37.718 ms 24.944 ms
alex@MX2> traceroute 2001:dead:beef:200::22 source 2001:dead:beef:100::22
traceroute6 to 2001:dead:beef:200::22 (2001:dead:beef:200::22) from 2001:dead:beef:100::22, 64 hops max, 12 byte packets
1 2001:dead:beef:100::a (2001:dead:beef:100::a) 18.501 ms 16.012 ms 14.699 ms
2 2001:dead:beef:200::22 (2001:dead:beef:200::22) 17.125 ms 17.940 ms 15.989 ms
PING 172.16.1.22 (172.16.1.22): 56 data bytes
64 bytes from 172.16.1.22: icmp_seq=0 ttl=63 time=35.906 ms
64 bytes from 172.16.1.22: icmp_seq=1 ttl=63 time=74.570 ms
64 bytes from 172.16.1.22: icmp_seq=2 ttl=63 time=61.460 ms
--- 172.16.1.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 35.906/57.312/74.570/16.055 ms
alex@MX2> ...::22 source 2001:dead:beef:100::22 count 3
PING6(56=40+8+8 bytes) 2001:dead:beef:100::22 --> 2001:dead:beef:200::22
16 bytes from 2001:dead:beef:200::22, icmp_seq=0 hlim=63 time=332.601 ms
16 bytes from 2001:dead:beef:200::22, icmp_seq=1 hlim=63 time=325.890 ms
16 bytes from 2001:dead:beef:200::22, icmp_seq=2 hlim=63 time=341.242 ms
--- 2001:dead:beef:200::22 ping6 statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/std-dev = 325.890/333.244/341.242/6.284 ms
alex@MX2> traceroute 172.16.1.22 source 172.16.0.22
traceroute to 172.16.1.22 (172.16.1.22) from 172.16.0.22, 30 hops max, 40 byte packets
1 172.16.0.254 (172.16.0.254) 498.274 ms 69.079 ms 71.222 ms
2 172.16.1.22 (172.16.1.22) 52.327 ms 37.718 ms 24.944 ms
alex@MX2> traceroute 2001:dead:beef:200::22 source 2001:dead:beef:100::22
traceroute6 to 2001:dead:beef:200::22 (2001:dead:beef:200::22) from 2001:dead:beef:100::22, 64 hops max, 12 byte packets
1 2001:dead:beef:100::a (2001:dead:beef:100::a) 18.501 ms 16.012 ms 14.699 ms
2 2001:dead:beef:200::22 (2001:dead:beef:200::22) 17.125 ms 17.940 ms 15.989 ms
So this is basic L2 switching and IRB functionality for MC-LAG. With this type of IRB configuration (with mcae-mac-synchronize) you cannot use any routing protocols between IRB and CE. For more advanced Layer 3 gateway services you need to configure unique IP addresses on IRB interfaces on each MC-LAG peer. This is theme for the next post.
2 thoughts on “MC-LAG lab – basic L2 connectivity”
Great article – thanks. This article really helped me with configuration QFX5100 + MC-LAG.
But I have 1 problem. You set static ARP, but there is still no irb’s MAC in the ethernet table on ICL link.
>but there is still no irb’s MAC in the ethernet table on ICL link < This is the reason that you need to configure static ARP entry - remote IRB's MAC not learned automatically via ICL link. In this static ARP entry you need to configure MAC and L2 interface for remote IRB's IP (more in the next post).