Always Know Where the Default Gateway Resides

The other day I was assigned the task of replacing two switches on our wireless network. Our wireless network is completely separate from our wired network and we typically set up our network in this style of topology:

We use a backbone area that is a flat Layer 2 topology to the wireless controller.  We use Layer 3 switches at the access layer to create a separate VLAN and IP address space for the wireless access points.  The access layer switches run OSPF and include a default route.  The route to the access points is advertised via OSPF.

However, in the building that I was replacing switches in the topology was closer to this:

VLAN 44 used Layer 2 back to the core Layer 3 switch while VLAN 55 used a Layer 3 switch like our usual design.  At least, that's what I thought.  As we'll see, the two assumptions I made carried some consequences.

-

I've duplicated the behavior I witnessed on my home lab equipment.  The actual topology where I work was larger but this set up will show what I ran into.  I used 2) 3550s, 2) 2950s and 2) 1721s.

The second 1721 router in the lower left corner is being used as an end-user device in order to keep the VLAN in an 'up' state.

-

VLAN 11 is used as a backbone VLAN for the switches to connect back to the R1 router.

VLAN 22 is used for the access point VLAN, but I made the assumption that R1 was the default gateway.  I was tasked with replacing SW1 and SW3 with newer models.  I configured the SW1 to behave per our usual design (only the backbone VLAN is allowed out of the switch and the access point VLAN stays local to the switch and OSPF is used to advertise the access point network).  SW3 was unique that it used two VLANs for the access points [I've only set up one on this topology].  I never looked at the config on SW2 before replacing the switches.

VLAN 33 is a different backbone VLAN that is used only between R1 and Dist-Switch.

I replaced SW3 and all of the connected access points came back online and everything was working as expected.

I replaced SW1 and those access points came online.  However, the access points connected on SW2 went offline!  But the access points on SW3 were still online.

So, I logged into SW2 and tried pinging the the VLAN 22 gateway of 22.22.22.1:

And the pings failed, which surprised me since I assumed that R1 was the gateway.

I logged into SW3 and tried the same thing:

And the pings succeeded!  At this point I was really confused.  How could pings fail from one switch but succeed from another?

After much digging around (nearly an hour!), I was able to get a hold of the engineer and he found some documentation that said that SW1 was actually the gateway for VLAN 22!

When I installed SW2, I configured the switch to only allow VLAN 11 (#switchport trunk allowed vlan 11).  This configuration cut off Layer 2 access to the VLAN 22 gateway.

I added VLAN 22 to the trunk and the rest of the access points immediately came back online and service was restored.

---

But how were the access points on SW3 able to reach the VLAN 22 gateway?

I was able to replicate the configurations and recreate the problem here:

On R1, we have a router-on-a-stick configuration with the three VLANs.

Note that fa0.22 is not .1 (the usual default gateway).

-

Dist-Switch is a pure Layer 2 switch (no routing).  It uses an SVI on VLAN 33 and uses R1's fa 0.3's subinterface as the default gateway.

-

SW1 is still configured incorrectly (only VLAN 11 is allowed on the trunk link to Dist-Switch).  SW1 is running OSPF with R1 and uses VLAN 11 to reach R1.  Access point VLAN 22 is advertised to R1 and we'll see that plays a big role in the weird behavior.

-

SW2 is configured as a Layer 2 switch (even through it is a 3550).  This switch uses VLAN 22 (the access point VLAN) as its' default gateway.  At this point, 22.22.22.1 (located on SW1) is not accessible at Layer 2 since VLAN 22 is not allowed on the trunk link between SW1 and Dist-Switch.

-

SW3 is a Layer 2 only switch as well.  However, it was configured to use VLAN 11 (the backbone VLAn) and 11.11.11.1 (R1's fa 0.1 subinterface) as its' default gateway.

The differences in the default-gateway configuration was the reason why SW2's access points lost connectivity, but SW3's access points remained online.

-

On SW2, VLAN 22 and 22.22.22.1 were the exit point out of the VLAN.  Since I configured SW1 to not allow VLAN 22 on the trunk, 22.22.22.1 was not available from a Layer 2 perspective.

On SW3, it was using VLAN 11 (the backbone VLAN) and was able to reach R1.  When I sent pings from SW3 to 22.22.22.1 (SW1), the traffic went from SW3 to R1 to SW1.  

Since SW3 advertised 22.22.22.1 to R1 via OSPF on VLAN 11, R1 is able to forward the packets and receive a response from SW1.

Note that 22.22.22.0 is reached via 11.11.11.11 (SW1) on fa 0.11.

That explained why I was able to ping SW1 from SW3 (Layer 2 worked from SW3 to R1 and then the packets were sent by Layer 3 through R1 via OSPF), but not able to ping SW1 from SW2 (default-gateway was on VLAN 22 and only accessible by Layer 2 and the gateway was cut off from the rest of the VLAN because the VLAN was cut off of the trunk on the SW1 end).

--- 

The lessons learned:

Make sure that you know which device is the default gateway for each VLAN before replacing hardware or changing configurations.

If a VLAN is used by multiple devices via trunking, make sure the trunk is allowed on both ends of the link.

Ultimately, we'll have to re-engineer these access point VLANs so that the same access point VLAN is not used on multiple access switches (per our usual design).

Hey, wait. There's a tree missing here!

Yesterday, while planning a network switch replacement, I noticed a little problem that could have been a big problem if that switch was moved up one spot in the network.

I'm recreating the problem on my home lab and walking through how it was found and how to fix it:

First, here is the lab topology using dia (http://dia-installer.de/):

Straight forward physical topology.

So, I started my planning session with a quick 'show vlan brief' to verify which VLANs were running on each switch.  For this lab, we'll have VLANs 1, 11, 22, 33 and 44.

AS2

AS3

All five VLANs are configured on both switches.

The next command I issued was 'show spanning-tree summary' to verify the path back to the root switch.

AS2

Using this command we see that AS2 is the root switch for all the spanning-tree instances.

AS3

Do you see the problem?

VLAN0022 is missing from spanning-tree!

As you can see from the 'show vlan brief' command, VLAN 22 is configured on the switch, but it doesn't have a spanning-tree instance.

To verify this, you can run a 'show spanning-tree vlan 22' command:

This was the first time I've ever ran into a VLAN not running spanning-tree.

So, what is causing the problem?

I first starting by looking at the trunks.  If a VLAN does not have at least one port using the VLAN on the switch, then the VLAN is shutdown to prevent a black hole.  So, I ran a 'show interfaces trunk' command on each switch to verify each VLAN was allowed on the trunk:

AS2

No problem on AS2.  All five VLANs are allowed on the trunk.

AS3

Same deal here.  All five VLANs are allowed on the trunk.

Again, what is causing the problem?

Next, I ran a simple 'show running-config' on the problem switch (AS3 in this lab).  I'm not gonna lie, I had to run through the running-config a few times before I noticed it:

Why this command was running on a production switch, I'll never know.  But we were relieved that this configuration was on a 'spoke' switch and not on a 'hub' switch and that there was only one uplink.  If this switch was a 'hub' switch or had a redundant trunk, a broadcast storm surely would have happened.

So, the simple solution:

Ran a 'show spanning-tree summary' to confirm there is now a spanning-tree instance for VLAN 22:

If you have any questions, please leave a comment.

Thank you for reading!

Troubleshooting Trunk Links and Spanning Tree using Dia - Part 5 of 5

Welcome back to the conclusion of this series!

We are now on to VLAN 44 which will show just how useful making a spanning tree map in dia can be.

Just like on the previous posts, here is the original physical layer map:

And once again, let's start with the 'show spanning-tree vlan 44' command on DS1:

Only two of the six trunks are forwarding VLAN 44.  Gi0/1 is the root port back to the root switch and Gi0/2 is in the blocking state to prevent a loop.  First update to the map is as follows:

Next, the 'show spanning-tree vlan 44' command on DS2:

On DS2, we can see that all six trunks are forwarding VLAN 44 and that DS2 is the root switch.  The updated dia map is:

Forwarding on to AS1 and the 'show spanning-tree vlan 44' command:

Here, we can see that only two of the four links are being used for VLAN 44.  Fa0/13 is the root port back to DS2 and Fa0/14 is blocking to prevent a loop.  Of course, we document this on the map adding the circled-x on the end of the blocking port:

Last, we head over to AS2 and run the 'show spanning-tree vlan 44' command:

Now, we have some interesting information shown here.  Notice, AS2 believes it is the root switch for VLAN 44.  But we documented that DS2 is the root switch.  How is possible?

It is possible because AS2 doesn't have a connected neighbor using VLAN 44, but AS2 does have an active port on VLAN 44 (Fa0/21 and Fa0/22).  Since AS2 does not have a neighbor using VLAN 44 on their trunks (and AS2 is not using Fa0/23 or Fa0/24 on its trunk links up to DS2), AS2 believes it is the root switch for the VLAN.

Once we document this misconfiguration on the dia map, it becomes very clear.  Remember, I use the blue links to indicate that the VLAN is not on one end of the trunk link and the circled-x is used to indicate with end of the link the VLAN is not available on.

Because there are zero green links coming from AS2, it is clear that AS2 is cut off from the rest of the network on VLAN 44.

Now that Layer 2 is documented, let's verify where the HSRP Active router for the VLAN is located.  First, the 'show standby vlan 44' on DS1:

DS1 appears to the Standby router indicated by the 'State is Standby' and the 'Standby router is local' lines.  Let's verify that DS2 is indeed the Active HSRP router.

And DS2 is the Active router.  So, the finished map shows as:

---
Let's take a look at the finalized maps for each VLAN:

VLAN 1

VLAN 11

VLAN 22

VLAN 33

VLAN 44

So, now that we have our finalized maps, what would be some options to repair and optimize the current configurations?

At the beginning, I wrote that the odd numbered VLANs (1, 11 and 33) were going to use DS1 as the root switch for spanning tree and use DS1 as the HSRP Active Router and that the even numbered VLANs (22 and 44) were going to use DS2 as the root switch for spanning tree and use DS2 as the HSRP Active Router.  Additionally, when there are dual uplinks, I think it makes much more sense to utilize EtherChannels (logically combining two or more connections into a single connection).  EtherChannels optimize bandwidth (more connections toward the root switch) and simplifies the spanning tree instance (no blocking ports towards the root switch).

First off, this might be an opportune time to change the spanning tree protocol to rapid-pvst (rapid per VLAN spanning tree).

So, let's start there.  On each switch, we run the command 'spanning-tree mode rapid-pvst' in configuration mode:

When we change the spanning tree mode type, the spanning tree instance will need to reconverge and since the other switches are still in Per VLAN Spanning Tree (pvst), we will have to wait through the Listening and Learning stages before moving along to each switch.

We can see that the HSRP sessions break down as well.  Once spanning tree has reconverged, the HSRP sessions reconnect and routing is back online.  Once all four switches are running rapid-pvst, the following configuration changes will update the spanning tree much quicker.

Secondly, we will need to configure the trunk links to be identical (all the VLANs will need to be allowed on all the trunks).

The easiest way to do this is to use a range command on each switch that adds the line 'switchport trunk allowed vlan 1,11,22,33,44'

We can use the dia map to find out which ports can be added to the 'interface range' command instead of having to go through each port individually.  Remember, when a trunking command is added or changed, spanning tree will reconverge.  Obviously, this type of work needs to be done off-hours since the convergence will temporarily take down each VLAN.

Once each switch has identical allowed VLANs, it's time to plan our EtherChannels.  Again, dia can be used to help with this planning as well.

Under the SDL group of icons in dia, there is a circle that can be used to indicate an EtherChannel.

We can add the circles and change the shape into an oval and add a label to plan out the EtherChannel port numbers so the numbers don't overlap on the same switch.

There are a few different ways to configure EtherChannels (which will be a topic for another series).  So, for this article, let's just choose a quick configuration and move along.  Also, remember that when you change the configuration on one end of the links, the ports will go down until the other end is configured.  So, it is best to start with the remote end first so that you don't "cut your own arm off".

Once the EtherChannels are in place, it's time to make sure that the odd numbered VLANs are using DS1 as the root switch and that the even numbered VLANs are using DS2 as the root switch.

One of the many goofy things about spanning tree is that the 'priority' setting is backwards (at least in my way of thinking).  The switch with the LOWEST priority is the root switch (which is the opposite in the HSRP configuration).

Spanning Tree = lowest priority is the root switch
HSRP = highest priority is the root switch

So, when I set up a spanning-tree priority, I like to set the root switch with a priority of 4096, the secondary switch with a priority of 8192 and the other switches with a priority of 40960.

For DS1:

For the even numbered VLANs 22 and 44, we'll set the priority as 4096 on DS2, 8192 on DS1 and 40960 on the other two switches.

We can verify that spanning tree and HSRP are working as planned with the 'show spanning-tree summary' and 'show standby brief' commands:

We can see that DS1 is the Root bridge (root switch) for VLANs 1, 11 and 33 and is also the Active router for VLANs 1, 11 and 33.

DS2 confirms that it is the Root bridge (root switch) for VLANs 22 and 44 and is also the Active router for VLANs 22 and 44.

Now that we have the rapid-pvst running, the trunks configured, EtherChannels set up and the spanning tree priorities configured, our final maps look like this:

Odd numbered VLANs (1, 11, 33):

Even numbered VLANs (22, 44):

---

I sincerely hope that you download and become comfortable with using dia and that this series has been helpful to you in learning, configuring and troubleshooting trunks and spanning-tree instances.

Thank you for reading!