NSX, Dynamic routing and VPC

While the subject has come up quite a bit, it's still mostly shrouded in mystery and getting a proper answer on what does and doesn't work usually requires some shady voodoo rituals. SO let's talk about dynamic routing over VPC.

First of all, let's get one misconception out of the way: dynamic routing through a VPC has always been supported. This means that your VPC chassis only serves as a L2 transport for your dynamic routing traffic. As seen in the picture below, routing between L3-A and L3-B/L3-C is always supported since loop protection never kicks in.

What is important however is that cisco has changed their stance on dynamic routing over VPC from "not supported, go away" to "it's supported, but with very specific constraints. This was actually changed a bit over a year ago but apparently a lot of people still don't know this. A network integrator i was working with on NSX wasn't aware of this, and it seems more people - even those that work with Nexus VPC's daily - aren't.

So what are the constraints? If we look at http://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html, we can see that dynamic routing between Nexus-A and Nexus-B is very dependent on what model you have. If this is what you want, your best bet is to go with a 9k or 7k, which is usually where you'd want dynamic routing in a VPC setup as they'll most likely be located in your DC core or aggregation. Nexus 3k/3.5k also support everything under the sun, the really odd ones out are the 5k and 6k's which support dynamic routing over their VPC peer link only, which is obviously a limitation.

Now when we look at dynamic routing between L3-A and Nexus-B, we'll see that this is where we'll slowly start going into a hazy fog of compatibility issues. It is entirely dependent on your model whether or not dynamic routing from L3-A to Nexus-B. The most important thing out of this page to realise though is that "L3-A to Nexus-A peering is always supported for L2/L3.".

Now what does this all have to do with NSX? If we look at the NSX Edge deployment model, you'll usually peer those with your aggregation or core. So in this image your ESG would be L3-A and your physical peers would be Nexus-A and Nexus-B. Should be all fine if you have an L2 connection between your edge cluster and your physical switches, right?

Not exactly. Because the cisco document clearly states that this applies to physical L2 connections only. A virtual L2 connection would be a VLAN with a SVI configured, and as such would be traversing the VPC configured on your access switches to the aggregation/core switches and would be subject to loop prevention protocols in the VPC.

So in general, while Cisco does have better support for dynamic routing over VPC, i'd personally recommend either building a non-VPC network for dynamic routing - which in turn is a good argument for a dedicated edge cluster - or doing dynamic routing to non-VPC cores, so you'd get something like this:

Earlier this week we've had a good discussion about how to perform routing, and in the customer's case they have 7k's as DC cores with a new N9k ACI fabric below that, and just the headaches saved from not having to worry about VPC configuration and restrictions to specific firmware versions and types of switches made us decide to just plug in additional cables, create non-VPC VLANs and just live with the fact that VPC and dynamic routing is still a bit of a mess.

Fortunately we have dedicated border leafs connected to the 7k cores, which means we only need to configure the non-VPC VLANs on the border leaf, saving a lot of work. However, if you have a classic access/aggregation/core network you'll need to pay very careful attention to which switches your edges will peer with and which vpc's you will traverse.

Unfortunately the VMware documentation regarding dynamic routing and VPC's don't take the document linked above into account, so while it's still unclear what is and isn't supported, my personal preference is to always use dedicated non-vpc links for your routing networks, and hopefully Cisco and VMware will sort out what exactly is and isn't supported regarding VPC's and routing.