Episode 414 – ONTAP Networking Updates – Fall 2026 – Why Is The Internet Broken?

Episode 414 - ONTAP Networking Updates: Fall 2026

===

Justin Parisi: I'm here in the basement of my house, and with me today I have a special guest to talk to us all about networking with ONTAP, in general, networking concepts. And I can think of nobody better to do that than the, the man standing or sitting here on video with me right now, Richard Scheffenegger.

So Richard, tell us who you are, what you do, and how to reach you. 

Richard Scheffenegger: I'm Richard. This is a continuation of a loose series around networking topics that we've started, I believe, like four years, five years ago. I- 

Justin Parisi: We're just gonna reset. This is the first one. 'Cause I don't know if anyone's gonna remember that. We'll post a link to the other one, but we'll go ahead and just ... we'll say this is the new one, the first one 

Richard Scheffenegger: No no. We link to the other ones. Anyway, so I'm here based in Europe. I'm a solution architect with responsibility for the entire EMEA and Latin America region, and I'm specializing in networking specifically coming out of the performance corner, but also resiliency and with my past experience, I've had a very high exposure with networking technologies and that's really fruitful here to bring to the table. 

Justin Parisi: So whenever somebody asks, a networking question, if I don't know the answer to it, I always either point them to you or sometimes to Elliot, right?

But mostly to you. I'll go, "Richard's 

Richard Scheffenegger: probably-" No, there's a third one. There's a third member- Who's the third one? In this group, and that's the one that's responsible for APAC. That's Tim Kleingeld. 

Justin Parisi: Oh, there you go. Tim Kleingeld. Okay. So that's another good one, right?

Richard Scheffenegger: And speaking of resources, if anybody wants to reach out and discuss these networking topics, especially the advanced ones that we may not have time to cover in all of the depth specifically I'm reachable under SRichard, that's the easy one to remember @netapp.com.

And I'm always happy to help. 

Justin Parisi: So what's interesting about networking is it's one of those things that everyone relies on, everyone needs, but not everyone understands in depth. I am one of those people. I know ... I understand some topics, I understand some concepts, but when it comes down to all the latest and greatest features and functionality, it really isn't very well-publicized, so we end up missing out on some feature richness that we could be leveraging in our environments. This episode's basically designed to cover some of those things, maybe concepts that we don't know as well or maybe things that we hadn't even heard about that we might be using in our environment. So to start off with, we'll go real basic. We'll talk about VLANs and VLAN tagging, and where it fits in ONTAP and how it works there, and some of the use cases that we would use it for.

Richard Scheffenegger: VLANs is a concept that has been introduced in networking technologies something like almost 40 years ago now maybe 35. And the name implies a virtual LAN so it's a layer 2 technology. You basically create on top of physical networking devices multiple independent layer 2 domains that really don't talk to each other natively.

So you would have to pass through a router, a layer 3 device, to go from one VLAN to another VLAN. Or alternatively, you can also think of them as security domains walled gardens within which a specific application or a tenant in a multi-tenant environment is living without having a way natively on the network to peer or to go across the fence and impact somebody else's data traffic.

That's a VLAN. 

Justin Parisi: Yeah and, you'd use it for segmenting networks that maybe you don't want to talk to each other, right? So you wanna have that logical separation of those networks. Are there any performance benefits to using VLANs, or strictly just for, carving networks up into- So they are-

smaller 

Richard Scheffenegger: areas? ... not really performance benefits. They are really more from a security perspective and maybe performance in the sense that you're limiting the scope of your broadcasts. So when you're doing a broadcast like for example the most well-known, the most critical one that you can have in an IP network is an ARP broadcast.

Basically asking who has a specific IP address, because you want to talk to that IP end host. That is a broadcast, and it's typically flooded. So flooding is the technical term here meaning it's getting spread out across all the physical ports that make up either a physical network when you don't have VLANs, or you create a VLAN on top of your physical network, and then you limit the scope of this broadcast so it only reaches those clients that actually have a need to know 

Justin Parisi: Would it also be useful in situations where somebody maybe wants to run a denial-of-service attack using a broadcast type of approach to the

The scope would be limited to just those in those VLAN as opposed to the entire network? 

Richard Scheffenegger: Absolutely. However, in order for this particular attack vector to be viable, the attacker would already have to be quite deep inside of your network. So you would have to have a lot of other defenses prior to that level.

And I believe, at least from my reading of the tea lives in this aspect, and I try to be quite far away from security because otherwise people may mistake me for somebody who has any kind of knowledge about it, is that these kind of attacks are not as dominant anymore because there are much more easy ways to exfiltrate data or to encrypt data.

Basically what you can do when you have deep access, you don't really need to compromise the network or craft packets or inject specific data frames that, corrupt stuff. Then you really get much more benefit by exfiltrating data and then you can extort money for not publicly shaming people or something like this or encrypting your data set.

Which is an entirely different discussion that we can also cover with ONTAP, as with our ransomware protection story.

But again, this is something different. VLANs now are table stakes, so to speak, and predominantly they are being used, I would say, for security purposes.

So separation of broadcast domains, so this is basically a little bit of a performance aspect here, but more importantly that you have yet another layer of security limiting who can talk to who and when specific traffic protocols need to pass or a firewall need to get inspected before they traverse different security domains.

So that's really the use case of VLANs. One aspect that may be irrelevant again with this multi-tenancy approach, again, security, is that while you only have a limited number of physical ports, the number of VLANs can be much, much more. And in theory a VLAN number 100 on the first port doesn't necessarily have to do anything with a VLAN 100 on a second port.

So those two constructs, a VLAN 100 on port one and a VLAN 100 on port two are logically separate from each other. And you can then utilize this together with IP spaces. So this is what we call another networking construct, better known as VRF virtual forwarding and routing to really segregate even into the layer three, not only layer two, but also into layer three different clients security domains so that they are not physically or network layer able to communicate with each other, but have to traverse a firewall or some other more security-focused appliance to facilitate communication 

Justin Parisi: So you see a lot of VLAN tagging, you mentioned it's table stakes.

What about IP spaces? I don't think I've seen a ton of that out there with ONTAP. I know it's been there since probably the 8.0 releases, nine, 

Richard Scheffenegger: early nines. No I don't want to break the news to you, but we are utilizing IP spaces on every single ONTAP system. So for example cluster network, what we call or the backend traffic the clustering is running in its own dedicated IP space, especially for the reason to not interfere in any which way with the front-end traffic. So one of the easy examples is that we are utilizing IPv4 link layer local IP addresses starting with 169 if anybody has ever seen those. And in theory, you could have overlapping IP addresses in your production network. So we don't want- ... to interfere with that production network, obviously.

Therefore, our back end is separated out into its own dedicated IP space. But I probably should be referring to IP spaces because there's a multitude of those, to be absolutely correct. 

Justin Parisi: Yeah. And maybe I should rephrase my question. I don't see a lot of people setting them up themselves.

We definitely have them as system baked things. Basically whenever I look at an ASUP, I don't really see it set up, and I myself haven't utilized them a whole lot. So IP spaces as a client or customer configured option how often do you see that, and what are some of the use cases that people are leveraging for IP spaces aside from, getting the ability to have the same IP address across SVMs?

Richard Scheffenegger: So the- you see them quite a lot when we're talking about specific verticals, like for example, service providers. Either external service providers, which offer a service to the public that is ultimately mapped in ONTAP into its own dedicated SVMs, and those live inside of a dedicated one for each customer IP space.

But you can also have these for internal service providers. So when you're, for example, in the financial sector, you see them quite often being used, again, because they are... So to speak if VLAN is like your one yard fence, the IP spaces or VRFs is the equivalent of your three yard fence, right?

So it's much more hard to step over a three yard high fence than it is over a one yard fence. Yes. 

Justin Parisi: Okay. So you mentioned that service providers use it, and I guess that would be because they have so many customers that come into their environment and they require the ability to have whatever IP address they want at any given time.

So every customer would have their own dedicated IP space, essentially. And I would imagine you'd see that in our 1P service providers like Azure NetApp Files or Google Cloud NetApp Volumes because they leverage, back-end hardware for this use case. Now with IP spaces and multi-tenancy, what are some of the, major benefits that customers get out of that secure multi-tenancy that, maybe we don't cover enough that in your opinion? 

Richard Scheffenegger: One of the things that you can do relatively easy within one IP space is basically shift a volume from one vServer to another vServer, and both of them live in the side of the same IP space.

So with relative ease, you can by a malicious actors, let's say or by an accident, you're mistyping something host a volume on the wrong vServer, and then it gets exposed to the wrong set of clients, right? When the volume and the SVM are allocated to different IP spaces the misconfiguration is much more involved.

It's not a simple single mistype in a command. But you would have to do like something like three or four commands in a row all of them done properly in order to reassign a volume to another IP space. So it's much less likely, even if you have the credentials to misconfigure stuff, which is, to be honest probably more likely than a malicious actor interfering with the configuration in the first place.

Justin Parisi: Yeah. 

Richard Scheffenegger: Having said this you already touched on upon it overlapping IP spaces, so you don't have to- ... worry about the routing table, which is really the key here. Having similar entries, you can really ensure that traffic that is destined to a specific tenant, to a specific security domain, to a specific customer really goes out via the proper port or the proper VLAN, and not because of a mistyped IP routing entry in one or multiple data SVMs all of a sudden is sent out via the wrong interface appearing on an intermediate device which may no longer be under your control or which may be used to exfiltrate data.

So again, it's security. It is one of the key stepping stones when you're talking about our secure multi-tenancy approach. Another aspect where IP spaces come in handy is when you want to tweak your TCP for specific performance characteristics. What I'm referring here is the best practices for asynchronous SnapMirror, which is typically deployed across long distances maybe even satellite links, you know, transpacific, transatlantic, things like this.

If you tune your TCP to optimize for these kind of very big bandwidth but very high latency links, those optimizations are not the best to use within your campus, within your local network for regular client access. So this is another aspect where these IP spaces come in handy where basically you have the capabilities to have one set of rules governing SnapMirror and another set of rules governing your SMB or NFS access to your data SVMs.

Justin Parisi: So what are some examples of the differences with those optimizations? Like, why would they be bad ideas in localized data centers? 

Richard Scheffenegger: Just the other day quite literally, a customer came back and reported that they have poor performance for SnapMirror because, again, not optimized for the long distance relationship.

However, without the dedicated IP space, if we want to have the TCP optimized for this long distance, so the optimizations that I'm talking here is called the bandwidth delay product. So in order for TCP to utilize the full bandwidth, it has to have a receive window that is scaled effectively to the bandwidth times the latency that you go from the source to the destination.

So let's say you have one hundred milliseconds across the Atlantic and you have a 10 gig link, so that's 10 gigabits per second or roughly 1.2 gigabytes per second times a tenth of a second. So multiplying this out means we're talking about 125 megabytes of receive window size to utilize this 10 gigabit link to the fullest. If you do this, however and you set it up to not auto-scale, so to not dynamically grow because, again satellite links have a very high delay and each growth cycle takes one of these round trip times. So you don't want to do this slowly, but you want to have them set up instantaneously You consume quite a large number of main memory for this, right?

Which is not that bad of a thing for SnapMirror when we're talking about the typical used six, 12, 18 TCP sessions. The systems they do have plenty of space for that. However, now remember the system will also cater for, let's say, SMB traffic, and you have home share home drives on that.

So all of a sudden we are not talking about 10 TCP sessions being tuned and memory reserved for that, but all of a sudden we are talking about hundreds of TCP sessions, all of which want to have this pre allocated memory. And now this cuts into your budget for waffle buffer cache, meaning the waffle buffer cache has less memory to work with and therefore your read performance of the entire system will be degraded to a certain extent when you do this- -optimization outside of utilizing IP spaces. So this is where this- Okay. -segregation makes a lot of sense because you can optimize different settings for different environments. And SnapMirror is one of the prime examples. If you go into the best practices for SnapMirror, for asynchronous SnapMirror, then dedicating an IP space for these long-distance links is mentioned there as one of the key tunables.

Full disclosure talking about SnapMirror ActiveSync, our synchronous replication solution in that space that utilizes the default IP space. So if you don't do anything fancy, then it's all in the default IP space. However, SnapMirror ActiveSync typically is only allowed to be deployed with ten milliseconds of latency.

You still want to have quite high bandwidth, but again the number of pre-reserved memory for that purpose is not as demanding as it would be for an asynchronous SnapMirror going across the Pacific, for example. 

Justin Parisi: So with this optimization is mostly for the asynchronous. You wouldn't wanna use it for synchronous 'cause it would just kinda probably kill the system, right?

Richard Scheffenegger: You need other types of optimizations for synchronous than you need for asynchronous. For asynchronous, you don't care that much about the latency, for example. For synchronous, you care absolutely about latency. You- ... probably are better off with trading a little bit of peak bandwidth for a more optimized latency response when we're talking about synchronous replication solutions.

And again right now there are efforts underway to also make the IP spaces concept coexist and and work together with our synchronous solutions, again, for the very reason to optimize the tunings, the- ... settings to optimize s- for specific environments. 

Justin Parisi: Yeah. It's interesting.

I never thought of it as a way to have dedicated optimizations for specific networks. I think that's really valuable. 

Richard Scheffenegger: And just because it's called an IP network, not all IP networks are created equal, right? You have satellite links, you have Wi-Fi links. Very popular these days especially when we are talking about working from home customers or working from home clients at customer sites that then allow direct access to SMB home shares inside of ONTAP.

So something that we have improved with 9.19, which is already available, is we have an improved TCP stack that is specifically catering for those requirements when you are one of those sitting at home in the home office and your home share is hosted on a ONTAP filer, but you're accessing this via first hop is a Wi-Fi link from your laptop to your access point at home, then you have maybe a DOCSIS cable modem from your access point to the internet on top of which there's a VPN to your VPN concentrator.

And on top of all of that is then your SMB session all the way into the ONTAP system. So that's not the best scenario for our classical TCP implementation that we've been using for pretty much forever. And recently with ONTAP 9.19, we introduced a different TCP stack that is specifically optimized to deal with the things that are exposed when we're talking about public internet, VPN and Wi-Fi.

So it's a much more performant TCP stack. However, it's also slightly more CPU intensive And not quite as effective when it comes to campus or high-speed networking. So this is the reason why it's not the new default. But for users sitting at home the performance improvements are dramatic.

So just the other day I heard back that this new TCP stack, as mentioned in this specific environment, can have a factor of three to four improvement sometimes even higher than that. So it's really dramatic. But you don't necessarily want to deploy this for all the on-site, on-campus TCP sessions.

So again, maybe having or looking into IP spaces to deploy these optimizations for those end devices that are actually needing them is something to you want to do 

Justin Parisi: Yeah this makes me think we need an entire episode dedicated to optimizations for specific use cases.

So we'll table that until till we come up with that. Okay. But so that's TCP Rack you're talking about, right? Correct. And that's able to be set on the IP space independently from everything else? 

Richard Scheffenegger: Correct. 

Justin Parisi: Does it replace FlexCache as a necessity or does it enhance FlexCache or is it just completely separate from what FlexCache can do for those remote access use cases?

Richard Scheffenegger: It's orthogonal. So FlexCache, if deployed correctly, it really is cutting into a latency by virtue of locally answering the requests. So the client- ... only observes the latency to the local cache instance. And with a write back caching the latency from your local ONTAP instance, maybe it's a ONTAP Select, maybe it's a physical server back to the origin is pretty much out of the picture.

However, you would still want the latency for certain operations because FlexCache is a global coherent caching mechanism, so you still have to coordinate via the origin among all the clients. So the latency response there on that lag of the link still is very important.

So deploying a TCP Rack, if that is running over a VPN or a long distance link, for example, with a benign bandwidth would still be quite interesting to deploy. So they are running alongside each other, but they're complementing each other, right? 

Justin Parisi: Could it be used in conjunction with the FlexCache link, so linking from the origin to the cache- Yeah

using 

Richard Scheffenegger: TCP Rack? Yeah. That was what I was referring to. 

Justin Parisi: Okay. Okay, excellent. 

Richard Scheffenegger: But again, so FlexCache is built on top of our SnapMirror technology. And if you have been following the best practice of asynchronous SnapMirror you would have hopefully deployed IP spaces to begin with utilizing those IP spaces for the connectivity between the origin and all of the caches.

So changing and optimization of the performance characteristics on that path should then be possible without any side effects. 

Justin Parisi: You touched on earlier about multi-tenancy with IP spaces and that sort of thing. Every SVM can also leverage a functionality called failover groups, right? So the idea is that if you have an outage on your link or a port fails or even a storage failover occurs, independent IP addresses that float around the SVM will fail over to a specific set of interfaces so that we don't have a 100 gig interface failing over to a one gig interface, 'cause that'd be bad for performance, right? So tell me how that all kind of has evolved over the last few years and what technologies it leveraged versus what it does today. 

Richard Scheffenegger: Yes. Really failover groups live inside of broadcast domains.

So broadcast domain really is a construct that supersedes VLANs because I mentioned earlier, a VLAN 100 on the first port doesn't necessarily have anything to do with a VLAN 100 on the second port. So you still have to have an entity which binds those two things together. So if the VLAN 100 truly spans those two ports broadcasts sent out via port one will be received on port two and vice versa.

So that's the broadcast domain. Nowadays, broadcast domains typically get created under the hood automatically. So if you're not very careful, you get a long list of default dash whatever broadcast domains, and it will try to pick up all the physical ports or the configured VLANs which can see each other via a broadcast. Up until approximately 10, maybe seven years ago what we have been using for detecting which of those ports belong to each other and basically make up or should be belonging to the same broadcast domain we basically leveraged the ARP broadcasts, which I've touched upon earlier.

The ARP broadcast however, is special in IP networking, and nowadays almost all networking is IP networking. ARP broadcasts are no longer very reliable because they get intercepted. So certain network technologies especially those that are utilizing VXLAN as the foundation also have capabilities in the network to intercept and interact with broadcasts in order to remove those broadcasts from getting flooded out and taking CPU cycles on clients that don't need to know anything from those broadcasts.

With the advent of those overlay networks, so VXLAN or maybe better known Cisco ACI as one specific vendor's example ARP broadcast became unreliable to detecting which port can physically reach which other port. So we switched from ARP broadcasts to an I triple-- IEEE standard CFM. That's basically a broadcast protocol specifically designed for administrative and maintenance purposes to really detect this extent of where a broadcast really can go. So it's called a CFM protocol. And now we are utilizing that. Every approximately ten minutes a member port will send out one of these CFM broadcasts, and all of the other ports which receive the broadcast will basically make a note.

If a port is supposed to be receiving one of those broadcasts but doesn't receive the third of those broadcasts, so we're talking for those that kept count about half an hour or so then this will be flagged and ONTAP will complain that a broadcast domain no longer has full reachability.

And this may have implications for the next step, for the failover groups. So failover groups is your construct by which within a given broadcast domain, a certain number of ports, a subset of ports is eligible for hosting a specific IP address. And this IP address then can move within the failover group, which is within a broadcast domain, in such a way that when a link fails, a link is down, or the logical connectivity is detected to be missing to another port, which hopefully still has proper connectivity to the network.

However, personally, I'm not perfectly happy with this construct. It's the best that you can really do in a classic layer two environment. Why I'm not happy? Because CFMs are being sent out once every 10 minutes or so, and the fracture of a broadcast domain, let's say by somebody has not configured the VLAN on a switch-to-switch link.

And now that switch-to-switch link wasn't used, but another switch-to-switch link, which was the primary one has failed. Now this the, the switch domain starts utilizing this misconfigured port, and all of a sudden you have a VLAN 100 on the left-hand side of the network, and you have a VLAN 100 on the right-hand side of the network.

But those two sides can no longer communicate with each other, and there is really no immediate way to detect this Until ONTAP performs another check with CFM and then fails twice in a row before detecting this. So up to half an hour before the system really detects that something is amiss.

If you happen to have the misfortune of a link down event and the LIF needs to migrate to another port, chances are that you end up on the port that has less connectivity than another port in the same failover group. Because again, failover groups, they can comprise not only of two, but multiple ports, but within a failover group it's not clear which of those ports has the highest priority- Are alive ... is still ex- exactly. Is still alive- Yeah ... or has the highest reachability to the most or the critical subsection of clients. 

Justin Parisi: So you run the risk of failing over to a dead port, essentially. 

Richard Scheffenegger: Exactly. Or maybe not to a fully dead port, but to a port that is suboptimal, right?

But there's no way- 

Justin Parisi: But mostly alive. 

Richard Scheffenegger: Mostly alive, but not ... You really have no way of knowing ahead of time that this was not the best of choices. Speaking of best of choices obviously this is the current state of affairs, but we are not content with that. So there are features that are currently in the making in ONTAP one of which is called bidirectional forwarding detection, BFD.

And what we are looking here is to really establish for critical components, like, for example, your database server or your hypervisors to have a permanent, like every 10th of a second check if there is an ongoing communication. So even if you don't really have any data transaction going on, which is one other way to check the connectivity, right?

To really have a keep alive. You can think of it like a hello packet and then a response to the hello if this reachability to your critical resources is still given, and if it is not given, which of your failover ports still has this critical reachability. So then it will utilize this specific port rather than another one, which is maybe configured in the failover group, but ceased to have connectivity at the very same moment as your primary port ceased to have connectivity. 

Justin Parisi: So we've come a long way since IP Fastpath, right? 

Richard Scheffenegger: A very long way. 

Justin Parisi: Yeah.

Richard Scheffenegger: That was removed with a nine-six. 

Justin Parisi: Yeah. But I remember, like how problematic that was networking-wise for a lot of people, and we've done a lot since then. 

Richard Scheffenegger: Yeah. Fastpath really entrenched a misconception about IP networking. People have this notion of IP flows going bidirectional the same way that it's being sent, the traffic comes back, right? But this really is not how IP has been designed back in the 1980s or, sorry, 1970s even, because back in that day it was very clear that you have network paths that have very different characteristics in one direction versus the other direction.

So for better or worse in Germany you still have a lot of ADSL links, ... meaning you have higher bandwidth going from the internet to your home than from your home to the internet. So you may be having a download speed a couple megabits per second, but the upload speed may be only a couple hundred kilobits per second.

And links like this on a larger scale obviously were still dominating the internet back in the 1980s, 1990s. And so from an IP forwarding perspective, it w- would make absolutely sense to have the traffic in one direction utilize an uplink with high bandwidth, and on the reverse direction not go with the same link that has a low bandwidth on that unidirectional side but via another intermediate devices.

So asymmetric routing is really the state of affairs. And what we've done with ONTAP 9.6 with the removal of Fastpath is basically restoring the natural state of IP forwarding very much to the chagrin of many people that didn't know better or the ... under the false assumption that symmetric routing is the natural state of affairs, which really it isn't.

It's more of an exception than the rule. 

Justin Parisi: It's more overhead to manage too, I would imagine, doing a symmetric routing. 

Richard Scheffenegger: To really be symmetric, yes. Absolutely. Yeah. On the other hand, there are still valid reasons why you want to do this. So one of which, again we seem to be revolving around security.

So you have stateful firewalls which by the way can be multiple devices nowadays. But if you have a single device being a stateful firewall, that stateful firewall wants to inspect both the forward and the backwards direction to ensure that nothing fishy goes on. And therefore it needs to see both directions, both half connections, and that's where symmetric routing then becomes essential 

Justin Parisi: So you were hinting earlier with CFM about it not being optimal, and I felt like you were leading us towards the next logical topic, which is BGP.

Now, I may or may not be wrong with my my jump and conclusion there. So does BGP solve some of those CFM limitations with failover groups? Because essentially you have an IP address that can be presented across different ports. Is that accurate or is that ... Am I completely conflating the two?

Richard Scheffenegger: I would describe it differently. So but you are very correct in that is basically where I was going to go with this this discussion. BGP is effectively the next step in this evolution. The basic premise here when we're talking about resiliency is that BGP you can configure it to have down to three seconds active communication between the ONTAP node and the adjacent networking device, the adjacent router.

And not only are you limited to one or at most two routers or devices within the case of LACP. So LACP and MLAGs are limited really to physically only two different physical switches that you can peer to. With BGP peering you can connect to quote any, unquote, number of routers.

I believe we have tested up to 16 And all of them are actively managed by ONTAP, so to speak. So they are monitored actively. So only when a node has an active BGP peering inside of an IP space, because each IP space again is its own thing only then a virtual IP, which is the LIF that goes along together with BGP is eligible to failover to one of those nodes.

So a node that has lost all of its BGP peers will no longer be eligible to be hosting the IP address of a failed node, either because the node failed or the connectivity of that node has failed. So if the cluster or metro cluster is large enough, you have ample opportunity by the sheer fact that you can have more than two devices that you're peering with.

You can have five devices, you can have 16 devices. So that dramatically reduces the chances of such an unfortunate event where you're basically failing over into the blind because 10 minutes ago everything was fine. Some networking event brings down the connectivity on your primary port, and then the backup port may or may not be impacted as well.

With BGP, ONTAP knows. Without BGP, ONTAP doesn't know. It tries its best, but every once in a while we get the support ticket the choice was wrong in the similar manner as the old Indiana Jones movie. 

Justin Parisi: Oh, yeah. The goblet, right?

The- Yes, exactly. 

Richard Scheffenegger: The- ... 

Justin Parisi: the chalice you chose poorly. 

Richard Scheffenegger: Exactly. 

Justin Parisi: So i- my understanding is rudimentary, but it's basically with BGP, your LIF lives more at a domain level, like a cluster level. Whereas with normal networking, it's living at a port level. So BGP is more floating and the LIF can reside on any active port at any given time, given the bGP peering at NC ... 

Richard Scheffenegger: not only can it will be on- Yeah ... any active port on the node that is currently hosting. So it's not cluster level per se, it's node level. So any port that is configured with BGP and receives a packet destined for this virtual IP address, so the destination address, it matches one of the VIP addresses, will process the incoming packet.

So it doesn't matter if it comes in via port one, two, three or five it will be processed accordingly. Which by the way, also gives a very neat functionality improvement. Because if there is a network outage not only are you limited with detecting this LACP has a 90-second timeout by default, which by the way, we have made changes that you can tune it down to three seconds, but it needs to be actively tuned starting with 9.18.1.

Unfortunately, nobody really knows about this, so it's one of another well-hidden secrets that you actually have to change the LACP from slow to fast in order to get the three seconds But three seconds is not, sometimes three second is not the best. With BGP it's three seconds already but you can combine BGP with BFD in the future, and then we are talking sub-second ... detection interval. And the convergence, meaning how long does it take for the IP address to be active again on another port. When you're doing it in the classic way, one physical port hosts a specific IP address, and then you do a lift migrate, same node, just another physical port.

What has to go down in the background? You have to do a gracious ARP request, this ARP broadcast, which as I've mentioned earlier, may get intercepted by the things of Cisco ACI. Then maybe Cisco ACI has to distribute this updated information, the new MAC address to the IP address binding among the entire ACI cloud.

And then the ACI cloud has to come up with a list of devices where it reconstitutes these ARP broadcasts to notify your end device, "Oh, the MAC address to this specific IP address has changed." So if anything in this chain of events takes more time or is skipped You end up with a problem, a partial problem. Maybe only a single host has no connectivity for the next 15 seconds or maybe five minutes with a device or with this IP address. But with BGP, all of that happens more or less instantaneously. So as soon as the BGP is active, it announces out the reachability across all of the ports. It doesn't matter if the packet arrives via port one or via port two on the same node. If it has to move from node one to node five in a cluster, BGP will announce this out not on a timescale of seconds like with ARP, but on a timescale, scale of milliseconds, and it will be distributed in the routing protocol in a much more reliable manner than you can expect from an unreliable broadcast like a gracious ARP in the classical sense.

So it's much more dependable, so to speak. There are fewer things that are not under your control from a networking operator's perspective that can go wrong. If things go wrong, you have methods to validate and to check, which is more that what you can say with modern layer two overlay technologies like VXLAN or Cisco ACI, which is much more like a black box, and effectively you're hoping for the best, and ninety-nine point nine nine nine nine percent of the time everything works fine.

But then, this one out of 100,000 event, you really struggle to troubleshoot this because very often these are fleeting problems, right? They are resolving themselves within five minutes, maybe 15 minutes. But still, if you don't have the expertise on hand looking at this specific moment, at the specific moment in time it resolves itself and you don't find out what was the root cause for the lack of connectivity.

Justin Parisi: So lot of resiliency feature functionality there. Is there performance enhancements that come with this as well? Are we able to leverage more ports with a single IP address or is that strictly just, one-to-one relationship there? 

Richard Scheffenegger: With BGP, absolutely. With the classical layer 2, the best that you can hope for is LACP. And LACP has a certain number of caveats. So all of the ports have to be the same link speed. On some switches and with older versions of ONTAP even the type of link had to be the same.

So you couldn't bundle together an optical with a copper link, for example. That fortunately is gone, but now you cannot bundle 100 gig link with a 10 gig link for resiliency reasons, right? And prefer one over the other. That's not how LACP is working. You can only scale LACP up to I don't know, I think in the order of eight physical ports. But those physical ports can only be spread across at most two devices, two switches. So four ports here, four ports here. With BGP you have again a much higher variety of types of ports. You can host the IP address not only on 100 gig but also at the very same time on a 10 gig port.

And then utilize IP routing metrics, which can be configured either on the storage device or with BGP within the network itself so that it prefers obviously the 100 gig port as long as that one is available, but can then fall back to the 10 gig port if it needs. 

Justin Parisi: But there's still a one-to-one relationship of IP to node though, right?

Richard Scheffenegger: It's a one-to-one relationship between IP and node unless we are talking about special solutions that I've come up with around FlexCache. Yeah ... so I presume what you're going here with is there's another construct that BGP is also utilized for very frequently, especially when we're talking about web technologies.

So basically stuff that S3 the storage protocol of the web, so to speak, is known for, and this is Anycast. So Anycast for those of you who don't know, is basically you have a well-known numeric IP address. So the classic example that I usually bring is 1.1.1.1. So if you are somewhere in some remote parts of the world and you do have internet connectivity, but you don't know how you get to your DNS to proper fully resolve the names, then just use either 1.1.1.1, which is the one from Cloudflare, or 8.8.8.8, which is the one from Google .

And that is not a single server living somewhere in a data center somewhere in a single site. Rather, it's configured as a route, a slash-32 route in the global routing table. And whichever server is the closest to you will get your request. And with utilizing BGP, we can also make use of the same functionality having- The same IP address multiple times in the network.

But since this IP address, the virtual IP address, is not a physical IP address, it's not bound to a specific MAC address, it's not bound to a physical port, it's just represented as a routing entry. It can have multiple points of exit, multiple devices servicing requests. Now, the trick here is obviously ensuring that whatever service is behind this well-known IP address is the same service, is the same volume, has the same credentials and so forth.

So this is where the trick now revolves. And this is where FlexCache comes to the rescue, but it's not tightly locked with each other. So if you want to go down that path, if you want to have let's say, a software distribution share where all of your software packages live and you want everybody to know if you're setting up a new laptop, just go to 10.10.10.10 and log into the export or to the share there, and it will always be catered to you from the most local device so that it's low latency, high bandwidth and so forth then we have the solutions utilizing FlexCache with this.

Justin Parisi: So it's like a single namespace design at a network level? More or less 

Richard Scheffenegger: It's yes, you can think of it. So it's extending the notion of single namespace all the way down into the network so that, ... you're not dependent on additional services like distributed file system, DFS, that somebody else outside of the storage team has to administer and configure.

Or when you're talking about NFS arcane scripting in the automounter so that maintained list of NFS servers which all host the same data is then picked by some heuristic to hopefully point to the closest device. With- bGP and Anycast, it's basically exploiting the natural topology functions of a dynamic routing protocol to ensure that it's always the best device that will serve the data. 

Justin Parisi: Yeah. So basically I can access IP/volume anywhere from any place, and the routing points me to the right place and the cache- 

Richard Scheffenegger: Exactly

choose to 

Justin Parisi: do that. 

Richard Scheffenegger: Exactly. There's another aspect to this if you want to discuss this a little bit further. If you're talking about extremely high throughput demands, so think of machine learning or what colloquially known nowadays as AI 

Justin Parisi: i'm gonna stop you there 'cause we're gonna save this.

This is good stuff. We're gonna save this for a future episode. We're already very much at the top of the hour here that we're running for this. So lots of good content here. We didn't even get to RDMA. We'll get to that in another episode as well. So we'll definitely have Richard back to talk about all these concepts in depth. In parting what are some of the most common misconfigurations that you're seeing that people can avoid just by listening to this particular episode? Just give me like the top two. 

Richard Scheffenegger: The top two. So the ones that are most close to my heart is misconfiguring LACP. I've been on a chase to eradicate all the examples where we are giving LACP examples with IP-based hashing, because I don't think IP-based hashing should be showing up anywhere anymore.

Because really- ... nowadays it's, nConnect, it's SMB multi-channel. So you have the same end host with the same IP addresses, but multiple TCP sessions, and you still want to have those TCP sessions, not only for performance but also for resiliency reasons, traverse different physical paths. You cannot do this with IP-based hashing.

So port-based hashing it is. So that's the current number one because that's frequently not set up properly, and to rectify it, it's a big... it's not an outage. You do a LIF migration of the node that is impacted. Then you remove all the configuration manually because you have to remove all the VLANs, you have to remove all the broadcast domains, all the failover groups, blah, blah, blah. You need to change the home port of the migrated LIF because you cannot, delete the home port.

So it's a big mess, right? It's not disruptive per se because you do the LIF migration. Okay. Maybe you have a slight hiccup but it's just, a pain in the posterior to actually do it. Doing it right from the get-go is so much more easy and so much more fun to be honest.

Justin Parisi: It's an outage depending on your protocol. I mean- Yeah ... LIF migration's not entirely non-disruptive if you're using like SMB or NFSv4, that sort of thing. All right, so lots of good content here. We definitely have a lot to talk about in later episodes, so stay tuned for those. So Richard, again, if we wanted to reach you, how do we do that?

Richard Scheffenegger: SRichard, so just one word without any dots or dashes- Okay ... @netapp.com because I don't want anybody to butcher my last name. 

Justin Parisi: All right, excellent. We'll add that to the accompanying blog post, and we'll also include links to those previous episodes. I think we covered BGP in one of those as well.

Yes. This is probably a little more detail than we had in those. So Richard, again, thank you for joining us and talking to us all about the networking stack here with ONTAP. 

Richard Scheffenegger: It was a pleasure. Thank you.