VEEAM Invests in Faster & More Efficient Data Protection With Backup & Replication 8

Ever more data to protect without breaking the systems or the bank

One of my major concerns today in IT, weather it is on premises or in the cloud, is the cost, time, reliability and feasibility of backup and restores. This true for most of us. Due to the environments in which I deliver my services my main issue with backups is the quantity of data. The amount of data is staggering and growth is not showing a downward trend.

The big four: CPU, Memory, Network & Storage

Over the years we have seen a vast increase in compute, memory, network and storage capabilities and pricing. CPUs are up to 18 cores per socket as I write this. DDR4 memory is here and the cost is relatively low. We have affordable 10Gbps networking to throw at the problem as well or in some case 8 to 16Gbps Fibre Channel. So when it comes to CPU, memory and network we’re pretty well served.

Storage is evolving as well and we’re getting ever bigger and, if you have the budget that is, faster storage arrays in different flavors. But it remains a challenge. First of all to get the right amount of IOPS and storage capacity at an affordable price point is a balancing act. Secondly when dealing with backups we need to manage the source IOPS & latency against the target. But that’s not all, while you might want to squeeze every last IOPS & 1ms latency out of your backup target you can’t carelessly do that to your source storage. If you do, this might constitute a Denial Of Service attack against your applications and services. Even today storage QoS is either non existent, in it’s infancy or at best limited to particular workloads on storage solutions.

The force multiplier: Backup software capabilities & approaches

If you’ve made sure the above 4 resources are not your killer bottle neck the backup software, methods algorithms and the approach used will be either your biggest problem or you best friends. You need your backup software to be:

  • Capable
  • Scalable
  • Fast
  • Configurable
  • Scale Out

There are some challenging environments out there. To deal with this backup software should be able to leverage the wealth of capabilities compute, network, memory & storage are offering to protect large amounts of data reliable and fast. This should be done smart and in an operationally supportable manner. VEEAM has been working on this for a long time and they keep getting better at this with every release and it allows for scale out designs in regards to backups targets.

VEEAM Backup & Replication 8.0

There are many improvements in v8 but a couple stand out.

image

Consistency groups (Hyper-V)

Backup jobs can execute more than one VM backup task simultaneously from the same volume snapshot with “Allow Processing of Multiple VMs with a single volume snapshot”.

image

This means you can reduce the number of snapshots significantly where in the past you needed a volume snapshot per VM. VEEAM limits the the maximum amount of VMs you can backup per snapshot to 4 when using software VSS and to eight with hardware VSS. They do this because under heavy load VSS/CSV sometimes has issues. This number can be tweaked to fit your needs (no all environments are created equally) with 2 registry values under HKLMSOFTWAREVeeamVeeam Backup and Replication key:

  • MaxVmCountOnHvSoftSnapshot (DWORD)
  • MaxVmCountOnHvHardSnapshot (DWORD) registry values

Reducing the number of snapshots to be taken is good as it saves resources, speeds up things & as VSS can be finicky, not needing more than absolutely necessary is a good thing.

Backup I/O Control.

Another improvement is backup I/O Control which delivers capability to dynamically adjust the number of backup tasks based on IOPS latency. Under Options you’ll find a new Tabbed sheet, I/O Control. It contains the parallel processing option that used to be under “Advanced” tab in Veeam B&R 7.

image

The idea is to move to a more “policy driven” approach for handling the load backups can put on the storage. Until now we’d configure a number of X amounts of tasks to run against the source storage in order to keep IOPS/Latency in check. But this is very static and in a dynamic / elastic “cloud” world this isn’t very flexible nor is it feasible to keep tuned to the best number for the current workload.

I/O Control let’s you set limits on how much latency is acceptable for your data stores. Removing or adding VMs to the data store won’t invalidate your carefully set number of tasks allowed as it’s now the latency that’s used to dynamically tune that number for you.

I/O control has two settings:

 “Stop assigning new tasks to datastore at: X ms” :VEEAM looks at the latency (IOPS) before assigning a proxy (backup target) to a virtual disk or won’t launch the task until the load has dropped.  This prevents the depletion of IOPS by launching to many backups.

“Throttle I/O of existing tasks at: Y ms”: This will throttle the IO of already running  backup jobs when needed due to some application workloads in the VMs running on the source storage kicking in. The backups will be throttled so they’ll take longer but they won’t kill the performance of the applications while they are running.

These two setting allow for the dynamic and on the fly tweaking of the number of backups tasks running as well as their impact on the storage performance. Once you have determined what latency values are acceptable to you you’re done, VEEAM handles the tweaking for you. The default values seems to reflect industry best practices (sustained > 20 ms is considered problematic)

The below screenshot is for the backup job log and shows latency being monitoredclip_image002

With VEEMA B&R v8 Enterprise + You can even do this per data store, meaning you can optimize this per backup source. This recognizes that is no “one sizes fits all perfectly” and allows for differentiation. Yet it does so in a way that does not compromise on the simplicity of use that VEEAM offers. This sounds easy but from experience I know this isn’t. VEEAM manages to offer a great balance between simplicity and functionality for companies of all sizes.

Select “Configure”

image

In the “Datastore Latency Settings” you can add one, more or all data store you are protecting with VEEAM. This allows for differentiation when you have CSV that are used for SQL Server VMs versus stateless web servers of or other workloads that are not storage I/O intensive.

image

Select the datastore (in our case the CSV volumes in Hyper-V Cluster)

image

By selecting the desired datastore and clicking “Edit”  you can individually adjust the settings for that datastore.

image

Conclusion

It looks like we have some great additional capabilities in an already very good solution. I’ll be using these new capabilities in real life scenarios to see how these work out for us and optimize the backups of the virtualized environment under my care. Hardware VSS Providers, SANs, CSV’s normally need some tweaking and care to make them run well, so that’s what we’ll be doing.

Microsoft Keeps Investing In Storage Big Time

Disclaimer: These are my musing on the limited info available about Windows Server vNext and based on the Technical Preview bits at the time of writing. So it’s not set in stone & has a time limited value.

Reading the documentation that’s already available on vNext of Windows it’s clear that Microsoft is continuing it’s push towards the software defined data center. They are also pushing high to continuous availability ever more towards the  “continuous” side of things.

It’s early days yet and we just only downloaded the Technical Preview but what do we read in What’s New in Storage Services in Windows Server Technical Preview

Storage Quality of Service

  • They are giving us more Storage Quality of Service tied into the use of SOFS as storage over SMB3. As way to many NAS solutions don’t support SMB3 or only partially (in a restricted way) it’s clear too me that self build SOFS solution on a couple of servers is and remains the best SMB3 implementation on the market and has just gotten storage QoS.

Little Rant here: To the people that claim that this is not capable of high performance, I usually laugh. Have you actually build a SOFS or TFFS with 10Gbps networking on modern enterprise grade servers line the DELL R720 or 730? Did you look at the results form that relative low cost investment? I think not, really. And if you did and found it lacking, I’ll be very impressed of the workload you’re running.  You’ll force your storage to the knees earlier than your Windows file server nowadays.

  • It’s in the SOFS layer, so this does not tie you into to Storage Space if you’re not ready for that yet but would like the benefits of SOFS. As long as you have shared storage behind the SOFS you’re good.
  • It’s policy based and can apply to virtual machines, groups of virtual machines a service or a tenant
  • The virtual disk is the level where the policy is set & enforced.
  • Storage performance will dynamically adjust to meet the policies & when tied the performance will be fairly distributed.
  • You can monitor all this.

It’s right there in the OS.

Storage Replica

This gives us “storage-agnostic, block-level, synchronous replication between servers for disaster recovery, as well as stretching of a failover cluster for high availability. Synchronous replication enables mirroring of data in physical sites with crash-consistent volumes ensuring zero data loss at the file system level. Asynchronous replication allows site extension beyond metropolitan ranges with the possibility of data loss.”

Look for Hyper-V we already had Hyper-V replica (which is also being improved), but for other workloads we still rely on the storage vendors or 3rd party solutions. But now I can have my storage replicas for service protection and continuity out of the box with Windows.  WOW!

and as we read on ..

  • Provide an all-Microsoft disaster recovery solution for planned and unplanned outages of mission-critical workloads.
  • Use SMB3 transport with proven reliability, scalability, and performance.
  • Stretch clusters to metropolitan distances.
    Use Microsoft software end to end for storage and clustering, such as Hyper-V, Storage Replica, Storage Spaces, Cluster, Scale-Out File Server, SMB3, Deduplication, and ReFS/NTFS.
  • Help reduce cost and complexity as follows:

Hardware agnostic, with no requirement to immediately abandon legacy storage such as SANs.

Allows commodity storage and networking technologies.
Features ease of graphical management for individual nodes and clusters through Failover Cluster Manager and Microsoft Azure Site Recovery.

Includes comprehensive, large-scale scripting options through Windows PowerShell.

  • Helps reduce downtime, and increase reliability and productivity intrinsic to Windows.
  • Provide supportability, performance metrics, and diagnostic capabilities.

I have gotten this to work in the lab with some trial and error but this is the Technical Preview, not a finish product. If they continue along this path I’m pretty confident we’ll have functional & operational viable solution by RTM. Just think about the possibilities this brings!

Storage Spaces

Now I have not read much on Storage Space in vNext yet but I think its safe to assume we’ll see major improvements there as well. Which leads me to reaffirm my blog posy here: TechEd 2013 Revelations for Storage Vendors as the Future of Storage lies With Windows 2012 R2

Microsoft is delivering more & great software defined storage inbox. This means cost effective yet very functional storage solutions. On top of that they put pressure on the market to deliver more value if they want to stay competitive. As a customer, whatever solution fits my needs the best, I welcome that. And as a consumer of large amounts of storage in a world where we need to spend the money where it matters most I like what I’m seeing.

Tip for Microsoft: configurability, reliability and EASY diagnostics and remediation are paramount to success. Sure some storage vendor solution aren’t to great on that front either but some are awesome. Make sure your in the awesome category. Make it a great user experience from start to finish in both deployment and operations.

Tip for you: If you’re not ready for prime time with Storage Spaces , SMB Direct etc … do what I’ve done. Use it where it doesn’t kill you if you hit some learning curves. What about storage spaces as a backup target where you can now replicate the backups of to your disaster recovery site?

What You Need To Hear, Not What You Want To Hear

The usual disclaimer covers this blog. Dilbert® Life series are humorous post on corporate culture from hell and dysfunctional organizations running wild. This can be quite shocking and sobering to those who take themselves to serious. So these blog posts need to be read with a healthy dose of humor and be put into perspective. If you can’t do that, leave now. If it hits home too hard, you have other problems. It could be that you don’t like what you see in this mirror. Or perhaps …

You’re so vain, you probably think this blog is about you
You’re so vain, I’ll bet you think this blog is about you
Don’t you? Don’t you?

Many thanks to Carly Simon’s “You’re so vain” Smile

Shopaholic Organizations

There is a shocking addiction to trying to buy ones way out of problems. If the service desk process sucks then you buy a CRM package. If this doesn’t do what you hoped out of the box, have it customized. You don’t have 100% IT automation? You need to buy a CMDB! Need to track changes? Go ITIL & do ITLM/ITSM all over the board. Projects don’t respect their boundaries? Hire some PRINCE expertise. Can’t keep up with all the project & resource management? Buy a ERP and integrate it with the project management software you’ve been abusing. You have no clue what to do next? Hire management consultants! We have one for every flavor of management. Your employees suck? Hire consultants. Slow applications? Buy flash only storage and 40Gbps switches. Your employees are disengaged? Get a coach, buy a team building experience and a 5$ pizza discount coupon as an “atta boy”. Maybe you could even gamify the company to success? And if you feel all alone and misunderstood you can join all the peer groups & professional organizations you can find to play that same broken record to each other over and over again whilst hoping you catch a break to a better gig.

Whatever the problem you’re facing, there is a product to buy and help to be hired. Like a true addict you keep using more of the same in the hope it will work. Nice twist on what Einstein called the definition of insanity. Yet why do so many people think it will help, all evidence to the contrary?

The obsessive and compulsive need to buy stuff to fix or even solve problems, needs, lack of skills, knowledge and insights is staggering. Sure the world is full of people and companies that will gladly take your money. Why? Well that’s their business model. The only aim is to separate you from your money. They’ll tell you they understand you, that they’ve helped hundreds of people and businesses like you. So they’ll sell you whatever it is they sell and they couldn’t care less if you’re still around next year. Until perhaps the moment in 18 months they know they can sucker you again. The only line of defense you have against that is your own good judgment. It’s not that all of them their products or services have no value at all. The better vendors will even walk away from an engagement when it not mutually beneficial. But the core of the problem is that you are having issues and that’s your inability to deal with problems that cannot be solved by buying something. It’s very much like a shopaholic.

It’s a business model for someone

The idea that there is a an easy fix to solve the issues your facing and make sure you can shine as a successful leader instead of being stuck in your current mess is very temping one. There is always someone who understands this. Who’s ready to step up and deliver. Which would be great if it was not for a few simple rules:

  • A fool and his money are easily separated. And if not, as long as the money is good enough they’ll put in more effort.
  • Your problems are internal, they are caused by you and need to be fixed by you. Any addiction to whatever (products, services, consulting, coaching) are actually keeping you away from the solution.

image

  • You as a manager, perhaps even a leader, will have to step up. Be all you can be and if that is not enough step aside. Do the latter yourself before it’s done to you, it’s less messy that way.

Listen, when the money is gone, all that is left are your internal resources, if you’re lucky. Acting as if they don’t matter means they won’t be very engaged. All budgets are limited, but that doesn’t mean that you need to be a scrooge. It means you need to create and build a capable organization even when budgets are plentiful that can stand on its own feet. One that is able to analyze and decide independently what it needs to do and act on that. Spend your money there. Otherwise as soon as you run out, you lose all your capabilities to act. It’s like a ship without power, on top of not even not having a rudder. You’re a drift, floating between the sharks that bled you dry.

Also, if all your organization knows what to do is hire & buy everything from others it can easily replace it with a cheaper one that’s optimized that model needing 40% to 50% less employees & managers. Pure substitution play. Game over. Economics 101.

You need to get a clue, make it happen, you and your team, no one else.  But it has to start with you. If you need coaches, consultants, products just to get started you’re not going to make it.

Ouch, that hurt!

Deep down you know the painful truth. While it would indeed be great if you’d be able to hire a coach, consultant or buy service, product that can take away your pains it doesn’t work that way. You cannot purchase those magical bottles of pixie dust or unicorn tears that can put the struggles and headaches behind you allowing you to solely focus on enjoying a successful business and be forever bliss.

image

I could tell you that you’re in luck as I have a nice stash of pixie dust bottles I can use in a pinch and for a price. But that’s not it. It’s experience, knowledge, having to work and live with solutions, see the good, the bad and the ugly of both marketing, “marchitecture” in combination with grand and hopefully realistic visions of analysis & architects what’s need. The only thing this has in common with pixie dust is that is doesn’t come cheap or easy neither, but it does work Winking smile

Too many times solutions are nothing but rehashed marketing & sales pitches that succeed due to a lack of skill on both sides. All kinds of schemes are used to justify them. They don’t achieve much at all. These are often self-serving “quick fixes” to something that is as structural & often over-hyped, over complicated problem serving some people agendas.

So you spend your money and for a little while you experience the illusion that you’ve solved something. But like any addict, you, the shopaholic, will return hard and fast to reality. Poorer and sadly none the wiser. You coast from purchase to purchase never breaking this destructive pattern. You like to fool yourself into believing that you’re investing instead of spending money because you see so many successful companies buy the same products or services. It’s kind of painful and sad to watch. Some of you will blame the market, incompetent employees or dishonest vendors, lack of commitment, disobedience. While all these factors do exist and play their role it’s not the real cause of your woes. The environment you operate in is no different for you or competitors. Sure there might be a hobby business around, run by the son of a super-rich business tycoon but that’s a minority. No, the playing field is the same, so could it, however painful that thought, be you, that’s not made of the right stuff?

What if despite all your best efforts and even some pixie dust you still have issues that are killing your performance?  You can suck it up and BS your way out. Say that what you did is the best in the world and nothing more can be done. Hire consultants to audit whatever it is you want to audit (or whoever you want to put in their place if you’re really political), blame you predecessor, the lack of (upper) management vision or the current sun spots cycle. You can also really dive in and pint point where the issues are. But that’s hard, very hard. A lot harder than buying a vile of unicorn tears which seems the missing ingredient in any unrealistic project, overly ambitious architecture or design. It’s horribly difficult to obtain because it is scarce beyond imagination.

image

I’ll make you a deal. While I possess some flasks, they are the most expensive substance ever to come by. So if you require the tears of a unicorn, you’re going to need truck loads with money of large denomination kind.

But there are no unicorn tears. YOU will need to fix your problems. Forget about buying products, that’s in essence automation and optimization. If you do that to a problem you only make it bigger and worse faster. Forget about coaches and consultants, they’ll only enable you to move faster and more targeted if you know the goal, that is. They will not solve your problems. That’s your job.

Don’t try to improve things with tools and services until you really know what’s wrong. Look very deep, hard and honest at your company, your managerial results and your actions. If you only find you do things to save your own behind, cover your back and hopefully move ahead you’re not fit to lead anything at all and you’re a much a strategist as my hamster. But in defense of my hamster: he lacks any ambition.  As a leader / manager you should care a bit more. Action is needed, from you. Lip service is useless. Talk is cheap. Fear kills. Deflecting decisions and responsibility makes you lose all credibility. If you care, act like it. If you don’t care no one else will for sure. If you can’t be bothered to do the hard work, no one will. You can’t lead from behind.

So what needs to be done?

Stop what you’re doing right now. Observe, orient, decide, and act (OODA) and see the progress of intelligent decisions and watch how money invested differs in results so much from money spent. There is no substitute. You don’t need tools, coaches, taskforces, committees and services. Those are only for amplification, they are force multipliers and that’s great as long as you don’t apply them to your problems. Hard as it may sound, its (free) advise that you won’t get from a sales person. You cannot avoid your responsibilities.

The eyes of the world are upon you

You brought this on yourself. You stepped on the plate as a leader. So yes, your employees are watching and they don’t miss much what affects them. I know employees can act very entitled and be a major pain in the proverbial behinds, but this discussion isn’t about that. Do you want to know why they doubt you, don’t follow you, ignore or possibly even oppose you? Because you show no leadership and do not portray any sign of competence or insight. For the good of the company and themselves they do what they need to, with or without you. No one goes over the top anymore at the blow of the whistle. So don’t pull rank, instead try to become credible.

Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX

Introduction

I was deploying a bunch of PowerConnect 2808 switches that needed to provide connectivity to multiple VLANs  (Training, Guest, …)  in classrooms. I should have figured it out before I got there with my “assumption” based quick configuration loaded on the switches if I had just refreshed my insights in how the PowerConnect family of switches work. Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX series switch differs a bit from the higher-end PowerConnect.

Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX
Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX

PowerConnect port mode refresher

So before we go on, here are the basics on switch port (or LAG) modes in the PowerConnect family. Please realize that switch behavior (especially for trunk mode in this context) has changed over time with more recent switches/firmware. But the current state of affairs is as follows (depending on what model & firmware you have behavior differs a bit).You can put your port or LAG in the following 3 (main) modes:

Access

The port belongs to a single untagged VLAN. When a port is in Access mode, the packet types which are accepted on the port cannot be designated. Ingress filtering cannot be enabled/disabled on an access port. So only untagged received traffic is allowed and all transmitted traffic is untagged. The setting of the port determines the VLAN of traffic. Tagged received traffic is dropped. Basically, this is what you set your ports for client devices to (printer, PC, laptop, NAS).

Trunk

In older versions this means that ALL transmitted traffic is tagged.  That’s easy. Tagged received traffic is dropped if doesn’t belong to one of the defined VLAN on the trunk. In more recent switches/firmware untagged received traffic is dropped but for one VLAN, that can be untagged and still be received. Which is nice for the default VLAN and makes for a better compatibility with other switches.

General

You determine what the rules are. You can configure it to transmit tagged or untagged traffic per VLAN. Untagged received traffic is accepted and the PVID determines the VLAN it is tagged with.  Tagged received traffic is dropped if doesn’t belong to one of the defined VLANs.

Also see this DELL link PowerConnect Common Questions Between Access, General and Trunk mode

Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX

These are good switches for their price point & use cases. Just make sure you buy them for the right use case. There is only one thing I find unforgiving in this day and age: the lack of SSH/HTTPS support for management.

Go ahead fire up a 2808 and take a look at the web interface and see what you can configure. In contrast with the PC54XX/55XX etc. Series you cannot set the port mode it seems. So how can this switch accommodate trunks/general/access modes at all. Well it’s implied in the configuration of ports that seem to be set in general mode by default and you cannot change that. The good news is that with the right setting a port in general mode behaves like a port in access or trunk mode. How? Well we follow the rules above.

So we assume here that a port is in general mode (can’t be changed). But we want trunk mode, so how do we get the same behavior? Let’s look at some examples in speudo CLI. (It’s web GUI only device).

Example 1: Classic Trunk = only defined tagged traffic is accepted. All untagged traffic is dropped

switchport mode trunk
switchport trunk allowed vlan add 9, 20

So we can have the same behavior is general mode using

switchport mode general
switchport general allowed vlan add 9, 20 tagged
switchport general pvid 4095   

The PVID  of 4095 is the industry standard discard VLAN, it assign this VLAN to all untagged traffic which is dropped. Ergo this is the same as the trunk config above!

Example 2: Modern Trunk = only defined tagged traffic and one untagged VLAN is accepted

switchport mode trunk
switchport trunk allowed vlan add 9, 20
switchport trunk allowed vlan add 1 untagged

So we can have the same behavior is general mode using

switchport mode general
switchport general allowed vlan add 9, 20 tagged
switchport general pvid 1  

This example is what we needed in the classroom. And is basically what you set with the GUI. So far so good. But we ran into an issue with connectivity to the access ports in VLAN 9 and VLAN 20. Let’s look at that in the next Example

Example 3: Access port mode = only one untagged VLAN is accepted

switchport mode access
switchport access vlan 9

Switchport mode general
switchport general allowed vlan add 9 untagged
switchport general pvid 9

If you’re accustomed to the higher end PC switches you define the port in access mode and add the VLAN of you choice untagged. That’s it. Here the mode is general and can’t be changed meaning we need to set the PVID to 9 so all untagged traffic is indeed tagged with VLAN 9 on the port.

Setting Up an uplink between a PowerConnect 5548 and a 2808

Here’s the normal deal with higher range series of PowerConnect switches: you normally use the port mode to define the behavior and in our case we could go with a trunk or general mode. We use trunk, leave the native VLAN for the one untagged VLAN and add 9 and 20 as tagged VLANs.

The “trunk” port of LAG is left on the default PVID

Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX

So an “access” port for VLAN 9 is is achieved by setting the PVID to 9

And an “access” port for VLAN 20 is achieved by setting the PVID to 20

While the VLAN  membership settings are what you’d expect them to be like on the higher end PowerConnect models:

First, VLAN 1 (native),

Setting Up A Uplink (Trunk/General) With A Dell PowerConnect 2808 or 28XX

then LAN 9 (Corp),

and finally, VLAN 20 (Guest)

Conclusion

If it’s the first time configuring a PC2808 you might totally ignore the fact that needed to do some extra work to make traffic flow. There is no selection of access/general/trunk on a PowerConnect 2808. The port or LAG is “implicitly” set to general. The extra settings of the PVID and adding tagged/untagged VLANs will make it behave as general, trunk or access.

  • Set any other VLAN than the default 1 to tagged on the port or LAG you’ll use as uplink. So far things are quite “standard PowerConnect”.
  • You set the VLAN membership of your “access” ports to untagged to the VLAN you want them to belong to.
  • After that in on the “access” ports you set the PVID to the VLAN you want the port to belong to. If you do not do this the port still behaves as if it’s a VLAN 1 port. It will not get a DHCP address for that VLAN but for for the the one on VLAN 1 if there  is one, or, if you use a static IP address for the subnet of a VLAN on that port you won’t have connectivity as it’s not set to the right VLAN.

The reason we used the PowerConnect 2808 series here is that they needed silent ones (passive cooling) in the training rooms. Multiple ones to avoid too many cables running around the place. That was the outcome of 2 minutes at the desk of the project manager’s quick fix to a changed requirement. The real solution of cause would have been to get 24+ outlets to the room in the correct places and add 24+ ports to the normal switch count in the hardware analysis for the building solution. But after the facts, you have to roll with the flow.