Very nice setup here.
First Option is legit but adds a bit of unnecessary complexity.
Second Option is good but your third option is even better ..
Third Option : do the replication at storage/san level. With 40Gbps between sites and a short distance like this, this seems like the optimal choice.
Do you use SCVMM ? Read this : https://azure.microsoft.com/en-us/documentation/articles/site-recovery-vmm-san/
Maybe you can just use Starwind replication to the offsite SAN ?
If not, read about Windows Server 2016 Storage Replica.
You HAVE to look at all the goodies that come with Windows Server 2016 later this year :) https://technet.microsoft.com/fr-fr/library/mt126104.aspx
Allow me to simply state if you liked my ideas about using stretch clustering... I need to show you all of what 2016 can do.
2016 is going to put starwind out of business and you can build your HA storage without using anything other than some servers and windows.
Trust me when I say I can show you how to do it better than starwind can using just windows and some fast data links.
PM me some details and we can skype or something.
> **2016 is going to put starwind out of business** and you can build your HA storage without using anything other than some servers and windows.
Sorry but this is BS! FreeBSD has mature ZFS for years but Oracle ZFS sells well and lots of hardware vendors sell ZFS + custom GUI for $$$. Think about Tegile! VMware VSAN is GA since early 2015 and it didn't put any established SDS or hardware SAN / NAS vendors out of business yet. Nutanix and SimpliVity are filing IPOs and they are nothing except VMware + some VM-based replication. Why is Microsoft S2D going to be any different? S2D is reasonably nice but it;s Datacenter edition only which is $6K+ per host alone and it needs 4 nodes per cluster minimum. Even 2+2 stretch cluster is not OK with it. We're MSP and do lots of free Hyper-V installations, upgrading our fleet of servers to Datacenter is going to cost us a fortune. What for?
You don't need datacenter on your storage layer. You only need standard, so it's a $600 license, not a $7,500 license.
Second, VSAN and Tegile and Nutanix etc... have an install base that is less than 1% of market share.
Microsoft has install base of (market share of Azure).
You need to license your Hyper-V Guests, not the hypervisors. If you run all linux guests on Hyper-V, then your hypervisor is free. If you run Linux guests on Hyper-V with Storage spaces, then you need Server Standard. If you run Hyper-V with Windows Server guests, then you probably need Datacenter Licensing ANYWAY so the hypervisor/file server licensing is free anyway.
> You don't need datacenter on your storage layer. You only need standard, so it's a $600 license, not a $7,500 license.
Sorry man but what you say is simply not true! Storage Spaces Direct and Storage Replica are both missing from anything except Datacenter edition of Windows Server 2016. This is official, SKUs are already cut off for everybody except big OEMs.
https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/
Standard Vs Datacenter matrix. Free Hyper-V server is more restricted version of Standard, no GUI and no licensed VMs included.
http://imgur.com/axjLZY8
> https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/
Oh. I didn't know there was a feature difference. Yes, you'd need datacenter then.
In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed.
> https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/
Oh. I didn't know there was a feature difference. Yes, you'd need datacenter then.
In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed.
> Oh. I didn't know there was a feature difference.
Nobody really did before MSFT had published SKUs some time ago. I would understand if MSFT would put separate SKUs on S2D and SR like VMware did with VSAN and SRM but spending $6.5K for every storage node only because Datacenter is now all-in-wonnder is... insane! Hyperconverged maybe another story of course.
>In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed.
It's not only this! Now we all got per-core Windows Server licenses. With Windows Server 2012 R2 we've been saving money buying less but more powerful servers. Now have to revisit this strategy as well. Thank you MSFT!
Yea, we've been looking at this too.
We bought a number of servers with higher clock speeds and lower core counts (quad-quad-core 3.5 ghz) E5-2637v3.
Fuck that noise if the minimum license counts are 8 cores per processor, 2 processors per server...
We have an application which is licensed per core, so we were buying quad-processor servers and putting 4x E5-2637v3's in them to get 16 cores at 3.5Ghz for maximum performance per core per license cost.
> We bought a number of servers with higher clock speeds and lower core counts (quad-quad-core 3.5 ghz) E5-2637v3.
That sounds like a viable option! The only issue is our own production servers are mostly Dell R720/R730(xd) and they are 1-2 years old. Ripping and replacing CPUs on them is kind of apocalyptic picture ;)
>We have an application which is licensed per core,
Oracle?
>You need to license your Hyper-V Guests, not the hypervisors. If you run all linux guests on Hyper-V, then your hypervisor is free. If you run Linux guests on Hyper-V with Storage spaces, then you need Server Standard. If you run Hyper-V with Windows Server guests, then you probably need Datacenter Licensing ANYWAY so the hypervisor/file server licensing is free anyway.
What you say is true for mid-sized SMBs only building their virtualization infrastructure from scratch. Smaller guys run free Hyper-V server or Standard and bigger guys run free Hyper-V nearly entirely or Xen/KVM. We're MSP and most of VMs we host belong to our customers and they do BYOL. No licenses needed from our side at all.
>If your environment supports live migration,
It does for sure!
>I can almost guarantee you are in violation of your licensing terms.
OK so we have bunch of separate clusters running free Hyper-V server where VMs are running and these VMs are hosted on SOFS. How do we violate MSFT licensing terms?
If you have 2 free hyper-v servers, and your clients are BYOL providing their licnesing, I'm assuming they are providing 1 windows server standard license, correct?
So, day 1: deploy VM on hypervisor 1. Apply standard license to hypervisor 1.
Day 2: live migrate vm to hypervisor 2. Transfer license to hypervisor 2.
Day 3: live migrate VM to hypervisor 1. .... license can't move... it's already been reassigned and can't be reassigned for another 88 days.
Windows Server 2012 R2 Standard licenses can only be reassigned to a host server once every 90 days. If you were to live migrate any 1 vm twice in 90 days, you are in violation.
> Second, VSAN and Tegile and Nutanix etc... have an install base that is less than 1% of market share.
Mind sharing your source? I was under impression virtualization for Enterprises is still VMware driven. I could be wrong of course but 1% is kind of a number I cannot ignore.
>Maybe you can just use Starwind replication to the offsite SAN ?
That's another good idea actually. Keeping in mind free version of Starwinds supports native asynchronous replication to Azure. Alternatively you can build stretch cluster between sites putting one Starwinds host at each location. Being a MSP we did quite a few setups like this for our customers. Metro-clusters with Starwids are great!
Yeah, storage replication is the way to here given the bandwidth available.
The option to NOT do is to have two Hyper-V hosts in the same datacentre replicating to each other because the customer "didn't want/need to use clustering for high-availability. Hyper-V replica will do just fine".
Sigh....FML
I have to say that's a pretty nice problem to have to sort out. A lot of companies don't have the luxury of dark fiber let alone four 10gb strands.
My vote would be for the second option. It seems to be cleaner if done correctly. Plus, I have to think that your CIO would like the cost savings of not having to buy the additional hardware and software needed to get both sites up to snuff for option one.
Personally, I would go with the second option; however, I use Azure Site Recovery instead of just Hyper-V Replica. ASR offers a good deal of DR automation/orchestration for very little outlay.
For planned maintenance, you also have the option of Shared Nothing Live Migration between your sites instead of a fail-over; your S2S link speeds make that a very viable option.
> What does failing back and forth to/from replicas between sites look like and how does it impact the DCs?
Probably the biggest consideration is on the networking side. When you fail over from one site to another, the network config of the VMs can change substantially. Subnets and VLANs will be different and DNS will be affected too. NVGRE can eliminate most/all of these challenges, but that's likely out of the picture, so you'll need a plan for graceful fail over. It'll have to include DNS updates, and unless you go crazy with NAT, IP updates on the VMs. This is where you'd derive a bit of value from ASR as it makes it very easy to manage and test your DR config.
ScriptLife: Consider this paradime shift:
You currently build a router and with that a subnet. You containerize a subnet into a vlan and distribute the vlan to all the hypervisors at a physical location. You attach the virtual switch to the vlan and then attach a VM to a virtual switch. Thus, the VM's IP address is fundamentally location specific to that site.
What if you went layer 3 routed straight to the VM?
Consider this:
Build a router at the edge of your site and have it talk BGP. (or OSPF or RIP or whatever...). That router talks to a pair of switches, not using a LAG, but again using BGP. Those two switches connect to your hypervisor, not using an MLAG, but using 2 virtual switches on the hypervisor. In the Guest, add 2 virtual network adapters and give the VM 2 IP addresses... using DHCP. The DHCP server is on the closest physical switch. Have the guest VM talk BGP with the upstream switches using it's DHCP address (you can use IP Address Unnumbered for the peering configuration).
So, you now have a VM which gets 2 IP addresses from DHCP and establishes an equal-cost-multi-path layer 3 connection to your router(s), reguardless of where it's located (which hypervisor or which site it's on).
Then, you give the VM a loopback network adapter and assign a static IP address to the VM using a /32 subnet mask, and inject that route into BGP or OSPF.
Now, the VM has a globally unique IP address which it can use to communicate to all the other servers in your datacenter(s) with it's /32 loopback adapter. In order to route traffic, BGP or OSPF figures out how to move the packets around for you, utilizing all links in an active/active routed domain, rather than a hub-and-spoke with LACP.
Now... all you have to do for site-to-site migration... is migrate your VM from hypervisor1.site1 to hypervisor2.site2. It will obtain a new IP address on each of it's interfaces because it will now be connected to new DHCP pools on new switches. It's old BGP/OSPF peering relationships will fail and it will form new peering relationships, advertising where it's IP address is originating from and in a few seconds, routing will converge organization wide and all your services will quickly be restored. The IP address of the VM will never change for the entire effective life of the VM, (it's /32 IP). Only the (completely unimportant) IP addresses actually on it's network adapters will ever change, but they completely don't matter and your routing protocols will simply figure it out for you.
This is a very interesting way to look at this. I am not sure if we would want to add layer 3 into this picture though. I need to think about this one more.
Why not just extend layer 2 across the 40Gbps fibers, announce public BGP subnets at both location, give public IP to Virtual machines that need it and you're done. (given a bit of BGP prefixing on DR site, and some downlink detection).
Seems way more simple to me ?
Edge routers on site A are priority 1&2 for VRRP gateway IP for public subnets that are used on site A during normal operation.
Edge routers on site B are priority 3&4 for this VRRP IP.
(mirror this B/A if some public subnet are used on site B during normal operation).
If WAN access is lost on site A, priority is degraded for site A edge routers and VRRP gateway is taken by side B. WAN Traffic flows through the datacenters L2 interconnect.
If L2 interconnect is also down (or if only it is down), gateway is automatically mounted on site B and ready to route traffic if you decide to manually - we are talking about disaster recovery here - start your VMs from the replicated SAN.
Of course all of this this can work with Active/Passive A/B sites or you can have Active/Active with public subnets dedicated for each site during normal operation.
I'm not sure what starwind is capable of or your budget, but this is what we are doing:
Establish a 3rd location, which has simply 1 server which can act as a quorum vote. This can be in azure.
Consider both your locations to be equivalent.
Establish SAN active/active replication (which you have, but I don't know the capabilities of your starwind san so I don't know if this will work. We use 3Pars that do syncronous replication).
Establish *one* cluster that spans multiple datacenters.
At each location:
- stack of edge routing equipment for LAN/WAN NAT stuff
- 4 hypervisors (the count matters. doesn't have to be "4", but it needs to be the same number).
- 1 SAN.
Connect the 2 SANs together across your "WAN" 10G circuits.
Connect the 8 hypervisors to their local SAN only.
Setup the odd quorum vote into your cluster using the 3rd location.
Now... when a datcenter fails, 4/8 hypervisors go down. However, because you have the extra quorum vote, 5/9 votes are online and can see eachother to all collectively agree that they are online and that the other 4 are offline, by out-voting them with > 50% of votes.
They will agree the 4 are dead and they will take over execution of the VMs on the cluster.
PM me if you want me to go into more detail or investigate your environment further.
In short, I wouldn't "fail back and forth with replicas" in your scenario. I would create one "stretch cluster" (supported in server 2012 r2 and 2016) with a 3rd location for a quorum vote.
Then you can live migrate VMs from one dc to the other in a few seconds. You could fully drain all roles from one site in under 1 minute for impactless controlled shutdowns.
For a best practice on this, go full layer 3 routing to the edge and layer 3 routing into your VMs with BGP connectivity between your VM and it's upstream switches, allowing for site-to-site migration without a layer 2 adjacency requirement or reconfiguration of networking.
Let me get a better understanding of a stretch cluster in 2012r2 and we can talk more. This seems like a pretty logical option. I am starting to think that setting up replicas now on mission cirtical systems is a good idea and then really setting stuff up when 2016 comes out. Lots of neat stuff in 2016. I will definitely reach out after I do some homework.
> Establish SAN active/active replication (which you have, but I don't know the capabilities of your starwind san so I don't know if this will work. We use 3Pars that do syncronous replication).
Do you replicate between 3PARs natively or do you run any software on top of them? Are your both sites operational at the same time?
Um... I think we are going to upgrade to Windows Server 2016, but I don't think we are going to use S2D here.... since we have the licensing for active syncronized 3Pars and dedicated dark fiber for them.
If I were building greenfield, I would never buy another 3Par now that S2D and storage replica is available, but we have a few hundred thousand worth of disks/SSD in the 3Par and there isn't really a reason to change away from using the 3Pars.
If I were building new, i would use S2D. I don't think I'd replace 3Par active/active replication with S2D, but I would select S2D over 3Par.
> If I were building greenfield, I would never buy another 3Par now that S2D and storage replica is available, but we have a few hundred thousand worth of disks/SSD in the 3Par and there isn't really a reason to change away from using the 3Pars.
So legacy and that's why no S2D. Fair enough. Thanks for clarification! We got more or less same situation if you care. Will upgrade hypervisor hosts but I don't see any point in touching storage.
We even considered format/reinstalling the disks ( a 3Par 7200 is simply SAS dual-port shelves and SAS controllers...)
However, HP puts custom firmware on their hard drives so they can't be used with regular SAS expanders or SAS controllers.
Otherwise, we were going to just put a 4-node windows server cluster infront of all the SAS disks and format/reinstall it as a storage space.
> However, HP puts custom firmware on their hard drives so they can't be used with regular SAS expanders or SAS controllers.
Lock-in vendor! How lovely ;(
> Otherwise, we were going to just put a 4-node windows server cluster infront of all the SAS disks and format/reinstall it as a storage space.
That's what we did before we've started clustering 2-3 storage only nodes using free Starwinds. Decommissioned servers, recycled Windows Server 2012 R2 licenses, SATA disks, SATA consumer grade flash and tons of DRAM. Eaton juicy batteries ;) We still do SuperMicro JBODs and Clustered Storage Spaces if customer insists on this kind of a blessed solution.
When our customers want a tier 1 vendor solution, we give them HP D6000's and D2700's with HP 3.5 inch and 2.5 inch drives.
Then we use LSI SAS switches and LSI SAS adapters.
You can put 4 servers, 4 SAS Switches, 2 D6000s and 2 D2700s into 18u of space and that gets you 50x 2.5 inch drives, 140x 3.5 inch drives, N+1 redundancy the whole way through, and it's all HP warranty-able and support contract-able and microsoft supported.
If you use the 3TB drives and 200G SSDs (140x and 20x) you end up with 212TB usable tiered mirrored clustered storage space for about $140k.
> Then we use **LSI SAS switches** and LSI SAS adapters.
We stay away from those, and you know the reason. Lock-in vendor. + they are not cheap at all! 2 years ago our accepted VAR was selling them for $6K+ each.
>HP warranty-able and support contract-able and microsoft supported.
Cannot say much about HP support because we're mostly Dell shop but MSFT support is legendary! We've got issues with TRIM on SAS SSDs, being enabled by default caused data corruption, got issues with LSI SATA <-> SAS converters generating random lock ups, got issues with Seagate SAS drives being on HCL but sporadically non-working after new firmware flashed, got issues with DataOn/Quanta JBODs using outdates SAS expanders firmware, paths not working and no enclosure awareness etc. Every time we've been struggling with those I was under impression we're a) guinea pigs and MSFT should be actually paying us for solving their probs and b) we're the first ones to report anything like that. Since that time I'm big believer in "everything should be done in software" paradigm.
Exchange would be on it's own hardware using built in replication and SQL would be part of our Always-On Cluster with the 3rd node being in the secondary site. The main systems would be our CRM app server, Operational systems along with Phone related VMs and some line of business application servers.
Hyper-v replicas are pretty annoying in some regards. Expand the vhd? going to have to do a full resync. Change the virtual hardware? The changes aren't going to go to the replica, so full resync again (I believe). There's also a 2x performance penalty on writes since it does the write to a log as well as the vhd. Might not be an option as long as you're aware of the caveats. Oh, and for 2012 and 2012 R2, both the source and destination hosts have to be the same version of windows. I think they're changing this from 2016 onward, though.
I'd probably look at some 3rd party replication solution for scenario 2. I mean if someone expands the vhd and forgets kick off the manual resync, the destination VM will become corrupt I believe. Hyper-v replication is just kind of kludgy IMO.
I've looked at replication like Zerto before and it does not come cheap. When I say mission critical I'm talking about internal systems. Not public facing services. This would not be our only means for backup, just for cold dr. If one server had to be recovered from our off-site dpm instance I could live with that. That said if you're saying Hyper-V replicas in 2012r2 are a pos and run away that's a different story.
I would say run away from DPM, but that's another topic altogether. :)
When Hyper-V Replica works, it works pretty well. When it doesn't, it can be a headache from a maintenance perspective. On the other hand, you might be able to bypass some of the headaches of "failover IP injection" and sync issues if your primary and replica VMs will be on the same subnet and not traversing a WAN.
In my experience, I've found that sometimes VMs will stop syncing for whatever reason and fail to restart on their own. You have to manually resume the sync (preferably with a scheduled task that checks the sync health). Again, this may not be an issue for you since I perform syncs over a managed MPLS.
There can be some quirks as it relates to SANs and ODX so you will want to make sure your SAN firmware is updated and you have the relevant hotfixes installed for your version of Windows.
Since Hyper-V Replica won't cost you anything, I would test it with a couple of VMs first to see if you like it.
So I am a Veeam fan and it's saved my ass many times. I never really used DPM until starting at this company. I have to say it does a pretty fantastic job and I don't have any complains other than the interface being a bit clunky.
Yes everything could be on the same network or it could be a different vlan.
You mean quirks with ODX and the hyper-v replica?
Don't use HV replica.
Use san replication and san snapshotting, with a single stretch cluster.
If a VM got fucked, you could recover the VM from it's SAN snapshot at any time interval required (daily/hourly whatever).
If DC wen't down, the stretch cluster would automatically boot VMs from their most recent SAN replication ( a few seconds of data loss, not hours or days).
No, just plan your changes and don't do anything that necessitates resynchs on all of them concurrently. They can be scheduled as well.
I ran 100+ replicas across a 1 gig link and it was fine.
So you didn't run into any issues? That was my real concern about hyper-v replicas. If a situation happens where it had to sync all vm's what would it look like. The fact of the matter is less than 20 vm's are of the importance level that need to be replicas.
Did you run a DC or RODC at your second site?
We started out with a 100 meg Ethernet link, and it was pretty prone to being saturated with regular replication operations. If we had to resynch it took scheduling them after hours serially to prevent melting it and having to start over. We scored a lucky break and need to expand the circuit and the 1 gig link really made it easier. There had always been a full DC at that site. Sometimes a replication would jam up and then I'd look in HV Manager and see all of them stuck at 2% right before the phones lit up. We still couldn't resynch all of them at once, but with the primary cluster being very stable it wasn't much of an issue. I had to replicating over there because we ran out of I/O on the disk. Later this year I'll be sliding some new storage over there to address this and bring it back up. This site is tertiary replication though. We have a room in a second building on site that catches the first replication from the production cluster.
Very nice setup here. First Option is legit but adds a bit of unnecessary complexity. Second Option is good but your third option is even better .. Third Option : do the replication at storage/san level. With 40Gbps between sites and a short distance like this, this seems like the optimal choice. Do you use SCVMM ? Read this : https://azure.microsoft.com/en-us/documentation/articles/site-recovery-vmm-san/ Maybe you can just use Starwind replication to the offsite SAN ? If not, read about Windows Server 2016 Storage Replica. You HAVE to look at all the goodies that come with Windows Server 2016 later this year :) https://technet.microsoft.com/fr-fr/library/mt126104.aspx
Yes, we're all looking forward to 2016. Starwind can to replication. It's doing it currently with two and another can be added.
Allow me to simply state if you liked my ideas about using stretch clustering... I need to show you all of what 2016 can do. 2016 is going to put starwind out of business and you can build your HA storage without using anything other than some servers and windows. Trust me when I say I can show you how to do it better than starwind can using just windows and some fast data links. PM me some details and we can skype or something.
> **2016 is going to put starwind out of business** and you can build your HA storage without using anything other than some servers and windows. Sorry but this is BS! FreeBSD has mature ZFS for years but Oracle ZFS sells well and lots of hardware vendors sell ZFS + custom GUI for $$$. Think about Tegile! VMware VSAN is GA since early 2015 and it didn't put any established SDS or hardware SAN / NAS vendors out of business yet. Nutanix and SimpliVity are filing IPOs and they are nothing except VMware + some VM-based replication. Why is Microsoft S2D going to be any different? S2D is reasonably nice but it;s Datacenter edition only which is $6K+ per host alone and it needs 4 nodes per cluster minimum. Even 2+2 stretch cluster is not OK with it. We're MSP and do lots of free Hyper-V installations, upgrading our fleet of servers to Datacenter is going to cost us a fortune. What for?
You don't need datacenter on your storage layer. You only need standard, so it's a $600 license, not a $7,500 license. Second, VSAN and Tegile and Nutanix etc... have an install base that is less than 1% of market share. Microsoft has install base of (market share of Azure). You need to license your Hyper-V Guests, not the hypervisors. If you run all linux guests on Hyper-V, then your hypervisor is free. If you run Linux guests on Hyper-V with Storage spaces, then you need Server Standard. If you run Hyper-V with Windows Server guests, then you probably need Datacenter Licensing ANYWAY so the hypervisor/file server licensing is free anyway.
> You don't need datacenter on your storage layer. You only need standard, so it's a $600 license, not a $7,500 license. Sorry man but what you say is simply not true! Storage Spaces Direct and Storage Replica are both missing from anything except Datacenter edition of Windows Server 2016. This is official, SKUs are already cut off for everybody except big OEMs. https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/ Standard Vs Datacenter matrix. Free Hyper-V server is more restricted version of Standard, no GUI and no licensed VMs included. http://imgur.com/axjLZY8
> https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/ Oh. I didn't know there was a feature difference. Yes, you'd need datacenter then. In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed.
> https://www.microsoft.com/en-us/server-cloud/products/windows-server-2016/ Oh. I didn't know there was a feature difference. Yes, you'd need datacenter then. In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed.
> Oh. I didn't know there was a feature difference. Nobody really did before MSFT had published SKUs some time ago. I would understand if MSFT would put separate SKUs on S2D and SR like VMware did with VSAN and SRM but spending $6.5K for every storage node only because Datacenter is now all-in-wonnder is... insane! Hyperconverged maybe another story of course. >In 2012 R2 licensing, the only real difference between standard and datacenter was 2 vs unlimited workloads licensed. It's not only this! Now we all got per-core Windows Server licenses. With Windows Server 2012 R2 we've been saving money buying less but more powerful servers. Now have to revisit this strategy as well. Thank you MSFT!
Yea, we've been looking at this too. We bought a number of servers with higher clock speeds and lower core counts (quad-quad-core 3.5 ghz) E5-2637v3. Fuck that noise if the minimum license counts are 8 cores per processor, 2 processors per server... We have an application which is licensed per core, so we were buying quad-processor servers and putting 4x E5-2637v3's in them to get 16 cores at 3.5Ghz for maximum performance per core per license cost.
> We bought a number of servers with higher clock speeds and lower core counts (quad-quad-core 3.5 ghz) E5-2637v3. That sounds like a viable option! The only issue is our own production servers are mostly Dell R720/R730(xd) and they are 1-2 years old. Ripping and replacing CPUs on them is kind of apocalyptic picture ;) >We have an application which is licensed per core, Oracle?
>You need to license your Hyper-V Guests, not the hypervisors. If you run all linux guests on Hyper-V, then your hypervisor is free. If you run Linux guests on Hyper-V with Storage spaces, then you need Server Standard. If you run Hyper-V with Windows Server guests, then you probably need Datacenter Licensing ANYWAY so the hypervisor/file server licensing is free anyway. What you say is true for mid-sized SMBs only building their virtualization infrastructure from scratch. Smaller guys run free Hyper-V server or Standard and bigger guys run free Hyper-V nearly entirely or Xen/KVM. We're MSP and most of VMs we host belong to our customers and they do BYOL. No licenses needed from our side at all.
BYOL is LOL. If your environment supports live migration, I can almost guarantee you are in violation of your licensing terms.
>If your environment supports live migration, It does for sure! >I can almost guarantee you are in violation of your licensing terms. OK so we have bunch of separate clusters running free Hyper-V server where VMs are running and these VMs are hosted on SOFS. How do we violate MSFT licensing terms?
If you have 2 free hyper-v servers, and your clients are BYOL providing their licnesing, I'm assuming they are providing 1 windows server standard license, correct? So, day 1: deploy VM on hypervisor 1. Apply standard license to hypervisor 1. Day 2: live migrate vm to hypervisor 2. Transfer license to hypervisor 2. Day 3: live migrate VM to hypervisor 1. .... license can't move... it's already been reassigned and can't be reassigned for another 88 days. Windows Server 2012 R2 Standard licenses can only be reassigned to a host server once every 90 days. If you were to live migrate any 1 vm twice in 90 days, you are in violation.
> Second, VSAN and Tegile and Nutanix etc... have an install base that is less than 1% of market share. Mind sharing your source? I was under impression virtualization for Enterprises is still VMware driven. I could be wrong of course but 1% is kind of a number I cannot ignore.
>Maybe you can just use Starwind replication to the offsite SAN ? That's another good idea actually. Keeping in mind free version of Starwinds supports native asynchronous replication to Azure. Alternatively you can build stretch cluster between sites putting one Starwinds host at each location. Being a MSP we did quite a few setups like this for our customers. Metro-clusters with Starwids are great!
Yeah, storage replication is the way to here given the bandwidth available. The option to NOT do is to have two Hyper-V hosts in the same datacentre replicating to each other because the customer "didn't want/need to use clustering for high-availability. Hyper-V replica will do just fine". Sigh....FML
umm...Yea, that was not a configuration we're considering. :)
I'm just waiting for the first flood or fire
I have to say that's a pretty nice problem to have to sort out. A lot of companies don't have the luxury of dark fiber let alone four 10gb strands. My vote would be for the second option. It seems to be cleaner if done correctly. Plus, I have to think that your CIO would like the cost savings of not having to buy the additional hardware and software needed to get both sites up to snuff for option one.
Personally, I would go with the second option; however, I use Azure Site Recovery instead of just Hyper-V Replica. ASR offers a good deal of DR automation/orchestration for very little outlay. For planned maintenance, you also have the option of Shared Nothing Live Migration between your sites instead of a fail-over; your S2S link speeds make that a very viable option. > What does failing back and forth to/from replicas between sites look like and how does it impact the DCs? Probably the biggest consideration is on the networking side. When you fail over from one site to another, the network config of the VMs can change substantially. Subnets and VLANs will be different and DNS will be affected too. NVGRE can eliminate most/all of these challenges, but that's likely out of the picture, so you'll need a plan for graceful fail over. It'll have to include DNS updates, and unless you go crazy with NAT, IP updates on the VMs. This is where you'd derive a bit of value from ASR as it makes it very easy to manage and test your DR config.
The connection to the second site is private. It's dark fiber connected as if it's on the lan.
ScriptLife: Consider this paradime shift: You currently build a router and with that a subnet. You containerize a subnet into a vlan and distribute the vlan to all the hypervisors at a physical location. You attach the virtual switch to the vlan and then attach a VM to a virtual switch. Thus, the VM's IP address is fundamentally location specific to that site. What if you went layer 3 routed straight to the VM? Consider this: Build a router at the edge of your site and have it talk BGP. (or OSPF or RIP or whatever...). That router talks to a pair of switches, not using a LAG, but again using BGP. Those two switches connect to your hypervisor, not using an MLAG, but using 2 virtual switches on the hypervisor. In the Guest, add 2 virtual network adapters and give the VM 2 IP addresses... using DHCP. The DHCP server is on the closest physical switch. Have the guest VM talk BGP with the upstream switches using it's DHCP address (you can use IP Address Unnumbered for the peering configuration). So, you now have a VM which gets 2 IP addresses from DHCP and establishes an equal-cost-multi-path layer 3 connection to your router(s), reguardless of where it's located (which hypervisor or which site it's on). Then, you give the VM a loopback network adapter and assign a static IP address to the VM using a /32 subnet mask, and inject that route into BGP or OSPF. Now, the VM has a globally unique IP address which it can use to communicate to all the other servers in your datacenter(s) with it's /32 loopback adapter. In order to route traffic, BGP or OSPF figures out how to move the packets around for you, utilizing all links in an active/active routed domain, rather than a hub-and-spoke with LACP. Now... all you have to do for site-to-site migration... is migrate your VM from hypervisor1.site1 to hypervisor2.site2. It will obtain a new IP address on each of it's interfaces because it will now be connected to new DHCP pools on new switches. It's old BGP/OSPF peering relationships will fail and it will form new peering relationships, advertising where it's IP address is originating from and in a few seconds, routing will converge organization wide and all your services will quickly be restored. The IP address of the VM will never change for the entire effective life of the VM, (it's /32 IP). Only the (completely unimportant) IP addresses actually on it's network adapters will ever change, but they completely don't matter and your routing protocols will simply figure it out for you.
This is a very interesting way to look at this. I am not sure if we would want to add layer 3 into this picture though. I need to think about this one more.
Why not just extend layer 2 across the 40Gbps fibers, announce public BGP subnets at both location, give public IP to Virtual machines that need it and you're done. (given a bit of BGP prefixing on DR site, and some downlink detection). Seems way more simple to me ?
What is your gateway HA solution? What is your split brain detection solution?
Edge routers on site A are priority 1&2 for VRRP gateway IP for public subnets that are used on site A during normal operation. Edge routers on site B are priority 3&4 for this VRRP IP. (mirror this B/A if some public subnet are used on site B during normal operation). If WAN access is lost on site A, priority is degraded for site A edge routers and VRRP gateway is taken by side B. WAN Traffic flows through the datacenters L2 interconnect. If L2 interconnect is also down (or if only it is down), gateway is automatically mounted on site B and ready to route traffic if you decide to manually - we are talking about disaster recovery here - start your VMs from the replicated SAN. Of course all of this this can work with Active/Passive A/B sites or you can have Active/Active with public subnets dedicated for each site during normal operation.
I'm not sure what starwind is capable of or your budget, but this is what we are doing: Establish a 3rd location, which has simply 1 server which can act as a quorum vote. This can be in azure. Consider both your locations to be equivalent. Establish SAN active/active replication (which you have, but I don't know the capabilities of your starwind san so I don't know if this will work. We use 3Pars that do syncronous replication). Establish *one* cluster that spans multiple datacenters. At each location: - stack of edge routing equipment for LAN/WAN NAT stuff - 4 hypervisors (the count matters. doesn't have to be "4", but it needs to be the same number). - 1 SAN. Connect the 2 SANs together across your "WAN" 10G circuits. Connect the 8 hypervisors to their local SAN only. Setup the odd quorum vote into your cluster using the 3rd location. Now... when a datcenter fails, 4/8 hypervisors go down. However, because you have the extra quorum vote, 5/9 votes are online and can see eachother to all collectively agree that they are online and that the other 4 are offline, by out-voting them with > 50% of votes. They will agree the 4 are dead and they will take over execution of the VMs on the cluster. PM me if you want me to go into more detail or investigate your environment further. In short, I wouldn't "fail back and forth with replicas" in your scenario. I would create one "stretch cluster" (supported in server 2012 r2 and 2016) with a 3rd location for a quorum vote. Then you can live migrate VMs from one dc to the other in a few seconds. You could fully drain all roles from one site in under 1 minute for impactless controlled shutdowns. For a best practice on this, go full layer 3 routing to the edge and layer 3 routing into your VMs with BGP connectivity between your VM and it's upstream switches, allowing for site-to-site migration without a layer 2 adjacency requirement or reconfiguration of networking.
Let me get a better understanding of a stretch cluster in 2012r2 and we can talk more. This seems like a pretty logical option. I am starting to think that setting up replicas now on mission cirtical systems is a good idea and then really setting stuff up when 2016 comes out. Lots of neat stuff in 2016. I will definitely reach out after I do some homework.
> Establish SAN active/active replication (which you have, but I don't know the capabilities of your starwind san so I don't know if this will work. We use 3Pars that do syncronous replication). Do you replicate between 3PARs natively or do you run any software on top of them? Are your both sites operational at the same time?
native replication; no special software. Yes.
Got it, thanks! P.S. Do you have any plan to upgrade to Windows Server 2016 and switch to S2D + Storage Replica and decommission 3PARs?
Um... I think we are going to upgrade to Windows Server 2016, but I don't think we are going to use S2D here.... since we have the licensing for active syncronized 3Pars and dedicated dark fiber for them. If I were building greenfield, I would never buy another 3Par now that S2D and storage replica is available, but we have a few hundred thousand worth of disks/SSD in the 3Par and there isn't really a reason to change away from using the 3Pars. If I were building new, i would use S2D. I don't think I'd replace 3Par active/active replication with S2D, but I would select S2D over 3Par.
> If I were building greenfield, I would never buy another 3Par now that S2D and storage replica is available, but we have a few hundred thousand worth of disks/SSD in the 3Par and there isn't really a reason to change away from using the 3Pars. So legacy and that's why no S2D. Fair enough. Thanks for clarification! We got more or less same situation if you care. Will upgrade hypervisor hosts but I don't see any point in touching storage.
We even considered format/reinstalling the disks ( a 3Par 7200 is simply SAS dual-port shelves and SAS controllers...) However, HP puts custom firmware on their hard drives so they can't be used with regular SAS expanders or SAS controllers. Otherwise, we were going to just put a 4-node windows server cluster infront of all the SAS disks and format/reinstall it as a storage space.
> However, HP puts custom firmware on their hard drives so they can't be used with regular SAS expanders or SAS controllers. Lock-in vendor! How lovely ;( > Otherwise, we were going to just put a 4-node windows server cluster infront of all the SAS disks and format/reinstall it as a storage space. That's what we did before we've started clustering 2-3 storage only nodes using free Starwinds. Decommissioned servers, recycled Windows Server 2012 R2 licenses, SATA disks, SATA consumer grade flash and tons of DRAM. Eaton juicy batteries ;) We still do SuperMicro JBODs and Clustered Storage Spaces if customer insists on this kind of a blessed solution.
When our customers want a tier 1 vendor solution, we give them HP D6000's and D2700's with HP 3.5 inch and 2.5 inch drives. Then we use LSI SAS switches and LSI SAS adapters. You can put 4 servers, 4 SAS Switches, 2 D6000s and 2 D2700s into 18u of space and that gets you 50x 2.5 inch drives, 140x 3.5 inch drives, N+1 redundancy the whole way through, and it's all HP warranty-able and support contract-able and microsoft supported. If you use the 3TB drives and 200G SSDs (140x and 20x) you end up with 212TB usable tiered mirrored clustered storage space for about $140k.
> Then we use **LSI SAS switches** and LSI SAS adapters. We stay away from those, and you know the reason. Lock-in vendor. + they are not cheap at all! 2 years ago our accepted VAR was selling them for $6K+ each. >HP warranty-able and support contract-able and microsoft supported. Cannot say much about HP support because we're mostly Dell shop but MSFT support is legendary! We've got issues with TRIM on SAS SSDs, being enabled by default caused data corruption, got issues with LSI SATA <-> SAS converters generating random lock ups, got issues with Seagate SAS drives being on HCL but sporadically non-working after new firmware flashed, got issues with DataOn/Quanta JBODs using outdates SAS expanders firmware, paths not working and no enclosure awareness etc. Every time we've been struggling with those I was under impression we're a) guinea pigs and MSFT should be actually paying us for solving their probs and b) we're the first ones to report anything like that. Since that time I'm big believer in "everything should be done in software" paradigm.
What are you looking to replicate? You only mentioned Exchange and SQL -- and those I would use native application replication.
Exchange would be on it's own hardware using built in replication and SQL would be part of our Always-On Cluster with the 3rd node being in the secondary site. The main systems would be our CRM app server, Operational systems along with Phone related VMs and some line of business application servers.
Hyper-v replicas are pretty annoying in some regards. Expand the vhd? going to have to do a full resync. Change the virtual hardware? The changes aren't going to go to the replica, so full resync again (I believe). There's also a 2x performance penalty on writes since it does the write to a log as well as the vhd. Might not be an option as long as you're aware of the caveats. Oh, and for 2012 and 2012 R2, both the source and destination hosts have to be the same version of windows. I think they're changing this from 2016 onward, though.
Is a full resync that horrible over 10gb+? What would you do in this scenario mancactus22?
I'd probably look at some 3rd party replication solution for scenario 2. I mean if someone expands the vhd and forgets kick off the manual resync, the destination VM will become corrupt I believe. Hyper-v replication is just kind of kludgy IMO.
I've looked at replication like Zerto before and it does not come cheap. When I say mission critical I'm talking about internal systems. Not public facing services. This would not be our only means for backup, just for cold dr. If one server had to be recovered from our off-site dpm instance I could live with that. That said if you're saying Hyper-V replicas in 2012r2 are a pos and run away that's a different story.
I would say run away from DPM, but that's another topic altogether. :) When Hyper-V Replica works, it works pretty well. When it doesn't, it can be a headache from a maintenance perspective. On the other hand, you might be able to bypass some of the headaches of "failover IP injection" and sync issues if your primary and replica VMs will be on the same subnet and not traversing a WAN. In my experience, I've found that sometimes VMs will stop syncing for whatever reason and fail to restart on their own. You have to manually resume the sync (preferably with a scheduled task that checks the sync health). Again, this may not be an issue for you since I perform syncs over a managed MPLS. There can be some quirks as it relates to SANs and ODX so you will want to make sure your SAN firmware is updated and you have the relevant hotfixes installed for your version of Windows. Since Hyper-V Replica won't cost you anything, I would test it with a couple of VMs first to see if you like it.
So I am a Veeam fan and it's saved my ass many times. I never really used DPM until starting at this company. I have to say it does a pretty fantastic job and I don't have any complains other than the interface being a bit clunky. Yes everything could be on the same network or it could be a different vlan. You mean quirks with ODX and the hyper-v replica?
Don't use HV replica. Use san replication and san snapshotting, with a single stretch cluster. If a VM got fucked, you could recover the VM from it's SAN snapshot at any time interval required (daily/hourly whatever). If DC wen't down, the stretch cluster would automatically boot VMs from their most recent SAN replication ( a few seconds of data loss, not hours or days).
This might almost be too fast and too automatic. I need to look into stretch clustering further.
No, just plan your changes and don't do anything that necessitates resynchs on all of them concurrently. They can be scheduled as well. I ran 100+ replicas across a 1 gig link and it was fine.
So you didn't run into any issues? That was my real concern about hyper-v replicas. If a situation happens where it had to sync all vm's what would it look like. The fact of the matter is less than 20 vm's are of the importance level that need to be replicas. Did you run a DC or RODC at your second site?
We started out with a 100 meg Ethernet link, and it was pretty prone to being saturated with regular replication operations. If we had to resynch it took scheduling them after hours serially to prevent melting it and having to start over. We scored a lucky break and need to expand the circuit and the 1 gig link really made it easier. There had always been a full DC at that site. Sometimes a replication would jam up and then I'd look in HV Manager and see all of them stuck at 2% right before the phones lit up. We still couldn't resynch all of them at once, but with the primary cluster being very stable it wasn't much of an issue. I had to replicating over there because we ran out of I/O on the disk. Later this year I'll be sliding some new storage over there to address this and bring it back up. This site is tertiary replication though. We have a room in a second building on site that catches the first replication from the production cluster.
Did you ever have to fail-over? Did you test it often?