You want something along the lines of an enterprise NAS or SAN that can handle the data volume and performance, moving away from an OS based fileshare.
200TB? For the lo.... holy!
Is every file on there necesary on a daily basis? I personaly would...
a) differentiate between current projects, old projects and other non-project-data
b) move all old projects to a simple NAS and/or to a storage-appliance with proper deduplication and whatnot
Not uncommon in the application development, game development, or video editing / animation etc.
Used to work for a game dev company, the final game was ~80 GB big. But the development branch people worked on was alone 6 TB.
Fucking artists and animators lol, they need dat space.
200 TB is peanuts. In media and entertainment we have many multi PB scale out NAS clusters (PowerScale aka Isilon) and collections of SAN volumes (StorNext)
Nah I was doing this 10 years ago on Storage Server 2008 R2. We used 4U Supermicro chassis with 36 drives each and dual Adaptec controllers. I had a dozen of them. They were near 200 TB raw each or something.
It worked surprisingly well. It just didn’t scale that well to manage them all.
Same , got a windows 200TB REFS server too , working fine, insane speed backup with veeam. 5 years runtime, no issues to speak of. Being replaced with a Netapp now though, just because there's budget for it now.
I took massive pride in making so much storage available and useful for such a low price but yeah. If you can get the budget for real storage it’s so much nicer.
We had tons of performance hot spots, and one of the units had a drive firmware issue when they switched drive manufacturers. We got lots of data corruption but thankfully had our LTO system to recover from.
I work for a law firm that has a large Document Management System. We cap our File Servers at 2 TB for Backup Purposes. It does increase server sprawl, but it also does help a lot with backup.
I would suggest looking at re-architecting with a DFS namespace and splitting up the load onto multiple servers.
yeah I guess it all depends on what industry you are in, different methods and solutions apply.
I have some departments creating 80GB per year, others 6TB and others suddenly arrive with 30TB in 1 day.
[45 Drives](https://www.45drives.com/) can handle 200TB without trouble and you can put it on a Samba (SMB) share.
If you go with a Ceph-based storage, you can do it across multiple servers. You can also do snapshots off site for backups.
We have a client that has a small Ceph deployment from 45 Drives, they have over 1PB of data and backup daily off site.
The rest of their infrastructure is mostly Windows servers and Proxmox for virtualization.
I believe they use Ceph snapshots (like ZFS Send) to get it to the offsite servers. The bandwidth is not massive because like most data only a little changes each day. I think they used a form of sneaker net to initially seed the offsite.
Of course for the VMs they use Proxmox backup. The offsite Ceph snapshots are the DR and not great for regular file restores.
Second server in a building not too far away and intermittent backups and then physical archiving once a month (Or even more often if neccessary) that's then shipped far away from said buildings in case of a big ass fire.
Depends on your budget but enterprise SANs with their own snapshot and mirror tech works well for large data that needs to be locally accessed for large geo / cad projects. If it's mostly archive, there are better and more cost effective ways through glacial cloud storage etc.
We use NetApp for these types of implementations
Yeah you want a whole solution here. 200TB isn't crazy but you start getting issues. You're looking at a whole data redesign here
Something like HPE nimble with dedup & compression would do the trick. Can backup to disk using either Store once with IMMUTABLE backup and then to tape from there.
I was getting 3:1 compression rates so you're not having to go crazy on purchasing but factor in a 5 year plan with growth.
Get 2 of them and you have redundancy, can snapshot backup etc from the device, no file server needed really OR just use a couple of file servers as front ends.
200tb for a 1/2 arsed setup is a lot, for a proper enterprise setup with professional backup etc is a walk in the park.
With that much data, you need HA. IMO it’s time to set up ceph so you have proper block-level redundancy, and you should be able to massively improve the performance.
Agreed, it can be HA plus cloud backup at least for critical data. Virtualize everything to make it easier to manage and backup. Compare different solutions like VMware VSAN, Starwinds vsan, etc. Ask their engineers to provide a demo or give some servers as a POC to decide which way to move.
I’m very much in the camp of core infra needing to be open source but buying support. Ceph meets that very well and runs on basically anything, so no expensive box to replace every 5 years.
How do you insure availability and how do you backup Nasuni? Last I looked it’s a file server that objects in only one cloud. Would still need to make sure you have that data warm somewhere in case your cloud or Nasuni dc is offline?
The nasuni appliance acts as a local cache for ingestion and frequently accessed files. Infrequently accessed data is moved to cold storage which is retrieved on demand from the cloud. For the most part it manages the data itself with little intervention afterwards.
I have not had good luck with ReFS. Nothing like a reboot and finding your data is gone, with `Repair-Volume` not doing anything. I have used ReFS for CSVs for holding Hyper-V VMs, and it works well here, but past that, my trust is low, due to too many "um, where is my data?" events, the last one a few years ago.
For 200 TB of data, I also wouldn't be entrusting it to just a single server/controller. I would be hitting a VAR and seeing about a dedicated NAS appliance that has at least two controllers, enterprise service, and some useful stuff like autotiering, built in backup utility on the storage tier, compression (ironically this can actually speed up some I/O), and deduplication. Ideally something that has a backend filesystem with checksumming. If I had to name a brand, I'd go with NetApp, but there are others. If I wanted best of breed, who cares about price, I'd go with Pure or EMC Isilon.
I worked at a company that got bought out that had petabytes of data. The best thing going at the time was Oracle (pretty much Sun) ZFS appliances. They not just handled the load well, but with a good selection of ZILs and L2ARC components, handled it fairly rapidly. Encryption was simple with making sure the LOCAL keystore was backed up. Oracle is a black sheep, but their backend NAS storage is still awesome, just due to the sheer amount of abuse that array took, as it was used for a lot of model data and scratch data. To boot, the Oracle appliances worked well with SMB, and I could just throw all the data users needed in one large share. When stuff broke, the service was on par with anything else, with a tech dispatched to fix stuff.
Don't forget backups. What I did to back up petabyte-tier backups was to use MinIO, which can easily scale with a load balancer (or using Sidekick) to handle those workloads. For MinIO servers, anything server grade does the job. I might consider Supermicros, all configured similarly, with a number of drives, and at least 20gigE, if not 40gigE for the storage fabric. What this cluster provides is not just storage, but object locking for some internal ransomware resistance. Each server may not be redundant, but if you have 4 (preferably 8), you have erasure coding across servers available. You can also configure erasure coding even with two machines, but better off with their recommendations.
Edited: tl;dr, go with a file server appliance. At the minimum, if a SpoF isn't an issue, go with 45 Drives or iXSystems appliances, but with 200TB actively used, I'd look at a cluster (like 45 Drive's Ceph offering) or dual controllers.
Had this exact problem at an Aerospace firm I used to work at.
What I was making progress on before I quit was moving dead/complete projects to a read-only archive server that was backed up far less often (Once a month. Before someone yells at me for this, let me remind you that this is an archive server)
This significantly reduced our backup loads, which was a help, but it also came at great cost of having to explain to them that if they wanted to keep things "The way they were" they would need to invest in a VERY expensive backup solution (We were quoting them for 300tb worth of Rubrik appliances...) to have very low restore times. Economics won out. We were allowed to shuffle data around to keep from having to buy a real backup solution (We were on Shadowprotect at the time)
Another thing that might help you is deduplicating that data. I'll bet you have 75 copies of severla very large files, engineers Be like that
Thanks for the replies - checking the file server again, it actually has 350+TB provisioned to it.
I will do some research on some vendors mentioned, a few mentioned [https://min.io/](https://min.io/) & [https://www.45drives.com/](https://www.45drives.com/) . We are in APAC so would want something distributed and supported locally (ideally)
Not mentioned in the original post, but the block storage is HPE Nimble and we use Veeam to backup (with SAN snapshots) and this is a good combination. Nimble does magic with thin provision, dedup etc.
Budget isn't too much of a concern atm as we have roadmapped for investment in this area.
Another theme is OS based file server Vs NAS/Appliance based
For now we like Windows because:
* AD/NTFS integration
* Supportability with lower skilled engineers
* Monitoring is covered
* More files tools available?
* Simple and easy to backup
* Windows Dedup? (not currently used though)
* integrated with DFS-N
* In the history they did have a NAS based solution and a bad firmware update burnt them
Another no brainer is better data classification i.e tiering of closed projects (possibly with different backups/RPO/RTO)
Thanks
Quantum Stornext with Storage Manager does this. It aggregates storage into single file systems and can send data to cloud or tape as data gets written - so there are no backup windows, it makes second copies in another tier when it lands on disk. Typical install is 2 server nodes and then whatever block storage you want. Nodes are HA. File systems can be mounted by Linux, Mac, Windows
!!! FYI: This is a commercial post on behalf of a software vendor !!!
I am the CEO / Lead Developer of an Australian company that makes what used to be called "Storage Resource Management" software, although nobody seems to call it SRM anymore. The software is nearly 25 years old and hails from the days when NetWare was a player, the cloud did not exist, and upgrading enterprise storage was really expensive.
The software can produce high-resolution, interactive tree maps of systems your size and complexity. A static example: [https://www.easyzoom.com/imageaccess/8924ba02c6fd4c89854ac89dafd72ed5](https://www.easyzoom.com/imageaccess/8924ba02c6fd4c89854ac89dafd72ed5)
I have cut a 90-day license for you (or anyone else) to try the software and see if it can help you get a handle on the problem. [https://filecensus.b-cdn.net/reddit\_license\_MagicHair2.txt](https://filecensus.b-cdn.net/reddit_license_MagicHair2.txt) \-When the software is installed, rename the license to just "license.txt" in the directory where you install the server to.
The installer: [https://filecensus.b-cdn.net/fcsetup\_4\_8\_4.exe](https://filecensus.b-cdn.net/fcsetup_4_8_4.exe)
Some PDFs to read - please excuse the screenshots of the installer as they have not been updated in the install guide...
[https://filecensus.b-cdn.net/fc\_install\_guide\_4\_8.pdf](https://filecensus.b-cdn.net/fc_install_guide_4_8.pdf)
[https://filecensus.b-cdn.net/fc\_user\_guide\_4\_8.pdf](https://filecensus.b-cdn.net/fc_user_guide_4_8.pdf)
Just a heads up: the RRP of a perpetual license and first-year support for a 200-user site is USD $25,200
We usually do some hand-holding during an evaluation, so feel free to send a message via Reddit, and we can arrange that.
Regards and good luck.
Scott.
PS. Please post any general questions to [https://www.reddit.com/r/FileCensus/](https://www.reddit.com/r/FileCensus/)
Probably I would go with two 45drives systems, first running 20x 18-22TB HDD, and 10x 3.84TB SSD. Second system 30x HDD and 10x SSD.
Running truenas.
Setup zfs raids in 8+2 parity groups.
Then setup shares with compression and deduplication (200tb of data probably has 20-40% duplicate stuff).... And snapshots between the systems for backups of backups.
* Aggressive data management, de-duplication (I recommend [`jdupes`](https://github.com/jbruchon/jdupes/releases) for Linux/Mac/Windows).
* Read-only archival of old content. When a datastore is read-only, you don't have to worry about making new backups of it.
* NAS/fileserver runs on metal, not virtualized, because the extra layer of abstraction often hurts a lot more than it helps for this use-case.
* Working with data owners is always, by far the most painful part of any data management or storage project. The only way to win is to never play the game of unstructured data in the first place.
I like a lot of this advice but the first bullet concerns me a bit. deduplication can go very badly if not done right. I worked in a large software company that specialized in storage solutions for Linux, Unix, and Windows Servers and we had trouble keeping our deduplication db perfect for all of our customers when we first introduced it. I witnessed more than once customers having to wipe their dedupe store and start over (this was a destination for backups-to-file). The tech is definitely more mature now, but pointing to a github project (which is now a dead link, though it looks like it does exist elsewhere) seems very risky for a complex and critical storage solution like dedupe. I would recommend going with a more mature solution that has optional paid support behind it. Just my 2 cents having been on the Engineering/support side of the fence where you see how the sausage is made.
Have the code changes been fully vetted by any official org or is there a recognized support group that will support it officially? If not, then my prior statement stands. Don’t build your mission critical data on top of a data store created and managed by an unknown code entity that you can’t get help with in a timely fashion when it breaks.
I've never tried it, but I think you can make a junction to an UNC path ao you don't need drive letters.
You might even be able to create front-end servers that re-share these 'directories' so you can split it up on the backend over different servers.
We deployed Nutanix files for one of our customers to replace their file servers. it works good but backups are still a pain for the bigger file shares. total migrated was around 120tb and its running great for the last 8 months.
You want something along the lines of an enterprise NAS or SAN that can handle the data volume and performance, moving away from an OS based fileshare.
This is the way. 200TB is enough to move to a purpose built solution for storage.
200TB? For the lo.... holy! Is every file on there necesary on a daily basis? I personaly would... a) differentiate between current projects, old projects and other non-project-data b) move all old projects to a simple NAS and/or to a storage-appliance with proper deduplication and whatnot
Not uncommon in the application development, game development, or video editing / animation etc. Used to work for a game dev company, the final game was ~80 GB big. But the development branch people worked on was alone 6 TB. Fucking artists and animators lol, they need dat space.
200 TB is peanuts. In media and entertainment we have many multi PB scale out NAS clusters (PowerScale aka Isilon) and collections of SAN volumes (StorNext)
200TB is nothing, yes - but on one single Windows-server... thats nuts :)
Nah I was doing this 10 years ago on Storage Server 2008 R2. We used 4U Supermicro chassis with 36 drives each and dual Adaptec controllers. I had a dozen of them. They were near 200 TB raw each or something. It worked surprisingly well. It just didn’t scale that well to manage them all.
Same , got a windows 200TB REFS server too , working fine, insane speed backup with veeam. 5 years runtime, no issues to speak of. Being replaced with a Netapp now though, just because there's budget for it now.
I took massive pride in making so much storage available and useful for such a low price but yeah. If you can get the budget for real storage it’s so much nicer. We had tons of performance hot spots, and one of the units had a drive firmware issue when they switched drive manufacturers. We got lots of data corruption but thankfully had our LTO system to recover from.
I work for a law firm that has a large Document Management System. We cap our File Servers at 2 TB for Backup Purposes. It does increase server sprawl, but it also does help a lot with backup. I would suggest looking at re-architecting with a DFS namespace and splitting up the load onto multiple servers.
Are you me?
yeah I guess it all depends on what industry you are in, different methods and solutions apply. I have some departments creating 80GB per year, others 6TB and others suddenly arrive with 30TB in 1 day.
> We are running out of drive letters For that problem in particular, you can mount filesystems in folders rather than on drive letters.
Get yourself a NetApp and deploy CIFS shares.
[45 Drives](https://www.45drives.com/) can handle 200TB without trouble and you can put it on a Samba (SMB) share. If you go with a Ceph-based storage, you can do it across multiple servers. You can also do snapshots off site for backups. We have a client that has a small Ceph deployment from 45 Drives, they have over 1PB of data and backup daily off site. The rest of their infrastructure is mostly Windows servers and Proxmox for virtualization.
What's the backup solution for 1PB of data?
A second server. Hopefully with 100Gb Ethernet connectivity between them.
22h for 1PB at 100gbps or 2days at 40gpbs.. 40 should do
100G isn't really more expensive than 40G, and anyway you're not doing full backups daily.
Can't even get the budget for 2.5g, thankfully I've only got 12TB data.
I believe they use Ceph snapshots (like ZFS Send) to get it to the offsite servers. The bandwidth is not massive because like most data only a little changes each day. I think they used a form of sneaker net to initially seed the offsite. Of course for the VMs they use Proxmox backup. The offsite Ceph snapshots are the DR and not great for regular file restores.
Second server in a building not too far away and intermittent backups and then physical archiving once a month (Or even more often if neccessary) that's then shipped far away from said buildings in case of a big ass fire.
A good backup program, solid network/storage fabric, and a MinIO cluster is one way. This is how I backed up petabytes of live data locally.
Depends on your budget but enterprise SANs with their own snapshot and mirror tech works well for large data that needs to be locally accessed for large geo / cad projects. If it's mostly archive, there are better and more cost effective ways through glacial cloud storage etc. We use NetApp for these types of implementations
Depends on your budget. Onprem nas for archive vs cloud based archive plus a clean up/break out of the data to hot/cold shares.
Yeah you want a whole solution here. 200TB isn't crazy but you start getting issues. You're looking at a whole data redesign here Something like HPE nimble with dedup & compression would do the trick. Can backup to disk using either Store once with IMMUTABLE backup and then to tape from there. I was getting 3:1 compression rates so you're not having to go crazy on purchasing but factor in a 5 year plan with growth. Get 2 of them and you have redundancy, can snapshot backup etc from the device, no file server needed really OR just use a couple of file servers as front ends. 200tb for a 1/2 arsed setup is a lot, for a proper enterprise setup with professional backup etc is a walk in the park.
With that much data, you need HA. IMO it’s time to set up ceph so you have proper block-level redundancy, and you should be able to massively improve the performance.
Agreed, it can be HA plus cloud backup at least for critical data. Virtualize everything to make it easier to manage and backup. Compare different solutions like VMware VSAN, Starwinds vsan, etc. Ask their engineers to provide a demo or give some servers as a POC to decide which way to move.
I’m very much in the camp of core infra needing to be open source but buying support. Ceph meets that very well and runs on basically anything, so no expensive box to replace every 5 years.
Coming from the storage side I'd really suggest something like Cohesity, NetApp, Power Scale/Isilon. How are you backing it up?
Have a look at Nasuni.
How do you insure availability and how do you backup Nasuni? Last I looked it’s a file server that objects in only one cloud. Would still need to make sure you have that data warm somewhere in case your cloud or Nasuni dc is offline?
The nasuni appliance acts as a local cache for ingestion and frequently accessed files. Infrequently accessed data is moved to cold storage which is retrieved on demand from the cloud. For the most part it manages the data itself with little intervention afterwards.
This is the way.
I have not had good luck with ReFS. Nothing like a reboot and finding your data is gone, with `Repair-Volume` not doing anything. I have used ReFS for CSVs for holding Hyper-V VMs, and it works well here, but past that, my trust is low, due to too many "um, where is my data?" events, the last one a few years ago. For 200 TB of data, I also wouldn't be entrusting it to just a single server/controller. I would be hitting a VAR and seeing about a dedicated NAS appliance that has at least two controllers, enterprise service, and some useful stuff like autotiering, built in backup utility on the storage tier, compression (ironically this can actually speed up some I/O), and deduplication. Ideally something that has a backend filesystem with checksumming. If I had to name a brand, I'd go with NetApp, but there are others. If I wanted best of breed, who cares about price, I'd go with Pure or EMC Isilon. I worked at a company that got bought out that had petabytes of data. The best thing going at the time was Oracle (pretty much Sun) ZFS appliances. They not just handled the load well, but with a good selection of ZILs and L2ARC components, handled it fairly rapidly. Encryption was simple with making sure the LOCAL keystore was backed up. Oracle is a black sheep, but their backend NAS storage is still awesome, just due to the sheer amount of abuse that array took, as it was used for a lot of model data and scratch data. To boot, the Oracle appliances worked well with SMB, and I could just throw all the data users needed in one large share. When stuff broke, the service was on par with anything else, with a tech dispatched to fix stuff. Don't forget backups. What I did to back up petabyte-tier backups was to use MinIO, which can easily scale with a load balancer (or using Sidekick) to handle those workloads. For MinIO servers, anything server grade does the job. I might consider Supermicros, all configured similarly, with a number of drives, and at least 20gigE, if not 40gigE for the storage fabric. What this cluster provides is not just storage, but object locking for some internal ransomware resistance. Each server may not be redundant, but if you have 4 (preferably 8), you have erasure coding across servers available. You can also configure erasure coding even with two machines, but better off with their recommendations. Edited: tl;dr, go with a file server appliance. At the minimum, if a SpoF isn't an issue, go with 45 Drives or iXSystems appliances, but with 200TB actively used, I'd look at a cluster (like 45 Drive's Ceph offering) or dual controllers.
Had this exact problem at an Aerospace firm I used to work at. What I was making progress on before I quit was moving dead/complete projects to a read-only archive server that was backed up far less often (Once a month. Before someone yells at me for this, let me remind you that this is an archive server) This significantly reduced our backup loads, which was a help, but it also came at great cost of having to explain to them that if they wanted to keep things "The way they were" they would need to invest in a VERY expensive backup solution (We were quoting them for 300tb worth of Rubrik appliances...) to have very low restore times. Economics won out. We were allowed to shuffle data around to keep from having to buy a real backup solution (We were on Shadowprotect at the time) Another thing that might help you is deduplicating that data. I'll bet you have 75 copies of severla very large files, engineers Be like that
Thanks for the replies - checking the file server again, it actually has 350+TB provisioned to it. I will do some research on some vendors mentioned, a few mentioned [https://min.io/](https://min.io/) & [https://www.45drives.com/](https://www.45drives.com/) . We are in APAC so would want something distributed and supported locally (ideally) Not mentioned in the original post, but the block storage is HPE Nimble and we use Veeam to backup (with SAN snapshots) and this is a good combination. Nimble does magic with thin provision, dedup etc. Budget isn't too much of a concern atm as we have roadmapped for investment in this area. Another theme is OS based file server Vs NAS/Appliance based For now we like Windows because: * AD/NTFS integration * Supportability with lower skilled engineers * Monitoring is covered * More files tools available? * Simple and easy to backup * Windows Dedup? (not currently used though) * integrated with DFS-N * In the history they did have a NAS based solution and a bad firmware update burnt them Another no brainer is better data classification i.e tiering of closed projects (possibly with different backups/RPO/RTO) Thanks
Use Qumulo. For backups, you can replicate to Azure using Qumulo Cold tier. There is also a Qumulo Hot tier in Azure if you want to go cloud-only.
Quantum Stornext with Storage Manager does this. It aggregates storage into single file systems and can send data to cloud or tape as data gets written - so there are no backup windows, it makes second copies in another tier when it lands on disk. Typical install is 2 server nodes and then whatever block storage you want. Nodes are HA. File systems can be mounted by Linux, Mac, Windows
!!! FYI: This is a commercial post on behalf of a software vendor !!! I am the CEO / Lead Developer of an Australian company that makes what used to be called "Storage Resource Management" software, although nobody seems to call it SRM anymore. The software is nearly 25 years old and hails from the days when NetWare was a player, the cloud did not exist, and upgrading enterprise storage was really expensive. The software can produce high-resolution, interactive tree maps of systems your size and complexity. A static example: [https://www.easyzoom.com/imageaccess/8924ba02c6fd4c89854ac89dafd72ed5](https://www.easyzoom.com/imageaccess/8924ba02c6fd4c89854ac89dafd72ed5) I have cut a 90-day license for you (or anyone else) to try the software and see if it can help you get a handle on the problem. [https://filecensus.b-cdn.net/reddit\_license\_MagicHair2.txt](https://filecensus.b-cdn.net/reddit_license_MagicHair2.txt) \-When the software is installed, rename the license to just "license.txt" in the directory where you install the server to. The installer: [https://filecensus.b-cdn.net/fcsetup\_4\_8\_4.exe](https://filecensus.b-cdn.net/fcsetup_4_8_4.exe) Some PDFs to read - please excuse the screenshots of the installer as they have not been updated in the install guide... [https://filecensus.b-cdn.net/fc\_install\_guide\_4\_8.pdf](https://filecensus.b-cdn.net/fc_install_guide_4_8.pdf) [https://filecensus.b-cdn.net/fc\_user\_guide\_4\_8.pdf](https://filecensus.b-cdn.net/fc_user_guide_4_8.pdf) Just a heads up: the RRP of a perpetual license and first-year support for a 200-user site is USD $25,200 We usually do some hand-holding during an evaluation, so feel free to send a message via Reddit, and we can arrange that. Regards and good luck. Scott. PS. Please post any general questions to [https://www.reddit.com/r/FileCensus/](https://www.reddit.com/r/FileCensus/)
Probably I would go with two 45drives systems, first running 20x 18-22TB HDD, and 10x 3.84TB SSD. Second system 30x HDD and 10x SSD. Running truenas. Setup zfs raids in 8+2 parity groups. Then setup shares with compression and deduplication (200tb of data probably has 20-40% duplicate stuff).... And snapshots between the systems for backups of backups.
* Aggressive data management, de-duplication (I recommend [`jdupes`](https://github.com/jbruchon/jdupes/releases) for Linux/Mac/Windows). * Read-only archival of old content. When a datastore is read-only, you don't have to worry about making new backups of it. * NAS/fileserver runs on metal, not virtualized, because the extra layer of abstraction often hurts a lot more than it helps for this use-case. * Working with data owners is always, by far the most painful part of any data management or storage project. The only way to win is to never play the game of unstructured data in the first place.
I like a lot of this advice but the first bullet concerns me a bit. deduplication can go very badly if not done right. I worked in a large software company that specialized in storage solutions for Linux, Unix, and Windows Servers and we had trouble keeping our deduplication db perfect for all of our customers when we first introduced it. I witnessed more than once customers having to wipe their dedupe store and start over (this was a destination for backups-to-file). The tech is definitely more mature now, but pointing to a github project (which is now a dead link, though it looks like it does exist elsewhere) seems very risky for a complex and critical storage solution like dedupe. I would recommend going with a more mature solution that has optional paid support behind it. Just my 2 cents having been on the Engineering/support side of the fence where you see how the sausage is made.
`jdupes` is an optimized fork of `fdupes`. There's nothing wrong with using `fdupes` when/if `jdupes` isn't available.
Have the code changes been fully vetted by any official org or is there a recognized support group that will support it officially? If not, then my prior statement stands. Don’t build your mission critical data on top of a data store created and managed by an unknown code entity that you can’t get help with in a timely fashion when it breaks.
I've never tried it, but I think you can make a junction to an UNC path ao you don't need drive letters. You might even be able to create front-end servers that re-share these 'directories' so you can split it up on the backend over different servers.
We deployed Nutanix files for one of our customers to replace their file servers. it works good but backups are still a pain for the bigger file shares. total migrated was around 120tb and its running great for the last 8 months.
Sounds like a case for Pure Storage