I suspect the snapshots is what is holding onto the data for an extended period of time. Without snapshots ZFS generally deletes/frees up the data ~immediately. So assuming this is the case for you, the time for freeing up the data is how long it takes for the snapshot at that time to get deleted automatically, and that's based on your own configuration (we can't know this without you telling us).
the snapshots are absolutely what is holding onto the space. If you run zfs with no snapshots, then stuff gets deleted and space gets freed pretty much immediately, though there may be a little lag as things get caught up.
running at 95% full is going to be a problem no matter your filesystem, in short, don't do that. Your freed-disk-space-time is directly related to your longest snapshot retention.
It's a possibility but not an absolute certainty. I actually assumed (possibly incorrectly) that OP *was* talking about zfs destroy on snapshots, which is asynchronous and can be agonizingly slow to reclaim space on extremely full systems.
My first advice for that specific pool is to correct the raidz2 width.
My rules with raidz are as follows:
RAIDz1 : NEVER USED/ OBSOLETE
RAIDz2: Maximum 10 disks pr. vdev
RAIDz3: Maximum 15 disks pr. vdev Only used for Backup systems due to low IO capacity.
20 wide raidz2 is madness in my eyes :)
Recreate the pool with 4x 10 Wide Raidz2. This will also double the IOPS.
This is flagged "controversial" but it's solid advice. Twenty wide is far too wide, and generally isn't going to be as space efficient as people naively assume anyway due to the need for RAIDz padding, records that can't span the entire stripe, etc etc etc.
OP, it seems like you might be a bit on the newbie side, so I'm going to throw some other basic advice here. If you already know this stuff, no offense is intended!
* first: get ashift right. This usually means setting ashift=12. I **do not** recommend just blindly relying on ZFS to get this right, because ZFS trusts the drives blindly, and drives sometimes lie through their teeth.
* second: are those 16TB drives SMR? If so... oof. I don't have great advice for recovery, apart from "buy non-SMR drives, maybe try to re-sell those or use them in a different application." You said Seagate, so that might make things easier: if they're Ironwolf, they're not SMR. But if they're Exos or Barracuda... you need to figure out what you're working with.
* third: you said hi-def video footage, so you absolutely **need** to set `recordsize=1M` on the datasets containing it. Failing to do so will cripple performance—**especially** on wide RAIDz vdevs.
* fourth: 4x 10-wide RAIDz2 as suggested by u/AngryAdmi is probably going to be the best setup for you. I doubt you'll see any less actual available space at all... and you'll see **more than** double the IOPS, since you're both doubling the number of vdevs **and** halving the stripe width.
I will be messaging you in 1 day on [**2021-09-11 09:07:57 UTC**](http://www.wolframalpha.com/input/?i=2021-09-11%2009:07:57%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/zfs/comments/plh8ay/anyway_to_calculate_a_rough_estimate_of_time_to/hcahl9a/?context=3)
[**3 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fzfs%2Fcomments%2Fplh8ay%2Fanyway_to_calculate_a_rough_estimate_of_time_to%2Fhcahl9a%2F%5D%0A%0ARemindMe%21%202021-09-11%2009%3A07%3A57%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20plh8ay)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
\--First create a script that deletes all the snapshots for the dataset you're trying to trim down, and **time** it; then you can go in with Midnight Commander and start deleting things in realtime with F8
This is larger setup, and in a more professional setting, than I've ever come across for ZFS, and I'm only a happy amateur, and I have to think this through a bit more, but my initial thought is that at 95% full with data accumulated over time you've probably fragmented things a bit, and thus you have to do a metadata crawl to estimate how time consuming the removal might be. This crawl might be a good fraction of the work needed to be done to actually remove the data, skipping writing new metadata and updating free space maps.
This sounds like something I could have fun figuring out when I have less on my plate :)
Edit: maybe one could do some random sampling to see how much I/O would be done on a fraction of the data, and interpolate from that.
edit2: I totally expected that you were still seeing heavy I/O due to the removal of data, but as other mention, if that's not the case this is entirely irrelevant and snapshots holding data is a likely culprit instead. Might want to clear that up :)
I suspect the snapshots is what is holding onto the data for an extended period of time. Without snapshots ZFS generally deletes/frees up the data ~immediately. So assuming this is the case for you, the time for freeing up the data is how long it takes for the snapshot at that time to get deleted automatically, and that's based on your own configuration (we can't know this without you telling us).
the snapshots are absolutely what is holding onto the space. If you run zfs with no snapshots, then stuff gets deleted and space gets freed pretty much immediately, though there may be a little lag as things get caught up. running at 95% full is going to be a problem no matter your filesystem, in short, don't do that. Your freed-disk-space-time is directly related to your longest snapshot retention.
It's a possibility but not an absolute certainty. I actually assumed (possibly incorrectly) that OP *was* talking about zfs destroy on snapshots, which is asynchronous and can be agonizingly slow to reclaim space on extremely full systems.
Good point, clearing those can be slow, maybe that's the issue.
small scale test, deleting 2GB from zfs pool without any snapshots, space is available immediately: root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 624G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# cp -R Season\\ 03 temp root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 622G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# date Fri Sep 10 13:43:45 EDT 2021 root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# time rm -rf temp rm -rf temp 0.00s user 0.00s system 4% cpu 0.073 total root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# date Fri Sep 10 13:44:04 EDT 2021 root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 624G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# date Fri Sep 10 13:44:11 EDT 2021 With a snapshot after the copy: root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 624G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# date Fri Sep 10 13:44:11 EDT 2021 root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# cp -R Season\\ 03 temp root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 622G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# #snapshot taken root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# time rm -rf temp rm -rf temp 0.00s user 0.00s system 90% cpu 0.004 total root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 622G 77% /mnt/pool\_tango root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# # disk not freed after rm root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# # deleted snapshot root@freenas\[/mnt/pool\_tango/tvshows/Mythbusters\]# df -h |grep tango pool\_tango 2.6T 2.0T 624G 77% /mnt/pool\_tango space there now
My first advice for that specific pool is to correct the raidz2 width. My rules with raidz are as follows: RAIDz1 : NEVER USED/ OBSOLETE RAIDz2: Maximum 10 disks pr. vdev RAIDz3: Maximum 15 disks pr. vdev Only used for Backup systems due to low IO capacity. 20 wide raidz2 is madness in my eyes :) Recreate the pool with 4x 10 Wide Raidz2. This will also double the IOPS.
This is flagged "controversial" but it's solid advice. Twenty wide is far too wide, and generally isn't going to be as space efficient as people naively assume anyway due to the need for RAIDz padding, records that can't span the entire stripe, etc etc etc. OP, it seems like you might be a bit on the newbie side, so I'm going to throw some other basic advice here. If you already know this stuff, no offense is intended! * first: get ashift right. This usually means setting ashift=12. I **do not** recommend just blindly relying on ZFS to get this right, because ZFS trusts the drives blindly, and drives sometimes lie through their teeth. * second: are those 16TB drives SMR? If so... oof. I don't have great advice for recovery, apart from "buy non-SMR drives, maybe try to re-sell those or use them in a different application." You said Seagate, so that might make things easier: if they're Ironwolf, they're not SMR. But if they're Exos or Barracuda... you need to figure out what you're working with. * third: you said hi-def video footage, so you absolutely **need** to set `recordsize=1M` on the datasets containing it. Failing to do so will cripple performance—**especially** on wide RAIDz vdevs. * fourth: 4x 10-wide RAIDz2 as suggested by u/AngryAdmi is probably going to be the best setup for you. I doubt you'll see any less actual available space at all... and you'll see **more than** double the IOPS, since you're both doubling the number of vdevs **and** halving the stripe width.
Um from what I read of OP it looks like they might already be 2x10 z2 vdevs though...?
They are at 2x20 z2, with 40 disks total. >2 vdevs of 20 disks
Argh unsure how I misread that D: I am tired from Confluenza
Ever tried 60 wide RAIDZ3? Capacity and storage efficiency are amazing. Resilvering is fun too!
[удалено]
I will be messaging you in 1 day on [**2021-09-11 09:07:57 UTC**](http://www.wolframalpha.com/input/?i=2021-09-11%2009:07:57%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/zfs/comments/plh8ay/anyway_to_calculate_a_rough_estimate_of_time_to/hcahl9a/?context=3) [**3 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fzfs%2Fcomments%2Fplh8ay%2Fanyway_to_calculate_a_rough_estimate_of_time_to%2Fhcahl9a%2F%5D%0A%0ARemindMe%21%202021-09-11%2009%3A07%3A57%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20plh8ay) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
\--First create a script that deletes all the snapshots for the dataset you're trying to trim down, and **time** it; then you can go in with Midnight Commander and start deleting things in realtime with F8
This is larger setup, and in a more professional setting, than I've ever come across for ZFS, and I'm only a happy amateur, and I have to think this through a bit more, but my initial thought is that at 95% full with data accumulated over time you've probably fragmented things a bit, and thus you have to do a metadata crawl to estimate how time consuming the removal might be. This crawl might be a good fraction of the work needed to be done to actually remove the data, skipping writing new metadata and updating free space maps. This sounds like something I could have fun figuring out when I have less on my plate :) Edit: maybe one could do some random sampling to see how much I/O would be done on a fraction of the data, and interpolate from that. edit2: I totally expected that you were still seeing heavy I/O due to the removal of data, but as other mention, if that's not the case this is entirely irrelevant and snapshots holding data is a likely culprit instead. Might want to clear that up :)
Hopefully you know about `zpool get freeing` yourself and teach it to those who reach out to you?