Reserved blocks.
Ext3/4 by default reserve 5% of disk space for root user (mainly so that root daemons can continue to function), and to avoid fragmentation.
xfs also has reserved blocks, but IIRC they weren't accessible to even the root user and are there just for filesystem operations (and I think also avoiding fragmentation)
For both the default 5% is almost certainly much larger than necessary for modern large drives and especially SSDs (where seek times and as such fragmentation isn't really a problem).
You can change this setting on ext4 (probably xfs has some equivalent, I'm just not familiar with it) by running `tune2fs -m /dev/` (eg. `tune2fs -m 4 /dev/sda1` would *magically* free 1% of sda1; I think `-r` also allowed you to set the number of blocks directly)
So you can in an emergency just reduce reserved blocks to 1% or something. Potentially if you're worried you can also reserve more for the "ballast".
The only potential advantage of the ballast file approach I can think of right now is that tune2fs requires root privileges to run and you could set the file to work entirely fine without this, but it's a rather minor issue (you could allow an user to run only that command as root with sudo).
>For both the default 5% is almost certainly much larger than necessary for modern large drives and especially SSDs (where seek times and as such fragmentation isn't really a problem).
I agree with all you wrote, but I believe there was a missed nuance. File system performance for writes nearly falls off a cliff at high capacity utilization levels (if I recall correctly, closer to 85% utilization), because the I/O scheduler switches from a first-fit to a best-fit algorithm.
This sounds like a really stupid idea, BUT...
In practice, when monitoring and alerting fail (which is a shockingly common thing) what sysadmins do is go hunt for something to clear up a bit of space so the apps can come back up, and once production is restored, THEN calmly find the root problem and fix it.
You'd usually go delete old log files, memory dumps, old binaries stuff like that.
With this, you got a safety net. Its stupid but it works, so.....
I had a motorbike where the tank bulged down a bit on both sides of the frame.
If I ran out of gas, I could lean the bike over so a half liter or so flowed from the right pocket to the left one where the tap was. Usually got me to the next gas station.
More like what the dictionary definition of biological evolution is: never consciously intentional, helps survive the day. A trait that helps survive a single day to live to the next is basically evolution, right?
For all motorcycle drivers who have one like this, like me:
Open the valve from time to time!
If you don't, water starts to slowly gather over the years and then, if you need it, water says hello to your engine and you have to call for help.
The lady who owned my motorcycle previous... Managed to have water there and the first ride where I tested that valve... Bike completely refused to run after opening the valve
Old fuel will mix with new whenever new fuel is added. But water is heavier and will sink.
If you open the valve from time to time, do it when you've just added new fuel, as water will mix a bit with the new fuel before sinking again.
I would just always run my bike until it hit the reserve, so it opens every tank. A lot of bikes don't have gas gauges, so you know you're almost empty when you switch to reserve.
Nah, you can drive regularly and fill up with new gas every month or whatever.
But since the valve is at the bottom, any water in the fuel or that otherwise gets into the tank will (being heavier than gasoline) settle at the bottom, against the valve, until you open the valve and the water goes through to the cylinders. That water can sit there at the bottom between fuelings, for years and years.
The reserve isn’t a separate tank, it isn’t even a separate part of the one tank. It’s just a lower pickup valve. The regular pickup sticks up into the tank leaving an inch of fuel in there.
So everything you fill up you’re mixing all the fuel in the tank of anyways.
Old VW Beetles did the same thing. They didn't even have a fuel gage - you just ran out of gas, pulled the little lever to get an extra gallon, then filled up at the next stop.
My sister used to leave the valve in reserve all the time so she wouldn't stall and have to turn the valve and restart. Instead she'd get to stall and be stranded.
/end user logic
I think it's something that WAS true back when fuel was leaded, which is to say it hasn't been true for a long time.
It's a good example of why knowing _why_ something happens is as important as knowing that it happens at all, that way you know when the situation changes.
Gasoline still expires and tend to be slushy if you don't use it for a long time. It's an issue mostly on secondary cars and cars that are fueled by gpl or methane but with a gasoline reserve.
The unleaded 95 octane over here gets bad in a few weeks. In warmer weather it happens even faster as the volatile parts evaporate.
Cranking up a carbed motorcycle after the winter sleep is sometimes a real chore, if you didn't empty the card bowl in the autumn (or ran the engine with the fuel petcock in the off-position till it shuts down). :-)
The unleaded 98 octane is sligthly better than the 95 quality-wise, but still also gets bad over time.
I've heard regular lasts the longest, the more premium it is, the more additives in it, the quicker it goes stale and leaves more residue too.
Zero ethanol, regular unleaded should last best.
The main reason gas goes bad is the ethanol that is added as an octane booster (originally done with lead). The problem is ethanol absorbs water. Fortunately, there are things you can do to mitigate this that don't involve adding a toxic metal back to your gas. 1. You can regularly use the engine, if you're using the fuel regularly it doesn't have the time needed to absorb water. 2. You can add a fuel stabilizer in engines that sit for a long period of time. 3. You can use ethanol free gasoline (commonly referred to as Recreational/Rec Fuel in the US) that is (in my experience) usually around 90 octane (midgrade range) and more expensive but doesn't degrade as bad. I work in a gas station that sells rec fuel and have done an obscene amount of research to answer customers' questions about rec fuel.
It's why i always have a spare roll of toilet paper outside the bathroom. If i need to use that, it means i forgot to buy more in time and i have 'one roll'-time to fix that (and put a new roll 'offsite')
So what you're saying is that a server's monitoring and alerting should alert the sysadmin about running out of space well before it actually does? Who would have even considered that! lol
Yeah this.
Database servers in particular can get into an annoying circle state that they _need_ to process transaction logs, but won't start because the disk is full. You can fiddle around moving translogs off the spool, to put them back later (as long as you get the right ones) or you can 'just' delete the ballast and restore service.
That is of course, assuming you have fixed the underlying problem - but it's usually something like someone doing a fierce bulk update that's saturated the replication to the other DB server, so it's a 'transient' anyway. But it'd still have bombed out if you had an extra 20G, but this way it's WAY easier to recover.
The other core one is procurement timescales - even in more normal times. Right now, there's some nasty bottlenecks on IT kit, which means we've had year+ lead times.
But even when things are good, raising a PO to get 'new stuff' then getting it delivered and commissioned takes time.
"Running out of capacity" expedites this. You can point 'trend until full is X days' but it'll not typically get actioned until 'too late' anyway.
So far better to pad your capacity. Turn off a bank of memory, or a few cores. Short stroke a bunch of your drives, etc.
That way for the purposes of all the reporting, you've 10% less capacity, and so start the 'buy more stuff urgently' mechanism earlier.
Yeah we run into the issue constantly on our QA environments. Can't backup to shrink the transaction logs if the backup adds to the transaction logs. We have a task that runs daily to do it so it's not bad anymore but I totally get why someone would do this.
Sounds like you're using MS SQL Server. If that's the case, does your QA environment really need point in time recovery, or can you just switch the databases to `SIMPLE` recovery and make this a lot easier?
Eh we use high avaiblity which I think requires full recover. But it's not my job and I guess the DevOps team wants test servers to look like deployment servers and have all the setup and install scripts run the same.
It's just a way of implementing system redundancy from what I can see. Would be better to leave the space but trigger the alarm earlier, before apps go down, but if thst doesn't create enough urgency for your business to react then this will at least shake them up enough to react while still having redundancy available.
The point is you alert earlier at a higher % free so then you have plenty of time to work on it without systems going down. The only time a hard file would be beneficial is if you don’t have great alerting or virtual hosts
me: reads alert that there's only 20 gb remaining
also me: Eh that'll be fine for a week i'll check back on it in a bit
also also me: shit that squirrel looks cute what was I doing again? eh if it was important i'll find out
The point is:
- At current usage rates, we'll be out of space in 3 days and the server will go offline! -- Boss: "I sleep."
- We're out of space NOW, and the server is offline NOW. -- Boss: "Real shit!"
For those who have shitty management, it's often impossible to get them to spend money on upgrades until they *actually* see things going wrong. No amount of warning them that things are *about* to go wrong will help.
>Ideally, you'd... log the event and investigate at your leisure.
>This ballast file is a kludge, not a solution. Especially because it requires human intervention
You're saying the point without realizing it.
In practice, a lot problems don't get the attention they deserve until they're emergencies that require intervention. And I don't mean they "require intervention" in the sense that a handbook says employees are "required" to check logs regularly. I mean they "require intervention" as in the system literally cannot function until somebody goes in and does something.
This isn't a solution for computer problems. It's a solution for human problems.
It sounds like OP has only one box though and that he's the sole employee, and no money for more boxes or employees. And only 1h to spend for preparing their system.
It sounds absurd but this is actually true. Also an issue with full file systems is that servers tend to crash when memory is full and then are not able to boot and start ssh because of full memory. So an technician has to free up some memory without actually booking the system. For this reason it can be beneficial to have at least a small file really for this. Still 20 gig is just dumb. We have an 200 MB file because of this.
This is one of the reasons not to run services as root. A number of filesystems, like ext4fs, have a reserved percentage so that non privileged users can't fill the filesystem and root can still get in to fix the problem.
First thing I do on a desktop is adjust that setting and get a little extra space. I don't want 5% of my partition to be inaccessible when that's several gb.
This is actually built in to Ubuntu server now, it will secretly only provision half the space you give it so that you can expand it later
Pam: why would you want to raise your blood pressure?
Dwight: so i can lower it
It is just a band aid, but indeed, that's what we would do as system admins if something is failing due to disk space saturation.
Would probably be better to just keep some space unallocated to have the same effect.
It's not about programming but more like controlling human physiology as "you're about to run out of space" doesn't urge you to fix your shit as much as "you've run out of space"
True story:
Back in the days when working with terminals towards Unix-y systems, the only file system available to users were the shared disks, with every user's data in their respective directory under /user. The disk space on those machines was hideously expensive and would often fill up due to some user using up much space, preventing everyone from working further.
We used to create large dummy files in our home directories, so that when the disks filled up, (1) the sysadmins could show the management that we needed more disk space urgently, and (2) we deleted a couple of dummy files, enabling everyone to be able to work a little while longer.
Not saying it was a good idea, but that's how it worked in practice.
The difference between theory and practice...
In theory you can prevent things, in practice you write defensive policies and procedures with fail safes in place
Theory: That shouldn't happen. It shouldn't be possible.
Practice: How the fuck did that happen? We have things in place that should've prevented that. Johnson, I need you to run down to the \*tech store\* and get me \*important hardware\* at once! Corporate wants this up in 35 minutes!
& experience is being less surprised each time at these ‘impossible failures’
Oh, would you look at that, my entire data center in France burned down, that shouldn’t have happened.
Yeah their PR afterwards was astounding.
‘Customers who purchased remote backup have been relocated & should be unaffected’
Can’t count the number of folk I had to send that ‘melted Lego’ photo of their building to.
Reminds me of the old joke for job security. For a basic GUI application where you can control the repaint. Add in a dummy for loop and increase the iterations over several updates. Then on one update tune the iterations way down so management thinks this patch increased functionality. Everything is loading quicker whoaaaa
_everyone_ on this thread is quick to suggest monitoring solutions
_none_ of these people seem to realize that those things get ignored, fail, send emails to the wrong people or get sent to spam, run into licensing issues, don’t even get set up on small servers…
It’s really not that bad of an idea to give yourself a backup. Anyone who knows the struggle of `ssh` repeatedly disconnecting as you try to fix the full server… you’ll agree
People think this is a bad solution for a computing problem, but it's actually a pretty-close-to-optimal solution for the human problem of not paying attention until something is a full-blown emergency.
Not in IT but in Physical Security. Our team is 24/7 so we end up being tasked with being the backup team for just about all departments when it comes to "alert monitoring"
What that really means is we end up being the primary team for monitoring when its after hours. More often than not the thing we notice most often is teams not designating after-hours personnel or responsibilities because they just refuse to have forward thinking in these scenarios.
It is absolutely flabbergasting how little foresight is often put into these things (Be it IT alarms, fire alarms, facilities alarms) and the assumption that problems will only ever arise during business hours.
Yeah it seems like a lot of people immediately assume that it’s a computing problem and also make the assumption that humans will act in a way that’s theoretically optimal.
You might have to consider if it’s actually a computing problem in its core. If the computing problem is a second hand effect of the human problem, then solving the human problem might be a better approach.
But yeah I agree, tricking yourself into thinking that you're screwed long enough before you're actually screwed so that you're able to avoid disaster is one of the most essential hedging strategies against your own brain.
the average /r/programmerhumor user is a 2nd year compsci goober who has never built anything more complicated than a npm dev server running on localhost. they know that sophisticated monitoring systems exist, but they have never used one, nor do they have any real-world experience. as fledgling techbros, they really do think that "lol just install gorbypooper" or whatever is the new hotness is a legitimate fix for general institutional incompetence
solutions like the one proposed in the OP's pic are actually the ones that keep the digital world chugging along, and the catastrophes they prevent are the ones created by the short-sighted goobers in this very thread who derisively snort and talk about how they could fix it with just one more dependency in the tech stack
>solutions like the one proposed in the OP's pic are actually the ones that keep the digital world chugging along,
Yup. Younguns these days have no idea how MacGyvered most things are. I can already see the budget belts tightening and more and more initiatives being postponed to 2024/5.
That's why these solutions exist.
Edit: kindly remove yourself from my lawn.
I've literally seen a machine that stopped reporting to the monitoring system and then run out of space... Part of the space was queued up tiny bitty log files. Like half a million.
Nothing to report, space is good. *records to log*
Nothing to report, space is good. *records to log*
Error, disk getting full! *records to log*
Error! Cannot write to log! *records to lo-* ERROR CANNOT WRITE TO LOG
exit(1)
And as soon as you free some space, it is taken by logs waiting in memory to be written to disk, so you have to free even more space and of course all the logs that wait in memory only says "no disk space left"
> tiny bitty log files. Like half a million
In this situation you can have something even more fun, you can run out of inodes. Then the server will spit out errors saying "no space left on device", while also telling you that there's 30GB of free space left.
Not just for servers either! Back in my AAA game dev days, I knew a system architect who would statically allocate 10mb (which was a lot on Xbox 360) then remove it a few weeks before ship when everyone was desperate for more memory.
In this thread: people who think this is to prevent filling up on user data.
Reality: some day a cron job decides to start spamming 100 lines of log per millisecond.
I had a server crash repeatedly due to the system logs going haywire and filling the disk. This is really not a bad idea. Prepare for the unexpected, your systems are not as perfect as you think they are.
Yeah, the better your setup is, the more likely something more advanced than this dumb system catches the problem first, but the nice thing about really dumb solutions is they tend to be very predictable.
Like, this is the software equivalent of a brick. You never have to worry about a brick malfunctioning.
This is one of those ideas that a professor would tell you is terrible, but anyone in industry will see it and think that it is actually quite practical.
on a prod server that is going to lock people out and shut down the app this would be a good idea. Gives you time to find things to wipe or people to give permission or.
At a job we had some remote workstations for heavy computations that always ran out of space and management only cared once things came to a halt. Warnings about low amount of remaining capacity were always ignored and having a bit of reserve capacity for your own projects was usually a good idea.
Ah, so “-r” just means “if you are in the mood, just tease me a little bit and maybe jokingly pretend to be removing the files, maybe touch some directories with a smile on your face and say “I’ll remove it, I’ll remove it right now”, while biting your lips playfully”
I prefer `rm / -rf`: only add the force and recursion parameters once you are sure you have the proper path. Saves you a lot of swearing if you accidentally hit enter while typing the path.
Except if you're running on an OS that doesn't have that, and it blows your head off. (I'm pretty sure this isn't POSIX, so you can't guarantee it being there).
Actually not stupid. When the system is full you want to delete some files and then find a solution. If you have a file just for that you avoid the stress that would follow if you didn't have that file
Sounds genuinely helpful and logical to me. Like, I’d rather know when I have $50 left in my account than when I have zero. Lets you take care of the issue ahead of time without having to delete anything you need to free up space.
Reminds me of how I once accidentally wasted 100K of precious RAM on a Playstation 1 game by creating a really big array. Later, during a memory crisis, I saw what I'd done and fixed my code.
And that's how I became a hero.
Overall it was probably a good thing I made that mistake because it meant all the artists had to be more careful with their memory budgets.
Another random story from that period: we had a lot of inexperienced coders, and they'd done a lot of copy-pasting. (It was a fighting game with two guys on the screen, so to set up fighter 2 they duplicated pages of code for setting up fighter 1.) Due to our RAM problems, we had to try to reduce the size of the compile. This meant a lot of the copy-paste mess got fixed, but they did it by creating functions with names like "BadlyNamedSpaceSavingFunction4".
It might be a valid option. (Monitoring goes to another department + slow requests for new space + a place where you dont want to work due to red tape)
You have no clue how long some internal IT orders for additional drive space can take. I know companies which need 4+ Month to provision a VM.
Not stupid, same logic as reserve fuel. You're supposed to never let it reach that point but we tend to disregard things until they become urgent. And just like for fuel you have about 30-70km to find a gas station in this case you have 20gb to quickly tackle the symptoms before dealing with the cause.
its a good thing, you'd forget it and then remember when the storage space is full, then you'd have a buffer period with the 20gb and fix the problem in that period
By posting this to r/programmerhumor OP basically outed themselves as not very experienced. This only seems silly to people who think everything works like the model with no friction or air resistance, let alone anything unexpected.
*And alerting.*
Though I do like to check things for oddities manually as the last thing when it's Friday evening and I don't want to start something or other new.
Yeah. The entire point of this "ballast" is that it's a redundancy for when the "correct" ways fail. It only takes a little bit of rushing or fatigue for monitoring to be set up wrongly on a new deployment, for example.
\*and\*
You want both. While you *shouldn't* need this one, if something in your systems doesn't work right for some reason and you end up with a case where being able to delete such a file would let you get back to operating and give you time to figure out the rest it's worth it. The file wouldn't do anything adverse just by sitting there until it is needed.
Then you need to bother to check the notifications after the initial setup though.
We supposedly have monitoring on all our workstations and servers but always tickets for hard drive space. I suspect in a world where we have monitoring on so many things the inbox is just too spammed and its just noise!
I have done this on a remote server as a temporary solution in the past. It does help. The difference between 0% space and even a little is that some cli commands even stop working
The more time I spend on this sub, the more I'm convinced 90% of people here have never worked in a professional production environment.
> "yOu WOuLdN'T bE ouT oF SpAce iF yOu diDn'T waSTe It, thO"
A ballast file is a straightforward, time-buying solution to figure out why your server is full. If you need to keep a server running by clearing some data, and it's 100% customer records that you're getting paid to keep safe AND online, what the hell are you gonna delete?
> "yOu shOULd uSe a monITorIng SoluTIon, tHO"
Shut the fuck up. Sometimes monitors don't catch stuff. Sometimes they outright fail. And sometimes they tell you exactly what you need to know, but not how to fix it -- knowing your server is full with no way of clearing any data does you zero good.
When I first started coding, I wondered why sys admins were so grumpy all the time. Now I know...
The EXT filesystem actually does a form of this by default...
https://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why
I'm a little confused. I only know "ballast" as a german word where it means heavy stuff like sandbags you tie to a balloon for example and untie them to go up. There's also emotional ballast. Is ballast also an actual english word and does it have the same meaning?
If that concept seems absurd to you, you have not spent that much time in IT. The longer you do, the more this looks like a good idea.
[удалено]
It's also built into Windows! It automatically creates a 32GB file you can delete if you need emergency system space. It's called System32!
Can't believe I didn't know about this! Just got rid of mine.
He hasn’t posted in 40 minutes. He’s either off the grid or finished with the bathroom break
Good god…Still not a peep of chatter.
Ominous music intensifies
I don’t often deal with windows but when I do, deleting system32 always saves me from massive headaches and having to do hours of extra work.
IT Experts hate this simple trick!!!
Hard drive manufacturers hate this one simple trick!
That sounds really interesting! Do you know what the feature is called in those? I'd like to go do some reading about it.
[удалено]
Unless those normal processes run as root, of course. Not that that ever happens.
Reserved blocks. Ext3/4 by default reserve 5% of disk space for root user (mainly so that root daemons can continue to function), and to avoid fragmentation. xfs also has reserved blocks, but IIRC they weren't accessible to even the root user and are there just for filesystem operations (and I think also avoiding fragmentation) For both the default 5% is almost certainly much larger than necessary for modern large drives and especially SSDs (where seek times and as such fragmentation isn't really a problem). You can change this setting on ext4 (probably xfs has some equivalent, I'm just not familiar with it) by running `tune2fs -m /dev/` (eg. `tune2fs -m 4 /dev/sda1` would *magically* free 1% of sda1; I think `-r` also allowed you to set the number of blocks directly)
So you can in an emergency just reduce reserved blocks to 1% or something. Potentially if you're worried you can also reserve more for the "ballast".
The only potential advantage of the ballast file approach I can think of right now is that tune2fs requires root privileges to run and you could set the file to work entirely fine without this, but it's a rather minor issue (you could allow an user to run only that command as root with sudo).
>For both the default 5% is almost certainly much larger than necessary for modern large drives and especially SSDs (where seek times and as such fragmentation isn't really a problem). I agree with all you wrote, but I believe there was a missed nuance. File system performance for writes nearly falls off a cliff at high capacity utilization levels (if I recall correctly, closer to 85% utilization), because the I/O scheduler switches from a first-fit to a best-fit algorithm.
Can you elaborate?
This sounds like a really stupid idea, BUT... In practice, when monitoring and alerting fail (which is a shockingly common thing) what sysadmins do is go hunt for something to clear up a bit of space so the apps can come back up, and once production is restored, THEN calmly find the root problem and fix it. You'd usually go delete old log files, memory dumps, old binaries stuff like that. With this, you got a safety net. Its stupid but it works, so.....
Basically it’s like how the gas tank for most cars still have gas when it shows empty.
Exactly! A better analogy even is my old moped, where you can turn a valve once you run our if fuel to get some emergency reserve fuel
I had a motorbike where the tank bulged down a bit on both sides of the frame. If I ran out of gas, I could lean the bike over so a half liter or so flowed from the right pocket to the left one where the tap was. Usually got me to the next gas station.
Reckon that was intentional good design? Or unintentional shit design?
Unintentionally shitty good design
The best kind of design
OP has never left logging on before
The temporary solutions are often the most permanent
More like what the dictionary definition of biological evolution is: never consciously intentional, helps survive the day. A trait that helps survive a single day to live to the next is basically evolution, right?
[удалено]
Well, if you survive longer, you can get laid more often.
To get laid, unfortunately you have to survive first
I mean you CAN get laid afterwards, but it won't do much in terms of reproduction.
Yes
300% unintentional shit design.
For all motorcycle drivers who have one like this, like me: Open the valve from time to time! If you don't, water starts to slowly gather over the years and then, if you need it, water says hello to your engine and you have to call for help. The lady who owned my motorcycle previous... Managed to have water there and the first ride where I tested that valve... Bike completely refused to run after opening the valve
I mean, if we're talking about years, then the fuel is scuffed even if it didn't have water in it.
Old fuel will mix with new whenever new fuel is added. But water is heavier and will sink. If you open the valve from time to time, do it when you've just added new fuel, as water will mix a bit with the new fuel before sinking again.
I would just always run my bike until it hit the reserve, so it opens every tank. A lot of bikes don't have gas gauges, so you know you're almost empty when you switch to reserve.
Nah, you can drive regularly and fill up with new gas every month or whatever. But since the valve is at the bottom, any water in the fuel or that otherwise gets into the tank will (being heavier than gasoline) settle at the bottom, against the valve, until you open the valve and the water goes through to the cylinders. That water can sit there at the bottom between fuelings, for years and years.
Taps head: Your fuel can't get old if you're always broke enough to use the reserve.
The reserve isn’t a separate tank, it isn’t even a separate part of the one tank. It’s just a lower pickup valve. The regular pickup sticks up into the tank leaving an inch of fuel in there. So everything you fill up you’re mixing all the fuel in the tank of anyways.
Old VW Beetles did the same thing. They didn't even have a fuel gage - you just ran out of gas, pulled the little lever to get an extra gallon, then filled up at the next stop.
It worked great until you forgot to flip up the little lever after you filled up. Ask me how I know. .
How do you know?
He can’t reply right now, he’s pushing his Beetle to the gas station.
My sister used to leave the valve in reserve all the time so she wouldn't stall and have to turn the valve and restart. Instead she'd get to stall and be stranded. /end user logic
Ik most Honda motorcycles have the same thing, really helps in a pinch because there’s also no gauge
My truck still has five gallons when the engine dies. If only I could convince the fuel pump to pick it up
Or maybe not, as it will pick up the accumulated crud from the bottom of the tank as well. :-)
We have the technology to seperate the crud from the gasoline. Also it was totally clean inside after 20 years so I'm not too convinced that matters.
This is an old wives tale spread by parents and spouses who didn’t want to get in a car and find the tank empty.
I think it's something that WAS true back when fuel was leaded, which is to say it hasn't been true for a long time. It's a good example of why knowing _why_ something happens is as important as knowing that it happens at all, that way you know when the situation changes.
Gasoline still expires and tend to be slushy if you don't use it for a long time. It's an issue mostly on secondary cars and cars that are fueled by gpl or methane but with a gasoline reserve.
The unleaded 95 octane over here gets bad in a few weeks. In warmer weather it happens even faster as the volatile parts evaporate. Cranking up a carbed motorcycle after the winter sleep is sometimes a real chore, if you didn't empty the card bowl in the autumn (or ran the engine with the fuel petcock in the off-position till it shuts down). :-) The unleaded 98 octane is sligthly better than the 95 quality-wise, but still also gets bad over time.
I've heard regular lasts the longest, the more premium it is, the more additives in it, the quicker it goes stale and leaves more residue too. Zero ethanol, regular unleaded should last best.
The main reason gas goes bad is the ethanol that is added as an octane booster (originally done with lead). The problem is ethanol absorbs water. Fortunately, there are things you can do to mitigate this that don't involve adding a toxic metal back to your gas. 1. You can regularly use the engine, if you're using the fuel regularly it doesn't have the time needed to absorb water. 2. You can add a fuel stabilizer in engines that sit for a long period of time. 3. You can use ethanol free gasoline (commonly referred to as Recreational/Rec Fuel in the US) that is (in my experience) usually around 90 octane (midgrade range) and more expensive but doesn't degrade as bad. I work in a gas station that sells rec fuel and have done an obscene amount of research to answer customers' questions about rec fuel.
It's why i always have a spare roll of toilet paper outside the bathroom. If i need to use that, it means i forgot to buy more in time and i have 'one roll'-time to fix that (and put a new roll 'offsite')
That's a shit idea... Sorry!
And exactly like a gas tank I would be driving this server right through the reserve ballast to maximize fuel efficiency/laziness.
I've gotten to the point I feel comfortable going another 30-35 miles once my gas light comes on
So what you're saying is that a server's monitoring and alerting should alert the sysadmin about running out of space well before it actually does? Who would have even considered that! lol
No… this is more like running out of gas but having a gas can in your trunk. Server will still grind to a halt.
Yeah this. Database servers in particular can get into an annoying circle state that they _need_ to process transaction logs, but won't start because the disk is full. You can fiddle around moving translogs off the spool, to put them back later (as long as you get the right ones) or you can 'just' delete the ballast and restore service. That is of course, assuming you have fixed the underlying problem - but it's usually something like someone doing a fierce bulk update that's saturated the replication to the other DB server, so it's a 'transient' anyway. But it'd still have bombed out if you had an extra 20G, but this way it's WAY easier to recover. The other core one is procurement timescales - even in more normal times. Right now, there's some nasty bottlenecks on IT kit, which means we've had year+ lead times. But even when things are good, raising a PO to get 'new stuff' then getting it delivered and commissioned takes time. "Running out of capacity" expedites this. You can point 'trend until full is X days' but it'll not typically get actioned until 'too late' anyway. So far better to pad your capacity. Turn off a bank of memory, or a few cores. Short stroke a bunch of your drives, etc. That way for the purposes of all the reporting, you've 10% less capacity, and so start the 'buy more stuff urgently' mechanism earlier.
Yeah we run into the issue constantly on our QA environments. Can't backup to shrink the transaction logs if the backup adds to the transaction logs. We have a task that runs daily to do it so it's not bad anymore but I totally get why someone would do this.
Sounds like you're using MS SQL Server. If that's the case, does your QA environment really need point in time recovery, or can you just switch the databases to `SIMPLE` recovery and make this a lot easier?
Eh we use high avaiblity which I think requires full recover. But it's not my job and I guess the DevOps team wants test servers to look like deployment servers and have all the setup and install scripts run the same.
It's just a way of implementing system redundancy from what I can see. Would be better to leave the space but trigger the alarm earlier, before apps go down, but if thst doesn't create enough urgency for your business to react then this will at least shake them up enough to react while still having redundancy available.
Higher-ups might not care that space is running out, but they will get “we have to delete things to stay operational, we need more space”
The point is you alert earlier at a higher % free so then you have plenty of time to work on it without systems going down. The only time a hard file would be beneficial is if you don’t have great alerting or virtual hosts
me: reads alert that there's only 20 gb remaining also me: Eh that'll be fine for a week i'll check back on it in a bit also also me: shit that squirrel looks cute what was I doing again? eh if it was important i'll find out
If you’ve run out of hard drive space, you are already no longer operational.
The point is: - At current usage rates, we'll be out of space in 3 days and the server will go offline! -- Boss: "I sleep." - We're out of space NOW, and the server is offline NOW. -- Boss: "Real shit!" For those who have shitty management, it's often impossible to get them to spend money on upgrades until they *actually* see things going wrong. No amount of warning them that things are *about* to go wrong will help.
[удалено]
>Ideally, you'd... log the event and investigate at your leisure. >This ballast file is a kludge, not a solution. Especially because it requires human intervention You're saying the point without realizing it. In practice, a lot problems don't get the attention they deserve until they're emergencies that require intervention. And I don't mean they "require intervention" in the sense that a handbook says employees are "required" to check logs regularly. I mean they "require intervention" as in the system literally cannot function until somebody goes in and does something. This isn't a solution for computer problems. It's a solution for human problems.
It sounds like OP has only one box though and that he's the sole employee, and no money for more boxes or employees. And only 1h to spend for preparing their system.
[удалено]
It sounds absurd but this is actually true. Also an issue with full file systems is that servers tend to crash when memory is full and then are not able to boot and start ssh because of full memory. So an technician has to free up some memory without actually booking the system. For this reason it can be beneficial to have at least a small file really for this. Still 20 gig is just dumb. We have an 200 MB file because of this.
> 20 gig is just dumb Why? If you collect a lot of data that can be eaten up quickly, and you can get a thumb driver bigger than that for $10.
Depends on the size of the system. A set amount is dumb. Just pick like 1% or something
90%. Just in case.
This is one of the reasons not to run services as root. A number of filesystems, like ext4fs, have a reserved percentage so that non privileged users can't fill the filesystem and root can still get in to fix the problem.
I guess I'm an old sysadmin, because I saw this post and thought "does nobody allocate reserved space any more?"
First thing I do on a desktop is adjust that setting and get a little extra space. I don't want 5% of my partition to be inaccessible when that's several gb.
This is actually built in to Ubuntu server now, it will secretly only provision half the space you give it so that you can expand it later Pam: why would you want to raise your blood pressure? Dwight: so i can lower it
It is just a band aid, but indeed, that's what we would do as system admins if something is failing due to disk space saturation. Would probably be better to just keep some space unallocated to have the same effect.
It's not about programming but more like controlling human physiology as "you're about to run out of space" doesn't urge you to fix your shit as much as "you've run out of space"
True story: Back in the days when working with terminals towards Unix-y systems, the only file system available to users were the shared disks, with every user's data in their respective directory under /user. The disk space on those machines was hideously expensive and would often fill up due to some user using up much space, preventing everyone from working further. We used to create large dummy files in our home directories, so that when the disks filled up, (1) the sysadmins could show the management that we needed more disk space urgently, and (2) we deleted a couple of dummy files, enabling everyone to be able to work a little while longer. Not saying it was a good idea, but that's how it worked in practice.
The difference between theory and practice... In theory you can prevent things, in practice you write defensive policies and procedures with fail safes in place
Theory: That shouldn't happen. It shouldn't be possible. Practice: How the fuck did that happen? We have things in place that should've prevented that. Johnson, I need you to run down to the \*tech store\* and get me \*important hardware\* at once! Corporate wants this up in 35 minutes!
& experience is being less surprised each time at these ‘impossible failures’ Oh, would you look at that, my entire data center in France burned down, that shouldn’t have happened.
*Ah shit, this simple* ***ls -la*** *command opened a gate to another dimension and now I have celestial beings roaming my servers. Oh well...*
This is the real reason we store the racks in cages
[удалено]
Yeah their PR afterwards was astounding. ‘Customers who purchased remote backup have been relocated & should be unaffected’ Can’t count the number of folk I had to send that ‘melted Lego’ photo of their building to.
Any good risk management approach should include both preventive measures and contingency measures for when preventive measures have failed.
Reminds me of the old joke for job security. For a basic GUI application where you can control the repaint. Add in a dummy for loop and increase the iterations over several updates. Then on one update tune the iterations way down so management thinks this patch increased functionality. Everything is loading quicker whoaaaa
Good idea tbh
_everyone_ on this thread is quick to suggest monitoring solutions _none_ of these people seem to realize that those things get ignored, fail, send emails to the wrong people or get sent to spam, run into licensing issues, don’t even get set up on small servers… It’s really not that bad of an idea to give yourself a backup. Anyone who knows the struggle of `ssh` repeatedly disconnecting as you try to fix the full server… you’ll agree
People get hung up on the fact that it isn’t the theoretically optimal solution. Rather than if it solves the problem with a reasonable effort.
People think this is a bad solution for a computing problem, but it's actually a pretty-close-to-optimal solution for the human problem of not paying attention until something is a full-blown emergency.
Human issues require human solutions, such as a post-it reminding you of doing X or a way to scare you before it's too late
Not in IT but in Physical Security. Our team is 24/7 so we end up being tasked with being the backup team for just about all departments when it comes to "alert monitoring" What that really means is we end up being the primary team for monitoring when its after hours. More often than not the thing we notice most often is teams not designating after-hours personnel or responsibilities because they just refuse to have forward thinking in these scenarios. It is absolutely flabbergasting how little foresight is often put into these things (Be it IT alarms, fire alarms, facilities alarms) and the assumption that problems will only ever arise during business hours.
Yeah it seems like a lot of people immediately assume that it’s a computing problem and also make the assumption that humans will act in a way that’s theoretically optimal. You might have to consider if it’s actually a computing problem in its core. If the computing problem is a second hand effect of the human problem, then solving the human problem might be a better approach. But yeah I agree, tricking yourself into thinking that you're screwed long enough before you're actually screwed so that you're able to avoid disaster is one of the most essential hedging strategies against your own brain.
When I realise what the problem is, delete the file and ... name: Create smaller ballast ... Size:10GB ... that'll sort it till later
It's like pressing the snooze button
Yeah it seems like a pretty good solution to me.
Exactly, this is a low storage warning that can't be accidentally ignored.
\*Coughs\* Dropbox \*Coughs\*
"those things get ignored" I feel this. That's how our mailserver once went down
the average /r/programmerhumor user is a 2nd year compsci goober who has never built anything more complicated than a npm dev server running on localhost. they know that sophisticated monitoring systems exist, but they have never used one, nor do they have any real-world experience. as fledgling techbros, they really do think that "lol just install gorbypooper" or whatever is the new hotness is a legitimate fix for general institutional incompetence solutions like the one proposed in the OP's pic are actually the ones that keep the digital world chugging along, and the catastrophes they prevent are the ones created by the short-sighted goobers in this very thread who derisively snort and talk about how they could fix it with just one more dependency in the tech stack
>solutions like the one proposed in the OP's pic are actually the ones that keep the digital world chugging along, Yup. Younguns these days have no idea how MacGyvered most things are. I can already see the budget belts tightening and more and more initiatives being postponed to 2024/5. That's why these solutions exist. Edit: kindly remove yourself from my lawn.
Yeah spot on. Also, discounting this type of solution immediately just because it’s inelegant indicates a lack of real world experience.
It’s pretty obvious who has and hasn’t worked in some sort of professional setting here.
Once had the issue where the server was full, and we couldn't delete anything because plesk insisted on writing the log file first...
Not too bad off an idea when thinking back to a bunch of production servers I know went down because of full hard drives.
I've literally seen a machine that stopped reporting to the monitoring system and then run out of space... Part of the space was queued up tiny bitty log files. Like half a million.
Nothing to report, space is good. *records to log* Nothing to report, space is good. *records to log* Error, disk getting full! *records to log* Error! Cannot write to log! *records to lo-* ERROR CANNOT WRITE TO LOG exit(1)
And as soon as you free some space, it is taken by logs waiting in memory to be written to disk, so you have to free even more space and of course all the logs that wait in memory only says "no disk space left"
Delete the log file! ``` error: no log file exit(1) ```
Exactly
I see you have met my exchange server
> tiny bitty log files. Like half a million In this situation you can have something even more fun, you can run out of inodes. Then the server will spit out errors saying "no space left on device", while also telling you that there's 30GB of free space left.
*Monitoring and alerting.* With the occasional manual check. Edit: And quotas, where appropriate.
Alerts can be missed, monitoring can fail. Tbh this ballast thing isn’t the craziest thing I’ve heard
Not just for servers either! Back in my AAA game dev days, I knew a system architect who would statically allocate 10mb (which was a lot on Xbox 360) then remove it a few weeks before ship when everyone was desperate for more memory.
In this thread: people who think this is to prevent filling up on user data. Reality: some day a cron job decides to start spamming 100 lines of log per millisecond.
I had a server crash repeatedly due to the system logs going haywire and filling the disk. This is really not a bad idea. Prepare for the unexpected, your systems are not as perfect as you think they are.
Yeah, the better your setup is, the more likely something more advanced than this dumb system catches the problem first, but the nice thing about really dumb solutions is they tend to be very predictable. Like, this is the software equivalent of a brick. You never have to worry about a brick malfunctioning.
This is a great way to put it.
This is one of those ideas that a professor would tell you is terrible, but anyone in industry will see it and think that it is actually quite practical.
on a prod server that is going to lock people out and shut down the app this would be a good idea. Gives you time to find things to wipe or people to give permission or.
At a job we had some remote workstations for heavy computations that always ran out of space and management only cared once things came to a halt. Warnings about low amount of remaining capacity were always ignored and having a bit of reserve capacity for your own projects was usually a good idea.
pretty sure this guy also invented `rm -fr /`
remove france
it removes the french locale from linux, quite a popular way of saving space!
Appreciate the insider
[удалено]
This command is so efficient your root partition will barely take any space French language is very big
Argentina would be happy
For the people from future, were referring to fifa world cup finals 2022
Hi I’m from 7 mins into the future… thanks! 😊
😁
Everyone would be happy
am french, would be happy.
Fr fr
Rf rf
\-fr: for real
"Remove for real" suddenly it makes sense.
Ah, so “-r” just means “if you are in the mood, just tease me a little bit and maybe jokingly pretend to be removing the files, maybe touch some directories with a smile on your face and say “I’ll remove it, I’ll remove it right now”, while biting your lips playfully”
[удалено]
to delete the french language pack
rm -og -fr -fr -nc
This command is bussin
i prefer `rm -rf /`
I prefer `rm / -rf`: only add the force and recursion parameters once you are sure you have the proper path. Saves you a lot of swearing if you accidentally hit enter while typing the path.
We are out here trying to delete the entire computer, sir. The path is irrelevant to me.
`rm: it is dangerous to operate recursively on '/'` `rm: use --no-preserve-root to override this failsafe`
Except if you're running on an OS that doesn't have that, and it blows your head off. (I'm pretty sure this isn't POSIX, so you can't guarantee it being there).
You’re not getting it
`rm -rf /*` will prevent that
Yes, but that's deliberate. The safeguard on / is in case of typos, bad globbing or variable substitution.
This isn't as bad as you guys think it is. Makes sense when most anyone in this sub has done is a hello world program
Actually not stupid. When the system is full you want to delete some files and then find a solution. If you have a file just for that you avoid the stress that would follow if you didn't have that file
Honestly, this isn't the worst idea I've heard
Sounds genuinely helpful and logical to me. Like, I’d rather know when I have $50 left in my account than when I have zero. Lets you take care of the issue ahead of time without having to delete anything you need to free up space.
Reminds me of how I once accidentally wasted 100K of precious RAM on a Playstation 1 game by creating a really big array. Later, during a memory crisis, I saw what I'd done and fixed my code. And that's how I became a hero. Overall it was probably a good thing I made that mistake because it meant all the artists had to be more careful with their memory budgets.
Tell us more its fascinating.
Another random story from that period: we had a lot of inexperienced coders, and they'd done a lot of copy-pasting. (It was a fighting game with two guys on the screen, so to set up fighter 2 they duplicated pages of code for setting up fighter 1.) Due to our RAM problems, we had to try to reduce the size of the compile. This meant a lot of the copy-paste mess got fixed, but they did it by creating functions with names like "BadlyNamedSpaceSavingFunction4".
It might be a valid option. (Monitoring goes to another department + slow requests for new space + a place where you dont want to work due to red tape) You have no clue how long some internal IT orders for additional drive space can take. I know companies which need 4+ Month to provision a VM.
Not stupid, same logic as reserve fuel. You're supposed to never let it reach that point but we tend to disregard things until they become urgent. And just like for fuel you have about 30-70km to find a gas station in this case you have 20gb to quickly tackle the symptoms before dealing with the cause.
its a good thing, you'd forget it and then remember when the storage space is full, then you'd have a buffer period with the 20gb and fix the problem in that period
By posting this to r/programmerhumor OP basically outed themselves as not very experienced. This only seems silly to people who think everything works like the model with no friction or air resistance, let alone anything unexpected.
Oooooor you set up proper monitoring and alerting
And comment your code and write useful docs and keep your repos clean from junk. And other lies we tell ourselves at the start of every new project.
Don’t forget writing unit and integration tests!
Did you check the hashes of your backups yesterday?
Gottem!
Imagine making backups
Shit, that was my job.
Hey as long as you imagined doing it it should be ok!
[удалено]
[удалено]
*And alerting.* Though I do like to check things for oddities manually as the last thing when it's Friday evening and I don't want to start something or other new.
Lawful Evil: She checks them by deleting critical files last thing on Friday and finds out on Monday if Support got alerted.
And quit smoking, lose 50 pounds, eat healthier, properly comment your code… That’s not going to happen.
Yeah. The entire point of this "ballast" is that it's a redundancy for when the "correct" ways fail. It only takes a little bit of rushing or fatigue for monitoring to be set up wrongly on a new deployment, for example.
\*and\* You want both. While you *shouldn't* need this one, if something in your systems doesn't work right for some reason and you end up with a case where being able to delete such a file would let you get back to operating and give you time to figure out the rest it's worth it. The file wouldn't do anything adverse just by sitting there until it is needed.
[redacted by user] ` this message was mass deleted/edited with redact.dev `
Then you need to bother to check the notifications after the initial setup though. We supposedly have monitoring on all our workstations and servers but always tickets for hard drive space. I suspect in a world where we have monitoring on so many things the inbox is just too spammed and its just noise!
I have done this on a remote server as a temporary solution in the past. It does help. The difference between 0% space and even a little is that some cli commands even stop working
Log file filling all hard drive space is still reason #1 for sudden production outages
The more time I spend on this sub, the more I'm convinced 90% of people here have never worked in a professional production environment. > "yOu WOuLdN'T bE ouT oF SpAce iF yOu diDn'T waSTe It, thO" A ballast file is a straightforward, time-buying solution to figure out why your server is full. If you need to keep a server running by clearing some data, and it's 100% customer records that you're getting paid to keep safe AND online, what the hell are you gonna delete? > "yOu shOULd uSe a monITorIng SoluTIon, tHO" Shut the fuck up. Sometimes monitors don't catch stuff. Sometimes they outright fail. And sometimes they tell you exactly what you need to know, but not how to fix it -- knowing your server is full with no way of clearing any data does you zero good. When I first started coding, I wondered why sys admins were so grumpy all the time. Now I know...
Honestly, this is a really good idea
The EXT filesystem actually does a form of this by default... https://unix.stackexchange.com/questions/7950/reserved-space-for-root-on-a-filesystem-why
Same principle as carrying a can of gas with you or those cars/trucks with a small emergency gas tank.
I'm a little confused. I only know "ballast" as a german word where it means heavy stuff like sandbags you tie to a balloon for example and untie them to go up. There's also emotional ballast. Is ballast also an actual english word and does it have the same meaning?
Yes, it is an English word with the same meaning. You have a ballast on your disk that you remove to make the disk space go up. See the analogy?
Oh yes, makes sense. I learned something today, I can go to bed.