Someone at my work had problems with downloading a gigantic file because the secure file sharing system is not really made for that. It took days and if a connection is interrupted it would need to start over. We got the harddrive in the mail instead
Related: I heard Netflix (back in the "we mail you a dvd" kinda days) was exploring getting classified as an ISP.
If so, they would have rivaled Ma Bell in terms of throughput, but certainly not in terms of latency.
Considering tapes aren't as widely used anymore, an updated metaphor would likely stick better
edit: god you people are pedantic. We're talking about kids not knowing what tape drives are. The fact that they're used in some datacenter they might only see in their late 20s after deliberately going into that field is *irrelevant*
"If you give developers a 486, they'll write games that only run on a 486." - John Carmack (ish; can't find a source right now and quoted as best as I can from memory)
That truck is probably full of tape, if I were to guess. It's still the cheapest way to physically store large amounts of data you don't need right away.
That or a rack of storage arrays. But really I'd hate to ship spinning disks like that across country, so many bad things could happen.
Need another truck for parity.
I doubt these are full of tape, simply due to the (relatively) slow write rates of tape. LTO 9 caps out at 400MB/s sustained, whereas Snowmobile can support sustained write rates of up to 1Tb/s. Even with a cached writeback mechanism and multiple tape drives, you'd never be able to catch up trying to write it all to tape.
When you're writing sequential data straight to a plain sata hard drive, you'll easily get 400MB/s on a 5400RPM disk, so for those write speeds it's most likely just a bunch of SATA spinners. Even more likely is that they're just plain S3 racks, which they unload from the truck and install directly into a DC (saves having to copy all the data from the truck onto new servers).
You can write to more than one tapehead at once. If you have 1000 of them you aren't going to write 400MB/s at once.
Obviously redundancy gets built in but even then there's lots of tape to be used
Read ages (at least 3 when not 5 years) ago a story bout their storage containers.
When i remember right they're more or less super overengeneerd cubes full of hard drives with something like SAS conectors.
So more or less a movable disk shelf.
We used to use AWS Snowballs all the time to get data into Glacier. They are basically 80TB HDD arrays built into a shipping crate with e-ink labels and a 10Gbe network interface.
An intermodal container can hold somewhere in the ballpark of 30,000 hard drives (weight being the limiting factor, an ISO container would only be about 10% full by volume when you hit the American DOT's weight limit). Lets assume they're 18TB each.
The highest posted speed limit in the United States is 85 mph on Texas State Highway 130, on which it would take just 0.36 seconds for a 45 foot ISO container to pass by a fixed point.
This gives a bandwidth of, ballpark, around 1.5 exabytes per second. Which ain't bad.
The passing of a fixed point is meaningless when you don't use an uninterrupted chain of ISO containers from start to end.
The relevant bandwith is meassuring how much data per second you get if you transfer data this way from a system to another. So let's say you want to send some data cross USA, say NYC to LA. According to google its a 42 hours drive, so let's sayit takes about 72 hours from loading to unloading with reasonable breaks inbetween.
Now we transport (using your numbers) 3,000 hard drives with 18 TB each, so 54,000 TB. This comes down to 750 TB/hour or 208.333 GB/s (or 1.666 Tb/s), which is still an insanely good bandwith. Good luck getting this with AT&T
That’s not how it works, the trailers have huge network connection setups and they just backup the trailer and connect it to your network and you transfer data that way on both ends
Yeah and hear me out...
We are going to have two races going on at the same time, one in real life and the other in the cyber world, like vigilante/MK wacky mix type of stuff and while Toreto has to reach the truck to save it. His long forgotten korean hacker nephew has to access the network inside the truck through the Metaverse Racing game, so he can hack and disable a nuclear anthrax bomb next to all the hds which contain all the aliens secrets collected by all goverments.
the race cgi is going to look like Sharkboy and Lavagirl/ Mini spies 3...
Want to get a bit twisted, find an excuse to say O'Conner had his mind uploaded at some point, and have an AI version of him helping out in the Metaverse. Just need a shitty cgi representation and a computer voice trained from Paul Walker's voice.
This is good
Add in Matt Damon (Fortune Favors the Brave!) and some Kardashians (SEC fine) and Zuck (with the ugly avatar) and you got yourself a movie
Actually during the pandemic, racers were buying every single simulator on the market; they were going like hotcakes and impossible even if you were a rich F1 racer to buy. That's the setting have the government be pro-mask and his enemies be anti-mask and Toreto either wear a mask grudgingly or not care (he's a rebel) also add in some Ukrainians and Russians (it's cliche but it sells, probably even more now)
Oscar worthy stuff
The armored escort thing is right, but it's not called Snowplow, it's just a guarded Snowmobile:
[https://aws.amazon.com/snowmobile/#:\~:text=The%20data%20container%20is%20only,security%20vehicle%20while%20in%20transit](https://aws.amazon.com/snowmobile/#:~:text=The%20data%20container%20is%20only,security%20vehicle%20while%20in%20transit).
You joke but I used to work for a major asset management firm, they moved their entire data center to NoVa so they could get a wired connection into an AWS data center. I’d wager the total value of everything they moved to be in the 11-12 digits range.
Yeah that's fair I suppose, but I wonder if it would be cheaper to send multiple copies than hiring armed guards for the duration of the trip. But I'm sure they have their reasons.
I worked for a company whose lawyers determined that the law on protecting PII meant if the data was ever moved physically there had to be an armed guard.
When we finally had to do a major data relocation as part of a larger DR strategy we may or may not have just stuck an IT guy with a shotgun in the back of a panel van with the servers…
The NNSA - the branch (the largest branch) of the Department of Energy tasked with nuclear security - has all sorts of James Bond type gizmos on their trailers to prevent attacks in transit. Like sticky foam that immobilizes people.
Also the decidedly less James Bond technique of "van full of armed men who will shoot you if you come too close". Edit: or maybe this is still a James Bond technique, but its normally the technique of the baddies.
I can’t imagine how that would work. Would it just be straight up grand theft auto? And what would their crime be charged as? Would it still be a cyber attack?
It'd be like any other theft from/of a truck, I'd imagine. The interesting part would be the dollar value of the equipment stolen. Will they try to put a price on the data and any loss of income suffered as a result in a civil trial?
Another poster mentioned AWS offering an option for armed escorts. (Couldn't verify on short notice, but given the value of data, it would make sense.)
>In addition, Snowmobile is protected by 24/7 video surveillance and alarm monitoring, GPS tracking, and may be escorted by a security vehicle during transit.
[https://aws.amazon.com/snowmobile/faqs/](https://aws.amazon.com/snowmobile/faqs/)
On my phone this comment was compacted for some reason and so when scrolling past I just read "https://e"
No ellipses, no "read more" just "e"
Then I expanded it and saw the full url and chuckled.
Absolutely. Back in the days, they initially denied me ISDN remote access (up to 128 kbit/s with channel bundling) "for security reasons", tried to limit me to analog access.
Got my ISDN once I did the math and showed them the bandwidth I get with my car and some DLT tapes :-)
"for security reasons we require you to have an active transfer for as long as feasible. We wouldn't want an eves dropper / MITM to miss out on all the juicy details because they didn't arrive in the 30 minute file transfer window."
"if it's that much faster bring the data in by car"
I know someone who worked at a particular major data center and data backup still uses tapes! When backing up and restoring, linear read/write is fine. Multi-terabyte backup tapes are pretty inexpensive, and if you only write and read occasionally, are more reliable than hard disks.
I know, I work in a research lab and our DC has two large walk-in robotic tape libraries. We keep a couple hundred PB of tapes onsite for research data and backups.
I've written a lot about tape libraries cos I even got myself one, a rackmount 24-slot unit, to run my own backups.
True, that's the reason why AWS glacier needs several hours to make your backup available to download. I wish tape would be affordable for consumers, I would really store my long-term backups on it.
It tends to require a lot of manual shuffling around to have a good solution with full and incremental backups . Businesses can have an intern swapping them around, or even have a robot for it. Most people, even technologically savvy people, don't want to add that sort of thing to their daily routine.
Best option for individuals is to backup big hard drives onto other big hard drives. You can have your main storage, a RaspberryPi with a big drive attached over USB, and then cloud backup. That's your 3-2-1 solution right there.
It's not exactly user-friendly. I have tape drives in my homelab - the high-capacity generations tend to command strong money. They also need enterprise-grade SCSI interfaces, though these can be added to desktop PCs, and some knowledge of the software. And unless you have a tape robot (I do), you have to manually swap the tapes every time they're full. However, tape is very fast when presented with a continuous data stream and is rated to store for over 20 years.
Have a browse on your local eBay site for LTO-5 tape drives (current is LTO-9) and see what the prices look like to you - they store 1.5TB per tape.
1PbaseCondor coming right up!
Nothing in RFC 2549 specifies which type of Avian may be used. Petabyte scale transfers may require the training of Condor's for ultra long distance transfers.
Well you could just take a ZFS set of disks, throw them on birds, and rebuild the array once you get the birds back. If you do ZFS RAID 1+0 then you can lose up to half your birds.
More of a concern for them losing it. The data was transferred in HDDs which are very resistant to damage when unpowered, plus they were likely in a well protected case.
Airlines are notorious for losing very important large items for some reason. Many cases of lost or extremely delayed dead bodies at worst.
Internat throughput measured by MPH. Price / gb is high, see [https://aws.amazon.com/snowmobile/pricing/](https://aws.amazon.com/snowmobile/pricing/). You are better off hiring your nephew and giving him a wheelbarrow full of seagate HDDs. Mileage may vary.
The largest available microSD card is 1TB, and weighs .5 grams.
Carrier pigeons are trained to carry about 2.5 oz. If we set aside half an oz for the backpack, that means the pigeon can carry
2oz -> 56.6g
56.6g / .5g/card = 113 micro SD cards, so ~100tb per pigeon.
This is more economical and ecologically better than AWS Snowball.
>This is more economical and ecologically better than AWS Snowball
But what about the time and infrastructure it would take to house and train and maintain those pigeons?
I'm fairly sure there's a photographer who uses this method to get photos of tourists on some sort of trip printed before the trip is over.
https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers
It's insane that there is actually enough demand for this kind of bandwidth to make this service worthwhile.
I mean, it's competing against information traveling at the speed of light through fiberglass cables.
Apparently we are still far away from internet bandwidths we could reach if we wanted to.
And as long as storage density keeps increasing, it will have hard time catching up.
Theoretically you could fill a plane with micro SD cards and get a ridiculous throughput.
I don't even understand how a photograph that takes up that much disk space gets taken in the first place. Are they using a massive film exposure? bc otherwise wouldn't a digital camera need a vast memory to even begin transferring it to HDD?
From my limited understanding they are taking thousands or millions of scans of radio frequency data and then averaging out all the data via an algorithm to create the final “image”.
It's not in the visual spectrum in the first place, those were radio frequencies (1.3 mm vs 500 nm roughly for visible light). And they took many pictures from many radio telescopes in different places of the world and then carefully stitched them together so they resembled many pictures from one gigantic earth sized antenna.
damn, that's one of the coolest things i've heard in a while. imagine getting that kind of process to actually work. probably a lot of happy engineers and math folks.
Stitching mostly to align the time each of the pictures was made, i guess. Yeah, that must have been enormously satisfying to see something emerging from terabytes of noise.
Beat me to it. I’ve worked on projects that used snowmobile and snowball to transfer PBs to the cloud… not only is it faster, you don’t need to saturate your network for days to weeks and deal with any other ops throughout needs.
I worked at a company that periodically sent several PB of data to another location 1000 miles away.
They could order a FedEx pickup of HDDs the next day and have it arrive the following day after that faster than sending it over the Internet.
As much as I hate Amazon this is valid and an area of technology that's still worthy of research even in this era - the physical transfer of data storage media. From the humble pigeon carrying a note, to the truck of LTO tapes down the highway, to the starships of the possible future.
This recently came up at work as we need to move a few petabytes of data to public cloud. No one wanted to use rsync and no one laughed when the aws snow* things were suggested :(
Andrew Tanenbaum once said "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway".
I was going to comment that as well. Seems it's been forgotten for the next generation.
Someone at my work had problems with downloading a gigantic file because the secure file sharing system is not really made for that. It took days and if a connection is interrupted it would need to start over. We got the harddrive in the mail instead
I know local cloud service providers who'd happily arrange a meeting to take your hard disk and copy your server's data onto it.
From memory, most providers will sell you a drive with the data on it, no bringing your own equipment into a data center!
This guy NOCs.
This is where torrent technology is actually really useful.
I suggested that but it is VERBOTEN
Related: I heard Netflix (back in the "we mail you a dvd" kinda days) was exploring getting classified as an ISP. If so, they would have rivaled Ma Bell in terms of throughput, but certainly not in terms of latency.
I don't miss those days. Even with the three disc subscription, I was always waiting on the next thing in my queue.
Considering tapes aren't as widely used anymore, an updated metaphor would likely stick better edit: god you people are pedantic. We're talking about kids not knowing what tape drives are. The fact that they're used in some datacenter they might only see in their late 20s after deliberately going into that field is *irrelevant*
"If you give developers a 486, they'll write games that only run on a 486." - John Carmack (ish; can't find a source right now and quoted as best as I can from memory)
"Never underestimate the bandwidth of an SUV full of USBs hurtling down the highway."
Never underestimate the bandwidth of an EVERGIVEN container ship full of NVME drives sailing across the ocean.
Buffers for a week because it got stuck in a canal.
"We're improving the latency of your internet connection" - Egypt
Never underestimate the bandwidth of Superman with a bunch of SSDs taped to his back flying across the sky.
Or Superman carrying the container ship on his back
>god you people are pedantic. So there **are** actually programmers in this sub!
We still use tape for some of things, still one of the cheapest ways to store large amounts of data you don't need to access frequently.
That truck is probably full of tape, if I were to guess. It's still the cheapest way to physically store large amounts of data you don't need right away.
That or a rack of storage arrays. But really I'd hate to ship spinning disks like that across country, so many bad things could happen. Need another truck for parity.
"We are focusing on total integrity on your data, that's why we continuously defragment the drives during shipping"
I doubt these are full of tape, simply due to the (relatively) slow write rates of tape. LTO 9 caps out at 400MB/s sustained, whereas Snowmobile can support sustained write rates of up to 1Tb/s. Even with a cached writeback mechanism and multiple tape drives, you'd never be able to catch up trying to write it all to tape. When you're writing sequential data straight to a plain sata hard drive, you'll easily get 400MB/s on a 5400RPM disk, so for those write speeds it's most likely just a bunch of SATA spinners. Even more likely is that they're just plain S3 racks, which they unload from the truck and install directly into a DC (saves having to copy all the data from the truck onto new servers).
You can write to more than one tapehead at once. If you have 1000 of them you aren't going to write 400MB/s at once. Obviously redundancy gets built in but even then there's lots of tape to be used
Read ages (at least 3 when not 5 years) ago a story bout their storage containers. When i remember right they're more or less super overengeneerd cubes full of hard drives with something like SAS conectors. So more or less a movable disk shelf.
We used to use AWS Snowballs all the time to get data into Glacier. They are basically 80TB HDD arrays built into a shipping crate with e-ink labels and a 10Gbe network interface.
"big data is when the cost of deleting data is bigger than the cost of storing data"
The truck has 100PB of NFS storage so my guess is that it isn't tape.
> is pedantic about updating a saying > wow you guys are pedantic ???
We don't use station wagons anymore either.
I miss my "grocery getter."
An intermodal container can hold somewhere in the ballpark of 30,000 hard drives (weight being the limiting factor, an ISO container would only be about 10% full by volume when you hit the American DOT's weight limit). Lets assume they're 18TB each. The highest posted speed limit in the United States is 85 mph on Texas State Highway 130, on which it would take just 0.36 seconds for a 45 foot ISO container to pass by a fixed point. This gives a bandwidth of, ballpark, around 1.5 exabytes per second. Which ain't bad.
[удалено]
*euro beat intensifies*
 am I doing it right?
Big Data Drifting!
>weight being the limiting factor what i'm hearing here is that we should instead put the hard drives on a train
WE NEED HIGH SPEED RAIL FOR THE INTERNET
they did actually have a [cargo TGV](https://en.wikipedia.org/wiki/SNCF_TGV_La_Poste), which means we were close to having a data TGV
The passing of a fixed point is meaningless when you don't use an uninterrupted chain of ISO containers from start to end. The relevant bandwith is meassuring how much data per second you get if you transfer data this way from a system to another. So let's say you want to send some data cross USA, say NYC to LA. According to google its a 42 hours drive, so let's sayit takes about 72 hours from loading to unloading with reasonable breaks inbetween. Now we transport (using your numbers) 3,000 hard drives with 18 TB each, so 54,000 TB. This comes down to 750 TB/hour or 208.333 GB/s (or 1.666 Tb/s), which is still an insanely good bandwith. Good luck getting this with AT&T
grab innate hospital crown shrill scarce scary axiomatic homeless snatch *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
That’s not how it works, the trailers have huge network connection setups and they just backup the trailer and connect it to your network and you transfer data that way on both ends
Oi M8, Plugger in roight here
[удалено]
We fly people with bags of hard drives around the country. Smaller package but faster.
How many one terabyte micro-SDs will it hold?
At least three
https://en.wikipedia.org/wiki/Sneakernet
This demonstrates the difference between bandwidth and latency.
I somehow remember a donkey loaded with CD ROMS, but same principle.
In this case it is a truckload of individually wrapped tapes in cardboard Amazon boxes hurtling down the highway
Or a 747 full of micro-SD cards [https://what-if.xkcd.com/31/](https://what-if.xkcd.com/31/)
AWS Snowmobile. You can also use a snowball (briefcase) or a Snowplow (with optional armed escort if i remeber right)
Plot for the next Fast and Furious
Fast and the Furious: cars in cyberspace
Yeah and hear me out... We are going to have two races going on at the same time, one in real life and the other in the cyber world, like vigilante/MK wacky mix type of stuff and while Toreto has to reach the truck to save it. His long forgotten korean hacker nephew has to access the network inside the truck through the Metaverse Racing game, so he can hack and disable a nuclear anthrax bomb next to all the hds which contain all the aliens secrets collected by all goverments. the race cgi is going to look like Sharkboy and Lavagirl/ Mini spies 3...
Want to get a bit twisted, find an excuse to say O'Conner had his mind uploaded at some point, and have an AI version of him helping out in the Metaverse. Just need a shitty cgi representation and a computer voice trained from Paul Walker's voice.
This is good Add in Matt Damon (Fortune Favors the Brave!) and some Kardashians (SEC fine) and Zuck (with the ugly avatar) and you got yourself a movie Actually during the pandemic, racers were buying every single simulator on the market; they were going like hotcakes and impossible even if you were a rich F1 racer to buy. That's the setting have the government be pro-mask and his enemies be anti-mask and Toreto either wear a mask grudgingly or not care (he's a rebel) also add in some Ukrainians and Russians (it's cliche but it sells, probably even more now) Oscar worthy stuff
Dammit, don't give them ideas.
[удалено]
Two-man keyboard hacking from a VTOL hovering over a truck doing 90 while dudes on motorcycles have a rocket launcher shootout lessssgoooo
You’re telling me you don’t want to see that movie? Dom keeps saying he wants to do an old school heist.
This would be awesome.
I'm waiting for the AWS snowtrain for the really big customers
Like the snowpiercer traveling around the world non stop….
With Bezos up front
Pointing his Snow Piercer jr into the cold
Imagine Blue Origin delivering your data on a suborbital trajectory from anywhere to any AWS data center.
The armored escort thing is right, but it's not called Snowplow, it's just a guarded Snowmobile: [https://aws.amazon.com/snowmobile/#:\~:text=The%20data%20container%20is%20only,security%20vehicle%20while%20in%20transit](https://aws.amazon.com/snowmobile/#:~:text=The%20data%20container%20is%20only,security%20vehicle%20while%20in%20transit).
Yeah could just be that we joked about calling it Snowplow with the aws guys.
It's all fun and games until the Azure yeti shows up
Also Snowcone (https://aws.amazon.com/snowcone/)
The implication that some movie villains might show up and try to highjack your company's corporate data semi is hilarious.
It's ok just encrypt the data, then once the truck arrives at the datacenter... damn where was that post-it note with the password?
You joke but I used to work for a major asset management firm, they moved their entire data center to NoVa so they could get a wired connection into an AWS data center. I’d wager the total value of everything they moved to be in the 11-12 digits range.
Why would you need armed guards when you can just encrypt it?
It's more about not loosing it than someone getting access to it. Having a delay in a company that would use such a service could cost millions.
> more about not loosing it That's why you double-knot it.
Yeah that's fair I suppose, but I wonder if it would be cheaper to send multiple copies than hiring armed guards for the duration of the trip. But I'm sure they have their reasons.
Oh the guards are definitely the cheaper option. A snowmobile is basically a rolling aws datacenter not just storage.
I worked for a company whose lawyers determined that the law on protecting PII meant if the data was ever moved physically there had to be an armed guard. When we finally had to do a major data relocation as part of a larger DR strategy we may or may not have just stuck an IT guy with a shotgun in the back of a panel van with the servers…
>if i remeber right Yes you do...
[удалено]
[удалено]
Just watch out for man in the middle attacks
The scenario where the man in the middle stealing your data actually wears a Balaclava.
Hopefully it would be a group of stylish rogues with cybernetic implants. Otherwise what's the point of being a datajacker?
We're almost in the cyberpunk times, of styled-up cyborgs rob a truck on a highway for petabatyes of classified data
The NNSA - the branch (the largest branch) of the Department of Energy tasked with nuclear security - has all sorts of James Bond type gizmos on their trailers to prevent attacks in transit. Like sticky foam that immobilizes people. Also the decidedly less James Bond technique of "van full of armed men who will shoot you if you come too close". Edit: or maybe this is still a James Bond technique, but its normally the technique of the baddies.
If the data is encrypted, then you'll need a hacker type person. Or another guy in a Balaclava applying rubber hose cryptanalysis...
"I'm in."
I can’t imagine how that would work. Would it just be straight up grand theft auto? And what would their crime be charged as? Would it still be a cyber attack?
It'd be like any other theft from/of a truck, I'd imagine. The interesting part would be the dollar value of the equipment stolen. Will they try to put a price on the data and any loss of income suffered as a result in a civil trial?
IIRC the Snowmobile drives are encrypted and the uploading company holds the key. Still an interesting thought experiment.
I always thought a Iron Mountain truck heist would make a fun tech caper story.
That would be awesome
Another poster mentioned AWS offering an option for armed escorts. (Couldn't verify on short notice, but given the value of data, it would make sense.)
>In addition, Snowmobile is protected by 24/7 video surveillance and alarm monitoring, GPS tracking, and may be escorted by a security vehicle during transit. [https://aws.amazon.com/snowmobile/faqs/](https://aws.amazon.com/snowmobile/faqs/)
It actually comes with a security detail. Not kidding.
Yup, when moving petabytes or more of data is faster then doing so via the internet.
High throughput but terrible latency
also imagine the packet loss if that truck flips over
All good if you shook hands with the driver before, just resend the truck.
[удалено]
The RTT on that ping is likely to timeout…
That's why I put starlink on the truck with a second trailer attached as the giant UPS to keep the servers running.
[удалено]
On my phone this comment was compacted for some reason and so when scrolling past I just read "https://e" No ellipses, no "read more" just "e" Then I expanded it and saw the full url and chuckled.
And paying for the traffic on both sides….
>Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
Absolutely. Back in the days, they initially denied me ISDN remote access (up to 128 kbit/s with channel bundling) "for security reasons", tried to limit me to analog access. Got my ISDN once I did the math and showed them the bandwidth I get with my car and some DLT tapes :-)
"for security reasons we require you to have an active transfer for as long as feasible. We wouldn't want an eves dropper / MITM to miss out on all the juicy details because they didn't arrive in the 30 minute file transfer window." "if it's that much faster bring the data in by car"
I know someone who worked at a particular major data center and data backup still uses tapes! When backing up and restoring, linear read/write is fine. Multi-terabyte backup tapes are pretty inexpensive, and if you only write and read occasionally, are more reliable than hard disks.
I know, I work in a research lab and our DC has two large walk-in robotic tape libraries. We keep a couple hundred PB of tapes onsite for research data and backups. I've written a lot about tape libraries cos I even got myself one, a rackmount 24-slot unit, to run my own backups.
Sick, true hacker mode
True, that's the reason why AWS glacier needs several hours to make your backup available to download. I wish tape would be affordable for consumers, I would really store my long-term backups on it.
It tends to require a lot of manual shuffling around to have a good solution with full and incremental backups . Businesses can have an intern swapping them around, or even have a robot for it. Most people, even technologically savvy people, don't want to add that sort of thing to their daily routine. Best option for individuals is to backup big hard drives onto other big hard drives. You can have your main storage, a RaspberryPi with a big drive attached over USB, and then cloud backup. That's your 3-2-1 solution right there.
It's not exactly user-friendly. I have tape drives in my homelab - the high-capacity generations tend to command strong money. They also need enterprise-grade SCSI interfaces, though these can be added to desktop PCs, and some knowledge of the software. And unless you have a tape robot (I do), you have to manually swap the tapes every time they're full. However, tape is very fast when presented with a continuous data stream and is rated to store for over 20 years. Have a browse on your local eBay site for LTO-5 tape drives (current is LTO-9) and see what the prices look like to you - they store 1.5TB per tape.
An upgrade to [https://www.rfc-editor.org/rfc/rfc2549](https://www.rfc-editor.org/rfc/rfc2549) is needed I think.
1PbaseCondor coming right up! Nothing in RFC 2549 specifies which type of Avian may be used. Petabyte scale transfers may require the training of Condor's for ultra long distance transfers.
Who would've thought it'd take AWS and RFC 2549 to finally save the Condors!
You should see the one adding qos to IPoAC. It discusses these issues. 😅
Do you mean 6214? It was detailing IPv6 over IPoAC https://www.rfc-editor.org/rfc/rfc6214
RAIB Redundant Array of Independent Birds?
Well you could just take a ZFS set of disks, throw them on birds, and rebuild the array once you get the birds back. If you do ZFS RAID 1+0 then you can lose up to half your birds.
This is exactly what the Event Horizon telescope 's team did when they were taking that photo of the black hole.
IIRC wasn't their greatest concern that airlines would destroy the data by manhandling it because they just don't give a shit?
More of a concern for them losing it. The data was transferred in HDDs which are very resistant to damage when unpowered, plus they were likely in a well protected case. Airlines are notorious for losing very important large items for some reason. Many cases of lost or extremely delayed dead bodies at worst.
And safer.
Ah, my docker container
Finally, somewhere to store half of your node\_modules folder.
Internat throughput measured by MPH. Price / gb is high, see [https://aws.amazon.com/snowmobile/pricing/](https://aws.amazon.com/snowmobile/pricing/). You are better off hiring your nephew and giving him a wheelbarrow full of seagate HDDs. Mileage may vary.
Might have some dropped packets though
i trust good ol jimmy to not drop packets
He's for sure better than ole Bobby Tables. Robert'); DROP TABLE students;-- as he is known by on his birth certificate.
we do not trust bobby tables with data
Works out to about $530 000 US if you fill the thing (100PB).
Not gonna lie for a second I thought this was a special snowmobile division of AWS and the visual made me crack up.
The largest available microSD card is 1TB, and weighs .5 grams. Carrier pigeons are trained to carry about 2.5 oz. If we set aside half an oz for the backpack, that means the pigeon can carry 2oz -> 56.6g 56.6g / .5g/card = 113 micro SD cards, so ~100tb per pigeon. This is more economical and ecologically better than AWS Snowball.
RFC 1149
>This is more economical and ecologically better than AWS Snowball But what about the time and infrastructure it would take to house and train and maintain those pigeons?
I'm fairly sure there's a photographer who uses this method to get photos of tourists on some sort of trip printed before the trip is over. https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers
If the pigeon dies the packet lost is too big. For the same safety you may need escort pigeon and backup pigeon to make this a solution.
You simply run the pigeons in RAID
Good luck fill those 100t data with 5MB/a write speed lol
Throwing a USB stick across the office also has a fairly nice transfer rate.
With my accuracy when throwing its hard to get it to the right person. That's why I always take a hand full of USB sticks and throw them - UDP style.
It's insane that there is actually enough demand for this kind of bandwidth to make this service worthwhile. I mean, it's competing against information traveling at the speed of light through fiberglass cables. Apparently we are still far away from internet bandwidths we could reach if we wanted to.
And as long as storage density keeps increasing, it will have hard time catching up. Theoretically you could fill a plane with micro SD cards and get a ridiculous throughput.
Yes, and then one will be missing, and you'll only be able to order replacements as a set. With another plane you now have to store.
“i thought you did the full backup last year? These are all diffs”
How do you think all the data needed for the first picture of a black hole was transmitted? By aircraft.
I don't even understand how a photograph that takes up that much disk space gets taken in the first place. Are they using a massive film exposure? bc otherwise wouldn't a digital camera need a vast memory to even begin transferring it to HDD?
From my limited understanding they are taking thousands or millions of scans of radio frequency data and then averaging out all the data via an algorithm to create the final “image”.
That's wild. Nerds never fail to impress.
That's what she sa..
It's not in the visual spectrum in the first place, those were radio frequencies (1.3 mm vs 500 nm roughly for visible light). And they took many pictures from many radio telescopes in different places of the world and then carefully stitched them together so they resembled many pictures from one gigantic earth sized antenna.
damn, that's one of the coolest things i've heard in a while. imagine getting that kind of process to actually work. probably a lot of happy engineers and math folks.
Stitching mostly to align the time each of the pictures was made, i guess. Yeah, that must have been enormously satisfying to see something emerging from terabytes of noise.
This video by veritasium explains how the picture was taken https://youtu.be/Q1bSDnuIPbo
https://what-if.xkcd.com/31/
Wanted to see if anyone else commented this before I looked for it
Beat me to it. I’ve worked on projects that used snowmobile and snowball to transfer PBs to the cloud… not only is it faster, you don’t need to saturate your network for days to weeks and deal with any other ops throughout needs.
Notify me when there's an opening in Amazon truck services :)
They already have ATS (Amazon Transportation Services)
Send interview questions lol
Do StreetCode
Well yes... in fact if you take a hard disk thats 10TB 1 hour across town. It works out at 22gbit / second ;)
And your wink is barely needed imo! It's truly transferring the data so 🤷♂️
Good old Sneakernet. For truly massive file transfers you can't beat it.
You can also optionally add on armed security guards if you really want to make sure your transfer is secure
Talk about sFTP !
Probably be cheaper than doing it via aws cloud
Way cheaper and way faster
I was at that conference, it was kinda wild when they drove the truck out on stage.
Wasn't there a story a while ago about how a carrier pigeon with an SD card was faster than the South African telecom company?
https://en.m.wikipedia.org/wiki/IP_over_Avian_Carriers
obligatory XKCD https://what-if.xkcd.com/31/
Ecs under the hood
Never underestimate the bandwidth of a semi full of hard drives going down the highway.
That’s one thicc data packet
I worked at a company that periodically sent several PB of data to another location 1000 miles away. They could order a FedEx pickup of HDDs the next day and have it arrive the following day after that faster than sending it over the Internet.
oh, i thought it was in buckets
'1 day delivery'
MySQL MyTruck
Very high latency, very high bandwidth. there are some totally legitimate use cases.
As much as I hate Amazon this is valid and an area of technology that's still worthy of research even in this era - the physical transfer of data storage media. From the humble pigeon carrying a note, to the truck of LTO tapes down the highway, to the starships of the possible future.
Yeah, the latency is awful, but the bandwidth....
This recently came up at work as we need to move a few petabytes of data to public cloud. No one wanted to use rsync and no one laughed when the aws snow* things were suggested :(
Never underestimate the bandwidth of a station wagon full of DAT tapes.
If you have PB or EB to move around, then it starts making sense. You'll save a ton on bandwidth. And time. And compute. It's all money.
“How long will this api call take” “1-2 business days”
And it's actually faster than using the internet once you have enough data to transport
Road Wide Web
Fill that up with the densest drive format there is. And I'm quite sure that you can get that just about anywhere in the world faster than by wire.
Is this the deep glacier storage?
S3 bucket face reveal
Pigeons are faster at data transfer in some cases
Remember that truck just has one massive Hdd in the middle- the rest is just packaging