The talk linked at the end is really interesting, even as someone who doesn't know anything about filesystems outside of the classic Unix they teach in school and reading a lot of deep discussions about tuning ZFS. (None of my ZFS users are strenuous enough to require any tuning but I can't help but learn and optimize when there are levers to pull).
It's interesting that it's implementing all these modern features on top of extent-based non-COW transactional btrees (though it looks like APFS is the same way). With the very database-like architecture at the low level, it would be fun to stick a SQL interface on top. You should be able to do advanced queries very efficiently without having to fall back to scanning every file in the directory like you would normally have to.
I'm salivating looking at the caching/tiering features. People often try to implement a tiered filesystem with ZFS L2ARC, SLOG, and now the special vdev. But none of those features were intended for that and it shows.
I was able to solo-developer build a reasonably complex CI/CD system for a company that relied heavily on bcachefs in a tiering system: RAM --> SSD --> HDD, to run hundreds of builds per day across dozens of SKUs measuring in the tens to hundreds of gigabytes per build directory. Later was able to port bcache (not the fs) to a vmx driver for VMWare esxi because of that understanding I had gained earlier in the project, again, RAM to SSD to HDD. Which then lead to making a down'n'dirty FusionDrive script (it's on github) for macOS that can tier RAM and SSD for doing fast CI/CD agent builds on macOS.
Without getting in to any specifics of how bcache and bcachefs works, because I refuse to be known as the guy who mansplained how bcache works to Mssr Overstreet on HN, or as an older generation would say, "teaching Grandma how to suck eggs."
I worked for a time at a game studio (many times actually, but we’re just talking about this one time specifically) between 2016-ish and early 2020, and got involved with a project that had absurdly long build times – 7+ hours per build per SKU. Woefully under powered equipment to run the CI/CD on. Performed obvious h/w upgrade, but build times were still measured in 2+ hours. It was Unity3D, and it will do mysteriously inscrutable lengthy builds because some asset changed, which causes a cascade of other assets and modules to be rebuilt. These builds are too big to fit purely in to a RAM disk on any reasonably priced h/w I could convince the powers that be to obtain, at least at that time, given that I was a fresh face at the game studio and didn’t have the political capital built up yet. Also, they were tight fisted when it came to spending money on h/w.
Using initially bcache I was able to get the build times down to 35-ish minutes per SKU, some would still take 90-ish minutes because of a big asset check-in from the art team. I was mostly using bcache as a smart RAM disk – files permanently stored on HDD, with an SSD tier (256GB or 512GB IIRC), and then a RAM tier. It was a little brittle at times, we lost a few builds, but for the most part everything worked, and losing a build due to running in to an issue with bcache (very rare, usually attributable to PEBKAC) and with more tweaking, got the build times for each SKU under 20-minutes. We had about 30 SKUs for iOS, Android, Windows & Linux and almost everything was Unity3D for the game engine.
From there I started working with bcachefs; same tiering setup: RAM -> SSD -> HDD. Unity3D, as explained, does mysterious builds, and is single-threaded, and needs to grind its way through 15GB of high resolution art assets in PSD, MB, AI formats, run custom build scripts, etc. I have a few machines, with a little digging in to the Unity3D project indexing format I was able to make it so that a machine could take a few assets, fire up an instance of Unity3D, “compile” the assets, then send them back to a common cache. This is technically how Unity3D works – cache assets previously compiled – it’s really bad at it most of the time though, and at the time, when switching between target platforms, it would blow away its own asset cache. I could use bcachefs, because of the CoW, to keep a “safe copy” of the Unity3D cache on a HDD, make the entire project cache rapidly available to a Linux build agent to determine what assets needed to be rebuilt, and then when that was done, move the project cache to a MacOS or Windows machine for final assembly. And with more tweaking, and some horrendous Bash scripts, got our average build time under 7 minutes for our project. This was a multi-vendor project, and our studio became the studio that mastered all the builds and produced them sooner than anyone else.
Because of that work, the Director of Engineering said “can you do that for our other teams?” and that began another epic quest, but those other teams, they obviously had their own way of doing things and were resistant to change. I was first able to move them to their own individual installs of TeamCity running in a Docker container for doing the CI/CD, and then VMs (Linux, Windows, MacOS (shhh! Don’t tell anyone I virtualized our build Macs for our non-release builds on non-Apple h/w)) to build each game for each target platform with each game’s odd environmental quirks, e.g. MacOS for iOS and Android, Windows for PlayStation and XBOX, Linux for some other thing. Well all this is great but I was tasked with trying to squeeze a gallon in to a pint pot – “We should have a dedicated build server” said I. “These are the specs we need with many cores and much DDR4s.” And then a bean counter said “naw dog, I get by with this little 13” laptop for running my Excel, how about a nice quad-core with 32GB of RAM, and we’re being generous here. We’ll even throw in an SSD if you behave.”
Eventually, we got a machine with hundreds of cores and hundreds of gigs. And I couldn’t figure out why it didn’t perform how expected until I looked at the SSDs and talked to people and they said “We saved $50 buying our M.2 from Amazon, you should be happy, it was _twice_ the size of that one you specced.” And thus began a lengthy discussion of why I specced a U.2 enterprise drive with a fucking huge queue depth and no write-cliff. Once I got the h/w I wanted, well, we cannot run RAM disks in each individual VM because that will eat all of our RAMs. So I looked around and found a company that had a tiering driver, RAM -> SSD -> HDD and I said “hey, that sounds like bcache!” and I reached out to the company and they said “sure, it’s a $1,000. Per year. Per CPU.” Well that ain’t gonna fly, I just convinced the company to buy me $12,000 of h/w and two VMWare ESXi licenses, they aren’t going to spring for $4,000 a year in software licenses. So I did what any delusional developer with balls bigger than his brains would do – I said “I’ll write my own! It should only take a weekend or two!” Hahaha! Ha!
I started with the bcache source, and over several weeks of late nights and weekends was able to build it as a storage driver for VMWare’s particular flavour of Linux with all of its badly documented internals. I cribbed a lot of notes with the help of BinaryNinja and that other company’s bcache-like driver.
With bcache now loading as a storage driver in to VMWare, I could set up a group of VM’s and point them at a bcache (RAM -> SSD -> HDD) storage area that contained all the data each VM, in whatever client OS and target platform would need to create a binary build. And because it is bcache, well, we’re both caching an awful lot of very similar chunks of data, and also effectively giving each VM its own resizable smart RAM drive, without having to setup a RAM drive inside of each VM. The downside of this is that if my hacked up bcache vmx driver were to crash, or the SSD go offline, every VM using that for storage comes to a screeching halt and loses whatever data is stored. But that didn’t matter, because these are build machines – OS and tools stored on one virtual drive – build directory containing source code, assets and binary output stored on a less reliable, but far faster virtual drive. If my driver crashes, we simply restart each VM and the start-up script for the VM will recreate the bcache’d storage tier.
Right around this time Apple was dropping hints that they’d be moving away from Intel h/w, and also, for final builds on Apple h/w, things ran as slow as molasses in a Minnesota winter – mostly due to the fact that these are final builds and nothing to do with the fact that the bean counters decided that an underpowered five year old Mac with a 5400RPM HDD and 8GB RAM was sufficient for a critical piece of company infrastructure. 8GB isn’t enough RAM to create a RAM disk to hold the entire project in, nowhere close enough, but MacOS has this concept of a FusionDrive, whereby they would sell you 1TB HDD with a 32GB SSD glued on to it, and unlike several of the hybrid drives that are available in non-AppleLand, Apple had the “wisdom” to build the ability to make a FusionDrive right in to the OS – it was the OS file system driver that directly handled the gluing together. I was able to use that built-in facility to make a FusionDrive that glued a RAM Disk and an external SSD together which in any sane environment would be an absolutely fucking terrible idea. But for a machine that just spits out builds all day long, and doesn’t care if the entirety of the SSD contents are wiped out due to someone tripping over a power cable or unplugging the SSD or turning off the machine, it didn’t matter, just run this FusionDrive script at start-up and everything is back as it should be.
There was other fun things around that CI/CD project, such as how to get Linux to use a VPN where the client IT department has determined that only macOS, Windows, iOS and Android are permitted to connect. "Well we just send the magic strings that tell the remote VPN h/w that we're really an iOS device and we don't need any of their spyware installed thankyouverymuch." Interesting side-note, got a C&D for sharing that information from "large vendor of network h/w." Further interesting note: Engineer working at large vendor of network h/w posted a script on github on how to do that exact thing under their personal account about two months later.
That's really cool. I love hearing stories like this, it's nice to know my code is getting used in interesting ways :)
The RAM disk tiering thing has come up before, and it always strikes kernel developers as a gross hack because that's what the page cache is supposed to be for.
Perhaps we should expose a knob that completely disables fsync, for applications like this - then, dirty pages would only be written out by memory pressure.
My memory is from three or four years back now, and there are many more lines of code that have gone over the weir since then, so I am sketchy on certain details but one of the things I looked in to, when working with bcache, was how to disable certain features of the code that were clearly hitting an IOPS limit, one of those being the dirty page cache, rather than letting memory pressure handle the flushing of the cache. I was not smart enough or educated enough about the code to figure it out for myself.
Bcachefs is one of the most interesting projects going on in Linux kernel land these days, all the more remarkable because it is mostly the work of one person over the last 12 years. I hope it goes upstream soon; with a bit of polish it should be ready for widespread adoption.
Does anyone know if any of these next generation filesystems can do raid6 with more flexibility than normal linux software raid?
Say you do a 6x4TB raid 6 array in Linux and get a 16TB virtual drive. Now if you update 4 of the drives to 8TB will still get 16TB with 4x4TB of fully unused space. But 4x4TB is itself enough to do 8TB of raid6 that would get you a total of 24TB usable space where any two drives can fail. Does any of these filesystems just do that automatically and continuously upgrade the capacity as you phase in new disks?
Linux raid does that already. You can also add disks to grow an array or change it from raid5 to raid6. The gradual phase-in is what's missing. It makes sense as having what is effectively a raid0 of a 4 drive raid6 and a 6 drive raid6 in the example is a more complex configuration with uneven performance across the array. But for a home NAS meeting the reliability with maximum space ends up being all that counts.
I didn't understand even half of that but am still excited. I have no less than 6 external HDDs and SSDs lying around and it's been a pain to make a good stable ZFS pool out of them. `bcachefs` seems like a perfect match.
I eagerly await this work stabilizing to the point of merging to mainline. I'm impressed with how much progress Kent has made, but I'm frustrated by how often he seems to say some feature is basically ready except for this bug and except for that problem. I understand great designs often come from the mind of a single inspired individual, but I hope that he's ready to accept contributions from others and that more developers see the value in helping to get this across the finish line.
But the reality is that filesystem development is hard, and there aren't that many people clamoring to work on this stuff - but that's ok. Slow and steady wins the race, in the end.
I'm a big believer in steady incremental development, and being up front about what works and what doesn't - and also making sure the core is a solid foundation for everything we want to do.
For anyone who does want to get involved - come join us! irc.oftc.net#bcache
Unfortunately, file system development is a pretty niche skill set these days, and the majority of the experts in the field are employed maintaining existing file systems (ext4, xfs, apfs, etc).
One thing I’ve been bugging Kent to do is to write documentation about the design and internal workings on bcachefs; very little about modern file system design is actually written down anywhere, and a detailed reference manual would attract more people to work in this area.
Copy on write (COW, like zfs or btrfs), Full data and metadata checksumming, Multiple device support, Replication, Compression, Encryption, Snapshots, Caching
> The lock
ordering rules had become too complicated and this was getting us too many
transaction restarts, so I stole the standard technique from databases
Any details/links that explain this? The article seems to suggest that they used to detect the deadlock as it was about to happen, and then abort everything and retry. This doesn't seem too different from "when they happen". What is the optimization?
Previously, we were checking for lock ordering violations. But a lock ordering violation isn't a deadlock, just a potential deadlock.
Checking for a lock ordering violation is simpler because you only have to look at data structures for the current thread - you're just looking at what other locks the current thread is holding.
To check for a deadlock you have to do a full DFS of threads blocked on locks held by other threads, looking for cycles. Quite a bit trickier, but it was well worth it :)
Thanks for the explanation. Very interesting! I assume you only do the DFS when you detect a lock ordering violation? So if there is the potential for a deadlock you'll do extra work to make sure that you're actually deadlocked before aborting?
Just curious about the mention of persistent data structures. Is this about the FP notion of persistent data structures (which I suspect) or on-disk data structures? Or both? :)
I find that (FP) persistent data structures are like a super-power in lots of ways. (And it only costs a log-N slowdown in the worst case.)
I’m currently using a few different filesystems for different use-cases, and looking forward to bcachefs because (as I understand it) it covers some use-cases that I’ve been dreaming of for ages and not yet found any solution for:
- raid5/6 with odd-sized disks (btrfs does raid1 with odd-sized disks, mdadm & zfs do raid5/6 with equally-sized disks) - great for a homelab server that I upgrade ship-of-theseus style rather than the enterprise “buy a whole new server with whole new larger disks in one go”
- SSD as a cache integrated with HDD RAID (right now I’m using bcache + mdadm + ext4 for this usecase; having the filesystem coordinate all three of those layers feels more elegant and efficient)
- Different settings for different directories in the same filesystem (my super-important documents get RAID1 across 4 drives, my blu-ray rips that I can just re-rip if I lose them get RAID0, everything else gets RAID5)
It's interesting that it's implementing all these modern features on top of extent-based non-COW transactional btrees (though it looks like APFS is the same way). With the very database-like architecture at the low level, it would be fun to stick a SQL interface on top. You should be able to do advanced queries very efficiently without having to fall back to scanning every file in the directory like you would normally have to.
I'm salivating looking at the caching/tiering features. People often try to implement a tiered filesystem with ZFS L2ARC, SLOG, and now the special vdev. But none of those features were intended for that and it shows.