A few years ago I got on the MongoDB bandwagon. The pitch was scale: MongoDB was web scale, and I wanted my application ready for the internet, even if the first customers were going to be mates from the back streets of East Legon. I wrote the application and I wrote tests for it, and the tests failed in a way I could not explain. I called InsertOne, then called FindOne with the same id in the assertion, and nothing came back. The data I had just written was not there.
This was before AI assistants, so I spent a few days reading logs, driver documentation and forum threads. Then I gave up, moved the project to Postgres, and swore off MongoDB.
It took me years to understand what had probably happened. MongoDB at the time shipped with defaults that favoured speed over safety. The drivers did not wait for the server to confirm a write, so InsertOne could return before the database had done anything with my document. On the server side, data files went to disk roughly once a minute, so even a write the server had applied could exist only in memory for a while.
I had assumed that “the function returned” meant “the data is stored”. The once-a-minute flush puzzled me for a long time, because writing to disk sounded like the whole point of a database. Only later did I see that several layers sit between those two statements, and that each layer optimises for its own goals. The reasons behind those goals took me a few more years to work out.
The bottom layer, the one closest to the disk, is the file system. Learning what it does and why it behaves the way it does changed how I read the durability guarantees of any system that stores data on disk. This article covers that layer: what it does, where its odd behaviour comes from, how it evolved on the desktop, and why the choice of file system on a server is a performance decision.
What is a file system
Imagine you run a storeroom. People bring you boxes, and you keep them until they come back for them. The storeroom is full of identical shelf slots. Each slot has a number, slot 0, slot 1, and so on up to a few billion, but no label. A disk is a storeroom like this. You can tell it to put some data in slot N, or to give back what is in slot N, and that is all it knows how to do. It has no idea what a file is.
To run the place properly you need to keep some records. You need a ledger of names, so someone can ask for “my cats picture” without knowing which slots they are in. You need a card for each box that says who owns it, how big it is, when it came in and which slots its pieces are in, because a big delivery rarely fits in slots that sit next to each other. You need a list of empty slots, so you never put two boxes in one place. You also need some way to keep all of these records straight if the lights go out while you are halfway through putting a delivery away.
A file system is that set of records. On Unix, a directory is the ledger of names, and each name points to an inode, which is the card for the box. The inode holds the owner, the size, the timestamps and the places on disk where the data lives. Newer file systems record ranges of blocks, called extents, instead of listing every block one by one, and a bitmap or a tree keeps track of the empty space. The file’s name lives in the directory, not in the file. That is why one file can have two names (a hard link), and why renaming a file costs almost nothing.
Walking to the shelves for every box would be slow, so a sensible clerk keeps a desk by the door. New boxes go on the desk, the clerk says “got it”, and puts them away later in batches. The page cache in your operating system is that desk. When you call write(), the kernel copies your bytes into memory and returns straight away, and it writes them to disk later, often a few seconds later. Reads of recent data come off the desk too, so on a running machine you always see what you just wrote, even if none of it is on the disk yet. So my MongoDB problem was probably in the driver, not the disk. The data I could not read had most likely never left my application.
When you need the box on the shelf, you call fsync. It tells the clerk to put away everything for this file now and not to answer until it is done. Going to the shelves takes time: milliseconds on a spinning disk, and tens to hundreds of microseconds on a good SSD. A database that calls fsync after every write pays that price every time, so most databases save up a batch of writes and flush them together. MongoDB flushed once a minute, which made writes very cheap but meant a crash could lose up to a minute of them.
Then there is the power cut. Say the clerk has written a new box into the records but has not put it on the shelf yet, or has filled a slot but has not crossed it off the list of empty slots. Either way, the records no longer match the shelves. Older file systems fixed this by checking every record when the machine started, with a tool called fsck, and on big disks that could take hours.
Journaling file systems give the clerk a notebook. Before doing anything, the clerk writes down what they are about to do, for example “putting these pieces in these slots and updating these cards”. After a crash, the clerk reads the notebook and either finishes each half-done job or throws it away. Most journals only note changes to the records, not the contents of the boxes. That keeps the records correct, but it does not promise that every box is complete.
Copy-on-write file systems handle existing boxes in a different way. Say someone asks the clerk for a copy of a box that is already on the shelf. A normal clerk rebuilds the whole box item by item, and for a box that fills many slots that takes a long time. If the lights go out halfway, the original is fine, but now there is a half-made copy that nobody can trust. A copy-on-write clerk does not copy anything. They write a second card that says two people share the same box in the same slots, so the copy is done almost at once and takes no extra space.
When one owner changes their copy, the clerk leaves the shared slots alone, puts the changed piece in an empty slot, and then updates that owner’s card to point at it. Every other piece still comes from the old slots, and the other owner’s card does not change at all.
Doing things in this order also protects you from power cuts. A clerk who edits a box where it sits can be stopped halfway, which leaves a box that is half old and half new. A copy-on-write clerk never edits a slot that is in use. Until the card changes it points at the old piece, and after the change it points at the new one, so a power cut at any moment leaves you with one whole version or the other. ZFS, Btrfs and Apple’s APFS all work this way.
Snapshots use the same trick on the whole storeroom. Taking a snapshot makes the clerk a second owner of every box at once, which costs little more than copying the cards. After that, every change needs a new slot, because the old one is kept for the snapshot, and the clerk can only free a slot once no card points at it. A box that changes often also ends up with its pieces spread all over the storeroom. That causes trouble for databases, and we will come back to this in the performance section.
Some quirks that come from this design
A lot of the odd things people run into with file systems come straight from those records. Each record has its own rules, and sometimes the rules give results that look wrong from the outside. Most engineers hit the examples below sooner or later.
A deleted file can keep taking up space. Deleting a file removes its name from the ledger, but the card and the slots stay in use until the last person holding the box gives it back. In file system terms, the inode and its data stay around until the last process with the file open closes it. So df can say the disk is full while du finds nothing big. The usual cause is a log file that someone deleted while the service still had it open, and restarting the service gives the space back.
You choose the size of a slot when you format the disk. Every slot is the same size, and a box always takes whole slots. A box smaller than a slot still takes the whole slot, and the leftover space is wasted because no other box can share it. A box bigger than a slot gets split across several slots, and its card has to list where each piece is. Small slots waste little space, but big boxes take more work to track and to collect. Big slots make big boxes cheap, but waste space on every small one.
On ext4 the default slot, called a block, is 4 KB, the same size as a memory page on most Linux machines. So a 100-byte file still uses 4 KB of disk. You can see the same thing on Windows, where the Properties dialog shows “Size” and “Size on disk” as two different numbers.
So the slot size should suit what you plan to store. A mail server or a cache with millions of small files wastes less space with small blocks. A disk full of videos or large database files spends less effort on bookkeeping with big blocks. Most file systems fix the block size for good, so changing it means formatting again and copying everything back. ZFS is different: its version of the setting, recordsize, is set per dataset and can be changed later, although the new size only applies to data written after the change. This setting comes back in the performance section, where getting it wrong with Postgres gets expensive.
Slot size is not the only thing fixed when you format. ext4 also decides how many cards the storeroom can hold, and that number never grows. A workload with millions of small files, like a cache directory, a mail spool or a few hundred node_modules folders, can use up every inode while gigabytes of slots sit empty. The error says “No space left on device” either way, so the first thing to check is df -i, which counts inodes instead of blocks. XFS creates inodes as it needs them, so it rarely runs into this.
Renaming a file happens in one step, but only within one file system. A rename changes one line in the ledger, and the clerk does that in a single move. Lots of programs depend on this. They write the new version under a temporary name and then rename it over the old one, so anyone reading sees either the whole old file or the whole new one. Two file systems have two separate ledgers, so moving a file from one to the other means copying it and then deleting the original. mv does that copy for you, and the one-step guarantee is gone.
Some rules are leftovers from long ago. On Windows you still cannot make a file called CON, NUL or AUX, and Microsoft’s naming rules say to avoid con.txt as well, because Windows treats it as the same device name. MS-DOS kept those names for devices in the early 1980s, so a program that opened CON was talking to the console. Windows has kept the rule for forty years so that old programs still work.
Whether README.md and readme.md are the same file depends on the platform. macOS formats disks as case-insensitive by default. It remembers the name exactly as you typed it, but treats readme.md as the same file. A Git repository with both files works on Linux and breaks on a Mac, where one file replaces the other when you check it out. iOS uses the same APFS file system but is case-sensitive, so this is a default setting and not something built into APFS.
How the desktop got here
Microsoft’s first file system, the File Allocation Table (FAT), dates from 1977 and became the file system of MS-DOS in the early 1980s. It is about the simplest version of the storeroom records you can build. The table has one entry per slot, and each entry says which slot holds the next piece of the same file, so to read a file you follow the chain from one entry to the next. As files got deleted and rewritten over the years, those chains wandered all over the disk, and reading a file meant jumping between slots far apart. People on Windows 95 and 98 ran the defragmenter to fix this. It moved each file’s pieces back next to each other so the disk could read them in one go.
FAT’s limits are behind several things people still run into. Original FAT names allowed eight characters, a dot and three more. When Windows 95 added long file names, it kept a short 8.3 name next to each long one for older programs, which is where paths like C:\PROGRA~1 come from. Drive letters start at C: because A: and B: were the floppy drives.
FAT32 cannot store a file of 4 GB or more, because each file’s entry stores its size as a 32-bit number, and the biggest 32-bit number is one byte short of 4 GiB. The “32” in FAT32 is about something else, the size of the slot numbers in the table, so the name and the limit only look related. Because of this limit, a big video sometimes will not copy to a USB stick.
NTFS came with Windows NT in 1993 and added a journal, file permissions and something called alternate data streams. An alternate data stream lets a file carry extra hidden content that most tools never show you. Windows uses one, the Mark of the Web, to mark files you downloaded from the internet, which is how it knows to warn you before you run them, and malware has used the same feature to hide things. Later, Microsoft built ReFS for servers, which adds checksums to its records and, if you turn it on, to file data as well. Windows still installs on NTFS, but ReFS has started to show up on developer machines through Dev Drive, a Windows 11 feature for source code and build output.
Apple’s first Mac file system, MFS (1984), had no real folders. Every file sat in one flat list, and the folders you saw in the Finder were something the Finder faked. HFS (1985) added real folders, and HFS+ (1998) ran the Mac for nearly twenty years, picking up workarounds along the way. Time Machine, for example, needed hard links to folders, which most Unix file systems do not allow because they can create loops in the folder tree.
In 2017 Apple replaced HFS+ with APFS, a copy-on-write file system built for flash storage, with snapshots and instant file copies. It shipped on the iPhone first, in iOS 10.3, and the update converted each phone’s file system in place. Craig Federighi later said that Apple had run practice conversions during the iOS 10.1 and 10.2 updates, checked the results and then rolled them back. Apple moved hundreds of millions of devices to a new file system without asking anyone to back up first, and very few people noticed.
Unix went its own way. BSD’s Fast File System (1984) set up the pattern of inodes, directories and groups of blocks that most Unix file systems still use. Linux started with a simple file system borrowed from Minix, then moved through ext, ext2 and ext3, which added a journal in 2001. ext4 came in 2008 and is still the default on most Linux distributions.
File systems on servers
On a Linux server you will usually pick between ext4, XFS, ZFS and Btrfs. All four will keep your files safe, but each was built for a different kind of work, and some need more looking after than others.
ext4 is the safe default. People know it well, it does fine on most workloads, and every recovery and monitoring tool supports it. Its weak spots are the ones from earlier: you fix the number of inodes and the block size when you format. On most servers that never comes up, which is why ext4 is still the default on Debian, Ubuntu and most cloud images.
XFS came from Silicon Graphics in the early 1990s, for workstations that handled large video and science files. It splits the disk into allocation groups, and each group looks after its own slots and records. In storeroom terms there are several clerks, each with their own shelves and their own records, so lots of deliveries can happen at once without everyone queuing at one desk. That suits big files and many programs writing at the same time. Red Hat made XFS the default in RHEL 7 in 2014.
ZFS came from Sun Microsystems in 2005. It manages the physical disks as well as the files on them, a job that Linux normally splits between two separate tools. It keeps a checksum for every block and checks it on every read. The checksum is stored in the record that points to the block, not next to the data, so a disk that hands back the wrong block gets caught.
If you set ZFS up with redundancy, it can find and fix corruption that the disk never reported. CERN measured this kind of silent corruption in its own storage in 2007 and found it happened often enough to matter at their scale. ZFS’s licence does not fit with the Linux kernel’s licence, so it ships separately, which makes it more work to run on some distributions.
Btrfs brings copy-on-write, snapshots and checksums into the Linux kernel itself. Its RAID 5 and RAID 6 modes have had serious bugs for years and are still not recommended for data you care about, but the rest of the file system is widely used in production. SUSE has used Btrfs for its system partition since 2014, partly so that a broken system update can roll back to a snapshot. Fedora Workstation switched to it in 2020.
In the cloud you often do not choose at all. Managed Kubernetes usually formats volumes as ext4 unless the storage class says otherwise (GKE’s default storage class is one example), and most teams never touch that. That is fine for most workloads. It just means the choice of a file system is decided by which storage class you use and who wrote it.
Why the choice affects performance
The way the file system writes has to suit the way the application writes. Problems usually show up when the application already manages its own storage, because then two layers make the same decisions in different ways.
A database changes small pages in place. A copy-on-write file system writes every change to a new slot. So every time the database updates a page, the page moves, and over time a database file on Btrfs ends up in thousands of pieces spread across the disk. This is the scattering from the copy-on-write section. Reads that should be long and straight turn into lots of small jumps. The usual fix is to turn off copy-on-write for database files and virtual machine images with chattr +C. That also turns off Btrfs checksums and compression for those files, so the database’s own checksums are the only protection left.
On ZFS, slot size and copy-on-write can work against each other. The default record size is 128 KB, and Postgres writes 8 KB pages. When Postgres changes one page, ZFS has to write a new copy of the whole record, so it may read 128 KB, change 8 KB of it, and write 128 KB somewhere new. Postgres asked to write 8 KB, and the disk got 128 KB. This is called write amplification, and in this worst case it is 16 to 1.
On a spinning disk, write amplification mostly costs you speed. On SSDs and NVMe drives it also costs you the drive, because flash cells can only be written a limited number of times. Drive makers rate how much data a drive can take over its life, so a workload that writes up to 16 times more than it needs uses that up to 16 times faster. The drive then adds amplification of its own, because its controller shuffles data around inside to free up whole blocks for reuse, and each layer multiplies what the layer above it sends. ZFS reduces this by grouping changes together, so several changed pages in the same record can share one rewrite, but a database with random updates across a big file still gets close to the full 16 to 1.
People who run Postgres on ZFS lower recordsize for the database, usually to somewhere between 8 KB and 32 KB, and often turn on compression too. Some also turn off Postgres’s full_page_writes, a setting that protects against pages a crash left half written. Copy-on-write never leaves a half-written page, so on ZFS that setting repeats work the file system already does.
Kafka goes the other way and works with the file system. A Kafka broker adds messages to the end of its log files, one after another, and it relies on the operating system’s page cache instead of keeping its own cache in the JVM. Consumers that keep up read recent messages straight from memory, and the disk mostly sees long, straight writes, which every file system handles well. So the choice of file system affects how fast a broker runs. Kafka’s documentation covers both ext4 and XFS and has generally favoured XFS.
Kafka also ties back to the once-a-minute flush in my MongoDB story. By default, a Kafka broker does not call fsync after each write. It leaves flushing to the operating system and counts on replication instead: a write counts as safe by default once enough brokers have it in memory, on the bet that they will not all lose power at the same moment. MongoDB made the same trade-off, but Kafka tells you about it up front and gives you the settings to change it.
Mount options matter more than most teams think. Linux keeps track of when each file was last read, and older kernels updated that time on every read, which turned every read into a small write. Since 2009 the default has been relatime, which only updates it now and then, and the noatime option stops the updates completely. On ext4, the journal mode decides whether a file’s data has to reach the disk before the changes to its records are saved. The default, data=ordered, is the safer one, and data=writeback is faster.
Some systems decide the file system is more trouble than it is worth. Ceph, a distributed storage system, used to store its data as files on XFS. Ceph had to keep its own data consistent on top of what the file system was already doing, which meant writing a lot of data twice, once to its own journal and once to the file system. In 2017 Ceph switched its default to BlueStore, which writes straight to the raw disk and keeps its records in a small built-in database. The move took the Ceph team years, and much of that time went into rebuilding things a file system normally does. The Ceph developers later wrote up the whole story in a SOSP 2019 paper about why file systems made poor storage backends for them.
One of the most interesting pieces of technology I worked with in the past is Aerospike, a key-value database built for flash storage. Ceph moved off the file system after years of fighting it, but Aerospike was designed from the start to work without one. You can point it at files, but the recommended setup gives it whole SSDs or NVMe drives as raw devices, and Aerospike does the file system’s job itself.
In storeroom terms, Aerospike keeps the whole ledger on the desk. The primary index, which says where every record lives, sits in RAM and takes 64 bytes per record. The Enterprise Edition can also keep it in persistent memory or on flash, which saves RAM at the cost of the extra lookup. A read finds the record’s location in memory and then makes a single trip to the drive. The clerk never has to walk to the shelves to look something up before fetching the box.
Writes follow the copy-on-write clerk, but in bulk. Aerospike never changes a record where it sits. It collects new and updated records in a buffer in memory, and when the buffer is full it writes the whole buffer to the drive as one large block. The old copy of an updated record stays where it was, as dead space. The drive only ever sees large writes, one after another, which keeps its internal write amplification low. ZFS under Postgres, earlier in this section, has the opposite problem.
Aerospike also decides for itself when a write is done, and it makes much the same choice as Kafka. By default, the server acknowledges a write once it is in the memory buffer on the main node and on its replicas, in the same way that a Kafka broker treats a write as safe once enough replicas hold it in memory. Neither system waits for the disk before it says the write is done. Aerospike’s buffer reaches the drive when it fills up, or after a short timer (one second by default) when traffic is light, and both systems rely on the replicas not all failing at the same moment. If that is not enough, the strong consistency mode has a setting called commit-to-device that makes the server flush to the drive before it says the write is done, and every write then takes longer.
Where the contract broke
Programs and file systems work under an unwritten agreement about what is safe. Twice in the last twenty years it broke badly enough that people changed how they write software.
In 2009, people using ext4 started finding empty settings files after a crash. Lots of desktop programs saved settings by writing a new file and renaming it over the old one, without calling fsync. On ext3 this happened to be safe, because by default ext3 wrote the file’s data to disk before it saved the rename. ext4 waited longer before writing data, to speed things up, so after a crash the rename could survive while the new data did not, and the name pointed at an empty file.
The kernel developers said programs had always needed to call fsync and had been getting lucky. The application developers said everyone wrote files this way and the file system should handle it. ext4 added a workaround that spots this pattern and handles it safely. Nobody had written the rule down, and everyone had relied on how ext3 happened to behave.
In 2018, Postgres developers found out that one of their basic assumptions about saving data was wrong. When fsync failed on Linux, for example because of a disk error, the kernel could treat the unsaved data as written anyway and report the error only once. Postgres would try fsync again, the second try would succeed, and Postgres would think the data was safe when it had never reached the disk. Postgres had worked this way for about twenty years. The fix was to treat a failed fsync as fatal: stop the server and recover from the write-ahead log, which is the one copy Postgres can trust. Other databases checked their own error handling after this, and the episode became known as “fsyncgate”.
MongoDB has changed its defaults since my tests failed. Writes now go to a journal, and since MongoDB 5.0 the default write concern waits until a majority of the replica set has each write. The database I walked away from now does what I thought it did back then.
It has taken me a few years to understand the nuances of file systems and the role they play. It took managers who dared me to dig deep, understand problems properly and challenge my assumptions. Now, when I look at a datastore, I try to understand its philosophy on durability. I look for four key moments: when the server says the write is done, when the write reaches the other replicas, when it reaches the page cache, and when it reaches the disk. Each of those moments survives a different failure mode. A write that exists only in the server’s memory is lost if the process crashes. A write in the page cache survives a process crash but not a power cut. A write that has reached the disk survives a power cut on that machine, and a write that sits on several replicas survives the loss of a whole machine. Most systems let you choose the level per write: MongoDB calls the setting the write concern, and Kafka producers call it acks. When the server says a write is done, the question is which of these it waited for.
Understanding the workload you are deploying and its access pattern helps you evaluate file systems and make an informed decision. But for most of us, ext4 will be good enough.