Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.)
O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly.
How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.
"fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file.
aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.
The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.
> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write.
Note that Linux ext4 does not do this. On a power outage, you can get bogus trailing zeros on a file which you were appending, because the file size was updated before the data was written. I asked Ted T'so about this and he said it was working as intended.
I don't know about ZFS, but the usual write(2) API does not support this: It might for example return early with a short write because some interrupt occurred. Can happen for all "slow devices" (see signal(7)). And I think that's a good thing and am sure many programs expect this.
The signal(7) man page also states clearly that a local disk is not a "slow" device, so this seems moot.
read(2), readv(2), write(2), writev(2), and ioctl(2) calls on "slow" devices. A "slow" device is one where the I/O call may block for an
indefinite time, for example, a terminal, pipe, or socket. If an I/O call on a slow device has already transferred some data by the time it
is interrupted by a signal handler, then the call will return a success status (normally, the number of bytes transferred). Note that a
(local) disk is not a slow device according to this definition; I/O operations on disk devices are not interrupted by signals.
Oh! So that's where all the random nulls in my shell history came from. Zsh would sometimes complain about a corrupted history after a bad crash, and I always wondered what caused maybe 20 nulls to be added to the end of the file. Thanks!
> On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position.
I would never expect that. It's not how it works. But what has to work is the dup() case, or the inherited descriptors case like
$ ( do_this & do_that & } > /some/file
because here they share the same file descriptor (as opposed to having only the same file opened on stdout). Note this still doesn't guarantee that writes won't be interleaved. I think there is only a guarantee that writes up to 512 bytes won't be interleaved. The answer here is: Don't do simultaneous writes. Or implement your own synchronization mechanism. It's just not a requirement with a universally satisfying solution that should be implemented in the kernel.
> NTFS doesn't do append correctly.
If you meant NFS: It deliberately violates POSIX file system semantics. I guess it's just too hard to implement. For example, an NFS server can't know whether the machine which owned a lock crashed or is just temporarily disconnected from the network (and has still a process running thinking it owns a lock and not knowing the server is wondering).
In any case, not the API to blame.
> How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.
How could we possibly implement these "transactions" if the write is _not_ at the end of some file?
> It's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.
What the disk does is in the end controlled only by the disk itself (and of course environmental forces). So it's always best effort.
sync_file_range(2) allows you to specify ranges to sync, instead of the whole file, but it's not that useful anyway.
In most DBs, you have separate data and log. The log is always only appended to, and the nature of disks is such that grouping concurrent transactions together into a single commit is a big win regardless of how syncing works. So syncing all outstanding I/O to the log file is generally what you want to do anyway.
While the data store is always being written to, its writes do not need to be synced to disk until a log checkpoint is made (and the old log subsequently discarded). And since that is an intermittent asynchronous process, there is not much to gain from finer-grained syncing here either.
Actually, we made postgres use sync_file_range() for checkpointing if available in 9.6. Not for durability - there's a few to many caveats in the manpage - but to control how much work a later fsync() has to do. In many workloads checkpointing can generate a lot of writes, and the OS's writeback caching of those can generate a lot of dirty buffers in the kernel's page cache. If the kernel decides to flush those (on it's own or due to an fsync), latency for every other FS operation can skyrocket. We've seen stalls in the 10s of minutes. So we now regularly sync_file_range(SYNC_FILE_RANGE_WRITE), unless the feature is disabled of course, to control how much dirty data the kernel has.
Care to elaborate on that? I haven't seen issues with the two mechanisms I know of in NT:
1. CreateFile with FILE_APPEND_DATA
2. WriteFile with the OVERLAPPED offset set to ~0
Both of these get you writes that are as far as I know safe to have multiple processes writing to a common log file. Not sure about flushes to disk but you could probably get away with FlushFileBuffers for this.
NTFS handles append just fine. As another comment speculated, the parent may have meant NFS, which does notoriously have lots of problems with append, and many other POSIX semantics.
> Rename isn't quite an atomic replacement. If you crash before the rename, the new file hangs around. (Hence unwanted .part files.)
It seems O_TMPFILE cannot be used to atomically replace existing files either. If only one could use renameat2() on a raw fd or force linkat() to an atomic overwrite.
O_APPEND isn't airtight on all systems. On some older UNIX systems, multiple writers created with "open()" (not "dup()") do not share a file position. NTFS doesn't do append correctly.
How do you guarantee that, after a crash, the end of the file is at the end of some write? By updating the file size after the write. The file size update can be deferred during heavy write traffic, but you should always get a file size that ends at a write boundary.
"fsync" synchs the whole file, not just one I/O, which can take a while. Databases such as MySQL's InnoDB, which puts multiple tables in one file, can have independent I/O going on in different parts of a file.
aio(7) has the right mechanism, a callback/signal on completion. But it's not clear if the file system guarantees the data is safely on disk when the completion signal comes in.
The original article complains that UNIX/Linux file system semantics aren't well enough defined for database safety. He's right. They're close, but not quite there, because behavior after a crash is unspecified.