Thursday, June 16, 2005

How Not To Make A Page-Atomic Copy Of A File

Can you find the hidden bug?

import os

def copy(fIn, fOut):
blockSize = os.statvfs(fIn.name).f_bsize
while 1:
bytes = fIn.read(blockSize)
if not bytes:
break
fOut.write(bytes)

5 comments:

  1. fOut could be on a different filesystem. fstavfs would generally seem to make more sense. I don't see how "page-atomic" and "file-system block size" relate, necessariy.

    Actually, I don't know what "Page-Atomic Copy Of A File" means, so I'll stop now.

    ReplyDelete
  2. Consider it safe to assume that fIn and fOut are on the same local filesystem, are both real, regular files and that statvfs().f_bsize gives the actual filesystem blocksize for the filesystem they reside on.

    A "page-atomic" (perhaps there is a better term for this?) file copy is one where no page (a filesystem block worth of bytes) ends up in the resulting file which was not present in the source file.

    This could happen if a separate process is writing pages to the input file while the copy function is running, if the copy function does not perform only atomic reads. For example, if the first 4096 bytes of the file are 'X' and the copy function reads 1 byte at a time, if a process writes 'Y' over those first 4096 bytes, the output file may end up containing a series of 'X' followed by a series of 'Y', even though at no point were 'X' and 'Y' mixed in the source file.

    Pretty much all modern operating systems guarantee that reads and writes of the filesystem page size will be atomic.

    So with all that in mind, there is still a bug in the function in the original post.

    ReplyDelete
  3. Yes, exactly :)

    If the file is appended to between the read() at which EOF is reached and the next read() which would otherwise have returned '', and the file's size was not a multiple of the filesystem block size, the rest of the file will essentially be corrupt.

    Unfortunately for me, it took quite a long time to discover this shortcoming of this otherwise reasonable file copy routine.

    ReplyDelete
  4. Oh, that's actually nastier than I thought.

    I wonder if Python should somehow 'latch' the file object so that it stays at EOF once EOF has been reached (unless a seek or something happens, obviously). Might be hard to get right, though.

    ReplyDelete