Introduction

As I'm doing some work to create an RDBM like library to write data to a database, I wanted to make sure that my file was NOT sparse. I don't mind sparse files at all, they save us some space on disk. Only, when creating a database, pretty much all my blocks of data will need to exist, otherwise we may take the risk of not being able to allocate the inodes later and write in those blocks.

So I search Google to try to find information on NOT to create a sparse file. Make sure that my software prevents the OS from creating a sparse file. I could not really find anything that would clearly say how that works. Plus many of the results were about Linux tool you can use to create sparse file (nothing about preventing the sparse files!)

How to Create a Sparse File in C

The fact is that goes the other way around. You can on purpose create a sparse file, but you can't (do not have to) prevent the OS from creating a sparse file.

There are three C functions I know of that will allow you to create a sparse file. These may be used by higher level functions too, so be careful about those.

truncate() and ftruncate()

The truncate() functions are used to create and change the size of files. When making a file larger, the space added will automatically be a "Hole" in the file.

Internally, the file system saves the offsets where data is found. Anything else is assumed to be zeroes.

It is more than likely that this will only work if you have an exact page with all zeroes. Say your ext4 system uses 4Kb pages, an extend which is viewed as a "Hole" would at least be 4Kb and it will start/end at a 4Kb boundary (0x1000).

The following code:

fd = open(filename, O_CREAT | O_WRONLY, 0600);
r = ftruncate(fd, 1024 * 1024);
close(fd):
return r;

creates a 1Mb file which is sparse (uses only one idnode for its metadata and nothing for the contents).

lseek()

The lseek() function let you seek to any position between 0 and +inf. (well, may be not infinity but the maximum size allowed for a file on your file system).

If the position is after the existing end of the file and you start writing and the gap between the previous end of the file and the data written includes 4Kb pages of zeroes, then the file becomes sparse.

Note that you do not automatically grow your file if you just do an lseek(). The following code shows how you create a sparse file, although this one won't be empty:

fd = open(filename, O_CREAT | O_WRONLY, 0600);
lseek(fd, 1024 * 1024);
write(fd, "EOF", 3);
close(fd):
return r;

This creates a file of 1Mb + 3 bytes set to the characters "EOF".

The 1Mb of space is going to be sparse (a large hole).

What are Sparse File Good For?

If you ask me, I would say pretty much everything except files you are currently working on because those are likely to generate problems if you can't allocate space in the holes.

That includes any type of archive or binary file which is installed once and read many times. Files never expected to be overwritten/modified (except on an OS update) should be using the sparse feature. Of course, most binary files do not end up with a lot of zeroes in them. For example, the executables that start with large buffers set to all zeroes do not store those zeroes in the executable files. Instead they have a special section called BSS which when loaded extends to a large buffer with all zeroes.

However, from experience, such sparse files are rather rare. 99.9% of the files in your system just can't be sparse.

How to Avoid/Prevent Sparse Files?

So that translates in the ability to avoid sparse files by avoiding the use of the truncate() and lseek() to auto-grow the file. When you want to grow your file, you have to go to the end of the file and write a buffer of data there. The buffer can of course be all zeroes.

Note that the OS does not check whether you are writing all zeroes to a 4Kb page in your file. If you do an explicit write() it is assumed that you want that data in that file at that location.

So in C you could write a simple loop like this to create a new file and make sure it isn't a sparse file:

char buf[4096];
memset(buf, 0, sizeof(buf));
fd = open("filename", O_CREAT | O_WRONLY, 0600);
for(i = 0; i < 256; ++i)
{
    write(fd, buf, sizeof(buf));
}
close(fd);

This function writes 256 × 4096 buffers of zeroes, in other words, it creates a file of 1Mb in size and not sparse.

Reasons for Not Using Sparse Files

Here are several reasons why using a sparse file can be a bad idea:

Database File

For a database to be reliable, you want it to be able to write to its file. It is rather unlikely that your database files will be sparse anyway, however, it often creates files using blocks and when that happens new blocks could be sparsely created (with a truncate() call). To increase your reliability you certainly want database files which are updatable to be fully allocated on disk.

Swap File

I've seen quite a few person saying they wanted a sparse file for their swap file. I'm not too sure what their thinking is. I suppose they have no clue that this means the swap system can run out of swap space early because the drive on which the swap file is got full...

Also, once some space was used before, it becomes "unsparse" space and the OS won't change that automatically. As far as it's concerned, it's not going to put zeroes back all the time and then mark those sections sparse again. So over time you are likely to have a rally large amount ofdata used anyway.

VirtualBox Disk File

Whenever I create a virtualbox instance, I allocate the disk in full. I have a specific partition used just and only for those disks.

When I first started using those files, I was on HDD and it was dreadfully slow to use the sparse capability. Since then I always ask for the disks to be precreated in full.

Just like with the swap file, if you create a virtualbox instance for production, you don't want your non-full drive to fail because the main drive is now full. It makes for really strange errors!

Also, for test purposes, you may want to get a good feel for the speed things take and with a sparse file, it will add many extra I/O accesses which means the speed is going to be off.

So in most cases, having a full physically allocated VPS disk is a good idea.

Is a File Sparse?

If you are interested, you can check a file to see whether it is indeed sparse or not.

Command Line Tools

Here are a couple of command line tools that can help:

filefrag -v <filename>

The filefrag command line lists the extends of the file. If the file has 1 single extend and the size of that extend is the same as the file size, then the file is not sparse

There is one exception: an empty file will not have any extend (obviously?!)

A file with 2 or more extends definitely has holes.

du <filename>

The du command returns the size that the specified file uses on disk. In other words, if the file is sparse, du returns a size which is less than what it looks like when you do:

ls -l <filename>

Finally, if you want to know where the file is on your disk, use hdparam:

sudo hdparm --fibmap <filename>

The one drawback for this tool is you have to use sudo. But it tells you about your LBA position on disk and shows you the number of sectors used. The interesting part here is that the size won't be 0 like du tells you. This is because the OS needs a set of sectors to hold the metadata of the file. It is likely around 32 sectors of 512 bytes (or 4 blocks of 4Kb).

How do I do it in C?

If you read the lseek() manual page in full, you already know the answer to that. libc offers two new whence option which are:

SEEK_DATA, find the first chunk of data starting at the specified offset

SEEK_HOLE, find the first hold starting at the specified offset.

This allows you to generate a map of data and holes. If you find at least one hole then the file is considered sparse.