Home Fileserver: ZFS snapshots

The next thing you will want to master when running your own ZFS home fileserver is how to perform snapshots of your file systems, as this is a crucial part of helping to protect your data.

Understanding snapshots is the first step to understand before performing full and incremental backups, as backups will utilise zfs send/recv and these use snapshots to work out what to send.

What are snapshots?

• Snapshots are a way of taking a picture of the file system at a point in time.
• Snapshots are read only.
• A snapshot allows you to know the state of the file system when the snapshot was made.

Snapshots allow you to do special things like make a full backup of all the files/directories referenced by the snapshot.

Or you could receive the snapshot to form a new file system with a different name and, optionally, with different file system properties.

You can also use snapshots to form the basis of incremental backups, once a full backup has been made.

You can even create a clone of a file system from a snapshot. A clone is a writeable file system based on the contents of the snapshot.

So you see that snapshots form the basis of many useful operations with file systems, and understanding them and how to use them is crucial to successful ZFS file system administration.

We will perform the ZFS commands as root user, but something called ZFS Delegated Administration exists, which allows permissions to be granted to users to enable them to perform pool and file system operations, but this is beyond the scope of this article.

Create a test file system

First, let’s create a test file system ‘tank/test’ to experiment with:

# zfs create tank/test
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
tank                 766G   598G  25.3K  /tank
tank/test           24.0K   598G  24.0K  /tank/test

And populate it with 2 files:

# cd /tank/test
# echo "hello world: a" > a
# echo "hello world: b" > b

Now let’s create a snapshot of file system ‘tank/test’ and call the snapshot ‘1’:

# zfs snapshot tank/test@1

Creating a snapshot is virtually instantaneous, regardless of the size of the file system it references.

In reality, it would probably make sense to use a timestamp for the snapshot name, as this would allow easier identification of when the snapshot was taken.

Now let’s take a look:

# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
tank                 766G   598G  28.0K  /tank
tank/test           26.6K   598G  26.6K  /tank/test
tank/test@1             0      -  26.6K  -

We can see that in the ‘tank/test’ file system, the files take 26.6K of disk space.

Note that the snapshot of it called ‘tank/test@1’ takes no space — see the ‘USED’ column. This is because the snapshot simply references the files in the ‘tank/test’ file system. This will change later when we start to modify and delete files.

You can also see that the snapshot references 26.6K of files — see the ‘REFER’ column.

You can also see that the ‘MOUNTPOINT’ is undefined, because snapshots cannot be mounted.

So to recap, at the point in time that snapshot ‘tank/test@1’ was created:

1. The file system ‘tank/test’ contains two files: ‘a’ and ‘b’. We’ll call these the original files ‘a’ and ‘b’ later.
2. The snapshot ‘tank/test@1’ references files ‘a’ and ‘b’ and consumes no disk space.

Delete a file

Now we’ll delete file ‘a’ in ‘tank/test’ and create a new snapshot.

# rm a
# zfs snapshot tank/test@2
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
tank                 766G   598G  28.0K  /tank
tank/test           49.3K   598G  26.0K  /tank/test
tank/test@1         23.3K      -  26.6K  -
tank/test@2             0      -  26.0K  -

Now you can see that the first snapshot we made (tank/test@1) has now started to consume disk space (23.3K).

That is because snapshot ‘tank/test@1’ referenced file ‘a’ which has now been deleted from the file system.

But a snapshot will always continue to reference the same files forever, or at least until the snapshot is destroyed.

The only way that a snapshot can continue to reference files that have been deleted from the file system the snapshot was created from, is to take ownership of the modified/deleted files. Thus, a snapshot initially consumes no disk space, but it starts to consume disk space as the files it references get modified or deleted.

So, effectively, when file ‘a’ was deleted from the file system, the blocks it used became owned by the snapshot ‘tank/test@1’, so that the snapshot can continue to reference the file forever.

So to recap, at the point in time that snapshot ‘tank/test@2’ was created:

1. The file system ‘tank/test’ contains one file: ‘b’.
2. The snapshot ‘tank/test@1’ references files ‘a’ and ‘b’ and consumes some disk space because it has taken ownership of file ‘a’ that was deleted.
3. The snapshot ‘tank/test@2’ references only file ‘b’ and consumes no disk space.

Copy-on-write

Just to clarify, ZFS uses a system called ‘copy-on-write’, which means that if a file is modified, ZFS will create a new file containing the modifications, then move the pointer to the new file and finally delete the old file, all in one transaction. But if the file deleted is referenced by one or more snapshots, then the data blocks occupied by the original file will become owned by the snapshots that reference the original file. In this way, when a file is modified or deleted, snapshots referencing the original file will consume disk space as they assume ownership of the original file’s blocks. But the file system in which the file existed will no longer reference the blocks used by the original file, but will reference the new blocks used by the new modified file.

Once you understand this copy-on-write mechanism and the way snapshots continue to reference the blocks used by the original files, you will have grasped the key to the magic of snapshots.

Modify a file

Now we’ll modify file ‘b’ in ‘tank/test’ and create a new snapshot.

# echo modified >> b
# cat b
hello world: b
modified
# zfs snapshot tank/test@3
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
tank                 766G   598G  28.0K  /tank
tank/test           71.3K   598G  26.0K  /tank/test
tank/test@1         23.3K      -  26.6K  -
tank/test@2         21.3K      -  26.0K  -
tank/test@3             0      -  26.0K  -

Now you can see that the second snapshot we made (tank/test@2) has now started to consume disk space because it has taken ownership of the original file ‘b’.

However, the file system ‘tank/test’ contains the modified version of file ‘b’, as you would expect.

So to recap, at the point in time that snapshot ‘tank/test@3’ was created:

1. The file system ‘tank/test’ contains one file: ‘b’, which is the new modified version of the original file.
2. The snapshot ‘tank/test@1’ references files ‘a’ and ‘b’ and consumes some disk space because it has taken ownership of file ‘a’ that was deleted, and the original file ‘b’ which has now been modified.
3. The snapshot ‘tank/test@2’ references only the original file ‘b’ and consumes some disk space because it has taken ownership of the original file ‘b’, because it was modified.
4. The snapshot ‘tank/test@3’ references only the new modified file ‘b’ and consumes no disk space.

Note that snapshots ‘tank/test@1’ and ‘tank/test@2’ both have taken ownership of the original file ‘b’, and so will reference the same blocks, so there will be no duplication and wasted space.

Create a new file

Now we’ll create a new file ‘c’ in ‘tank/test’ and create a new snapshot.

# echo "hello world: c" > c
# zfs snapshot tank/test@4
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
tank                 766G   598G  28.0K  /tank
tank/test           94.6K   598G  26.6K  /tank/test
tank/test@1         23.3K      -  26.6K  -
tank/test@2         21.3K      -  26.0K  -
tank/test@3         21.3K      -  26.0K  -
tank/test@4             0      -  26.6K  -

So to recap, at the point in time that snapshot ‘tank/test@4’ was created:

1. The file system ‘tank/test’ contains two files: ‘b’, which is the new modified version of the original file, and the new file ‘c’.
2. The snapshot ‘tank/test@1’ references files ‘a’ and ‘b’ and consumes some disk space because it has taken ownership of file ‘a’ that was deleted, and file ‘b’ because it has been modified.
3. The snapshot ‘tank/test@2’ references only the original file ‘b’ and consumes some disk space because it has taken ownership of the original file ‘b’, because it was modified.
4. The snapshot ‘tank/test@3’ references only the new modified file ‘b’ and consumes no disk space. Oh, but it seems it does consume some disk space! Why?
5. The snapshot ‘tank/test@4’ references only the new modified file ‘b’ and the newly created file ‘c’, and consumes no disk space.

I was surprised that the snapshot ‘tank/test@3’ has now started to consume disk space after we created a new file ‘c’. I will try and find out why, in order to have a correct understanding of what is happening. Update: Apparently the snapshot probably consumed space because I did an ‘ls’ on the directory, thus causing the ‘atime’ attribute to be modified. See this comment:
http://breden.org.uk/2008/05/12/home-fileserver-backups-from-zfs-snapshots/#comment-1267

Snapshot dynamics

Although the files referenced by a snapshot are static, the amount of space a snapshot consumes is dynamic.

A snapshot will always consume zero space when it is created, but whenever a file referenced by the snapshot is modified, the snapshot will consume additional space, in the amount according to the size of the original file, because it takes ownership of the original file.

The file system releases ownership of the original file when it is modified, and takes ownership of the new file created, due to the copy-on-write operation.

In a similar way, when a file is deleted from the file system, the file system releases ownership of the file, and if any snapshots reference the file being deleted, they will take ownership of the file, and jointly, assuming more than one snapshot references the deleted file (data deduplication?).

See the audit log below where I’ve tried to track the dynamics of what happened to each snapshot as new file system events occurred.

Note that I have used the * character to indicate that this is the modified file, not the original file.

1. create file ‘a’ and ‘b’

snapshot | references | owns | USED  | REFER
---------------------------------------------
      @1 | a  b       | -    |    0K | 26.6K

2. delete file ‘a’

snapshot | references | owns | USED  | REFER
---------------------------------------------
      @1 |    b       | a    | 23.3K | 26.6K
---------------------------------------------
      @2 |    b       | -    |    0K | 26.0K

3. modify file ‘b’

snapshot | references | owns | USED  | REFER
---------------------------------------------
      @1 | -          | a b  | 23.3K | 26.6K
---------------------------------------------
      @2 | -          |   b  | 21.3K | 26.0K
---------------------------------------------
      @3 |    b*      | -    |    0K | 26.0K

4. create file ‘c’

snapshot | references | owns | USED  | REFER
---------------------------------------------
      @1 | -          | a b  | 23.3K | 26.6K
---------------------------------------------
      @2 | -          |   b  | 21.3K | 26.0K
---------------------------------------------
      @3 |    b*      | -    | 21.3K?| 26.0K
---------------------------------------------
      @4 |    b* c    | -    |    0K | 26.6K

I don’t know why snapshot 3 has started to consume disk space in this last case, so I’ll try to find out and update this page with the results.

Checking the snapshots

Note that as each snapshot is created, the amount of space consumed by the file system increases. This is because the snapshot contents are stored within the file system in the ‘.zfs’ directory. Now let’s take a peek into these snapshot directories:

# ls -l /tank/test/.zfs
total 0
dr-xr-xr-x   6 root     root           6 May 12 14:49 snapshot

# ls -l /tank/test/.zfs/snapshot
total 16
drwxr-xr-x   2 root     root           4 May 12 14:50 1
drwxr-xr-x   2 root     root           3 May 12 15:02 2
drwxr-xr-x   2 root     root           3 May 12 15:02 3
drwxr-xr-x   2 root     root           4 May 12 18:58 4

# ls -l /tank/test/.zfs/snapshot/1
total 4
-rw-r--r--   1 root     root          15 May 12 14:50 a
-rw-r--r--   1 root     root          15 May 12 14:50 b

# cat /tank/test/.zfs/snapshot/1/b
hello world: b

# ls -l /tank/test/.zfs/snapshot/3
total 2
-rw-r--r--   1 root     root          24 May 12 17:35 b

# cat /tank/test/.zfs/snapshot/3/b
hello world: b
modified

What next?

By creating the snapshots, we have protected ourself from accidental file deletion, because the snapshot will still reference the deleted file, and will take ownership of any deleted files, assuming the snapshot originally referenced the file when the snapshot was created.

Now that we have created multiple snapshots of the file system after various file system events, we have many options available to us.

We can create a writable clone of the file system from any of the snapshots.

We can use snapshots to create full and incremental backups of the file system. See the next post in this series of ZFS articles for how to perform backups using snapshots.

For more ZFS Home Fileserver articles see here: A Home Fileserver using ZFS. Alternatively, see related articles in the following categories: ZFS, Storage, Fileservers, NAS.