Solved ZFS and 'pooled' file permissiveness

daridro

daridro

Hello,
New to FreeBSD and to ZFS I did some test to better understand some concepts.
Hereunder 2 tests, then my questions.

Sorry if this thread is a bit long.

Test 1

I created a file (d1.dat), then created a zpool based on this file.
TLDR: When I move the file (d1.dat), ZFS still refers it.

Code: 
r# dd if=/dev/zero of=/bingo/d1.dat bs=1M  count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.019527 secs (5369910605 bytes/sec)
r# zpool create foo /bingo/d1.dat
r# zpool status
  pool: foo
 state: ONLINE
config:
 
        NAME            STATE     READ WRITE CKSUM
        foo             ONLINE       0     0     0
          /bingo/d1.dat  ONLINE       0     0     0
 
errors: No known data errors
 
  pool: zroot
 state: ONLINE
config:
 
        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0
 
errors: No known data errors
r#
r# touch /foo/alpha
r# touch /foo/beta
r# tree /foo
/foo
├── alpha
└── beta
 
1 directory, 2 files
r# mkdir /bingo/bar
r# mv /bingo/d1.dat /bingo/bar/
r# zpool status
  pool: foo
 state: ONLINE
config:
 
        NAME            STATE     READ WRITE CKSUM
        foo             ONLINE       0     0     0
          /bingo/d1.dat  ONLINE       0     0     0
 
errors: No known data errors
 
  pool: zroot
 state: ONLINE
config:
 
        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
r#
r# touch /foo/gamma
r# tree /foo/
/foo/
├── alpha
├── beta
└── gamma
 
1 directory, 3 files
r# zpool export foo
r# zpool import -d /bingo/bar/ foo
r# zpool status
  pool: foo
 state: ONLINE
config:

        NAME                STATE     READ WRITE CKSUM
        foo                 ONLINE       0     0     0
          /bingo/bar/d1.dat  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
r# tree /foo/


/foo/
├── alpha
├── beta
└── gamma

1 directory, 3 files

Test2

Now I delete the file (dat1.dat).
TLDR: ZFS does not complain at all, moreover, I still can use the dataset created along with the pool.

Code: 
r# tar -cf /bingo/babar.tar /bingo/bar/d1.dat
r# rm /bingo/bar/d1.dat
r# zpool status
  pool: foo
 state: ONLINE
config:

        NAME                STATE     READ WRITE CKSUM
        foo                 ONLINE       0     0     0
          /bingo/bar/d1.dat  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
r# touch /foo/delta
r# tree /foo/
/foo/
├── alpha
├── beta
├── delta
└── gamma

1 directory, 4 files
r# zpool export foo
r# zpool status
 pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
r# tar -xvf /bingo/babar.tar
x bingo/bar/d1.dat
r# zpool import -d /bingo/bar/d1.dat foo
r# zpool status
  pool: foo
 state: ONLINE
config:

        NAME                STATE     READ WRITE CKSUM
        foo                 ONLINE       0     0     0
          /bingo/bar/d1.dat  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          nda0p3    ONLINE       0     0     0

errors: No known data errors
r# tree /foo/
/foo/
├── alpha
├── beta
└── gamma

1 directory, 3 files

PS : I did same tests with mirrored zpool (dat1.dat and dat2.dat), I obtained same results for tests 1 and 2.

Questions
Test 1
Q1: Is ZFS constantly monitoring 'pooled' file location? (but the zpool status after the move still display /bingo/d1.dat ONLINE and not /bingo/bar/d1.dat ONLINE)
Q2: What would be the behavior if it was a real hardware (hot disconnection and re-connection on another slot/port)?
Q3: Is it normal that I get no warning?

Test2
Q4: Is it normal that I get no warning (i.e this file is required by an active imported zpool) when I asked for the file (dat1.dat) deletion?
Q5: Where is stored the created file (delta in the example) after the 'pooled' file (dat1.dat) deletion?
Q6: If the file (delta) is stored in memory, what if the file (delta) size exceeds available memory size ?

Pardon me if I misunderstand some ZFS concepts, then please correct me.

Regards,
Darius
 
Don't bother with Alain's questions ... this is a much more fundamental question than ZFS.

To understand this, you have to understand that in Unix, there are two separate layers, namely files, and the names of the files. Files that are stored on disk can have zero or more names. Yes, you read that right: A file can have zero names. It can have one name. It can have multiple names. Whether the file exists is somewhat independent of whether it has a name or not. To actually delete a file (make it not exist), you have to do two things: (a) remove its last name, so it has zero names. (b) stop using it.

At the beginning you created a file /bingo/d1.dat. That file exists, it has a size of 100 MB, and it can be found under the name /bingo/d1.dat (I'll skip over the details that d1.dat is a dirent or directory entry in the directory /bingo, an unnecessary detail). You then opened that file using an application, in this case the application "zfs". ZFS happens to remember the name you gave it when it opened the file. So far, so good: One file on disk, one name for it, and the file is currently open.

Then you renamed the file. What that really does is: atomically (in a single operation, with no intermediate state ever visible to an observer), it removed the name /bingo/d1.dat, created a new name /bingo/bar/d1.dat. But ZFS still has the file open, and it still remembers that it had opened it under the name /bingo/d1.dat. Nothing in the world has told ZFS that there is a new name for the file. Nor does it matter very much: The file is still on disk, is still 100MB in size, and ZFS can continue to use it.

Then you asked ZFS to open the file again under the new name (when you told it to import in the directory /bingo/bar, causing it to search for any file in that directory that might be a suitable ZFS volume). Now the file is open again, and ZFS remembers that it last opened it under the name /bingo/bar/d1.dat. Fabulous. The only slightly tricky part is: ZFS seems to have noticed that the same zpool that it had historically opened under /bingo/d1.dat is now gone, since the same pool is available under a different name. This makes logical sense.

Note that at this point, ZFS is still holding the file open, since ZFS is still using the file.

Then you issued the command "rm ...". All that accomplished was to remove the name /bingo/bar/d1.dat. It did NOT destroy the file. It can't, because zfs still has it open. Note that there is now a file on disk, which is still being actively used, but that has no name. No amount of ls or find commands will be able to locate it.

At some point, ZFS will stop using the file on disk. I have no idea when that is; at the latest, it happens at the next system shutdown and reboot. As soon as the last application closes the file (stops using it), the file on disk is actually destroyed.

So what happened here? At the file name layer, you created a file name when you first did the dd command to create the file. You then created a new name while atomically removing the old name. You then removed the new name too. So the file went from not existing, to having 1 name, to having 1 different name, to having zero names.

At the file layer, you created the file on disk (size 100MB), you started using it. Sometime while using it you changed the name, but that is totally irrelevant to the applications that have it open. Then you closed it and re-opened it through a different name. And at some unknown point in the future, you will close it again, and that stops using it and will actually destroy it. Why destroy it? It has no name (therefore it will never be located or used again), and nobody is using it.

All clear so far? The fact that the application using the file (holding it open) happens to be "zfs" is not really relevant, it just adds confusing complexity here. You could have written a 2-line program in Python or perl that accomplishes the same thing.

Now, earlier I said that a file can have more than 1 name. This never happened in your example. To accomplish that, you usually use the ln command to create an additional hard link to the file. In reality, a hard link is just a name for a file; the first hard link comes into existence when the file and its first name are usually created. As an example: Instead of renaming the file, you could have kept it at /bingo/d1.dat, created a second link (name) to it at /bingo/bar/d1.dat (now the file has two names!), told ZFS to start using the new name, then delete both names. Same end result.

If you want to play with this further: Use the "ls" command with the "-l" option (to show you more details about the file, including the link count) and the "-i" option (to show you the inode number, which is the secret ID number that applies to the file, not to its name). You will see how one file can have multiple names.
 
ralphbsz Thank you so much for this clever and neat explanation. Now I understand better the inner mechanics.

In the case of a ZFS use I find quite dangerous that if the 'dat' file is unfortunately destroyed the user can still use it without having been warned, the risk is the loss of data if the user is whether not aware of the destruction or forgot that he has destroyed de 'dat' file.
 
daridro said:
In the case of a ZFS use I find quite dangerous that if the 'dat' file is unfortunately destroyed the user can still use it without having been warned,
Click to expand...
Doctor, it hurst when I do that.
Well, then stop doing it.

Serious: This is Unix. It gives you enough rope to shoot yourself in the foot. Any file that is used by ZFS to store data should have permissions such that normal users can't delete it. And if someone becomes root and deletes it anyway: we warned you that becoming root has consequences, didn't we?
 
You must log in or register to reply here.
Back
Top