ZFS zrepl - a dataset changes jut by auto mounting it and subsequent backups fail

X

xsouku04

With one dataset I get always an error, that the dataset can not be transferred, because was modified. " cannot receive incremental stream: destination zroot/d876/zroot/jails/www has been modified"

Apparently by auto mounting the dataset, changes to the dataset occur and subsequent snapshot transfers start to fail.

I can get get it resolved by using rollback. But it is not very convinient to watch for those errors and do rollbacks when needed. Is there any more elegant way how to deal with this?

One solution would be to have mounted read only.


If I want to avoid mounting I have to use "canmount": "off". Looks like canmount "noauto" has no effect.

Why canmount "noauto" has no effect? Would have readonly property have effect?
Is there a way how to avoid it? The mount points should never collide as each jail is home to one server or other, not both. And the backup is done to the other server.

My /usr/local/etc/zrepl/zrepl.yml is
YAML: 
 - type: sink
    name: sink
    root_fs: "zroot"
    serve:
       type: tcp
       listen: "10.1.5.50:8888"
       clients: {
          "10.1.8.76": "d876"
       }
    recv:
      placeholder:
        encryption: inherit
      properties:
        override: {
    #                  "canmount": "noauto"
                      "canmount": "off"
                        }



YAML: 
 - type: push
    name: d876_to_d550
    connect:
      type: tcp
      address: "10.1.5.50:8888"
#    "zroot/jails/home<": true,
    filesystems: {
     "zroot/jails<": true,
     "zroot/jails": false,
     "zroot/jails/dbZFSino": false,
     "zroot/jails/dbZFSino2": false,
     "zroot/jails/homer": false,
     "zroot/jails/dbZFS":false
   }
    send:
      bandwidth_limit:
         max: 23.5 MiB
      send_properties: true
    snapshotting:
      type: periodic
      prefix: zrepl_
      interval: 10m
    pruning:
      keep_sender:
      - type: not_replicated
      - type: last_n
        count: 10
      keep_receiver:
      - type: grid
        grid: 1x1h(keep=all) | 24x1h | 30x1d | 6x30d
        regex: "^zrepl_"
 
I have nine iocage jails on a host that are all started when the host is booted. They all automount their filesystems. I run zrepl on the host but not on the jails and have no problem pushing snapshots to the sink or pruning stale snapshots. I will take a look at my config on Monday.
 
I have a zrepl also on the host. zrepl is doing backup of the jails to another server, so in case there is something wrong with the server all jails can be run on the backup server instead. The backups are automatically mounted, and sometime the replication fails, claiming there has been done changes on the backup dataset. But I'm not aware of any changes. I have to resolve it manually by making rollback to the last snapshot to undo the "changes". Then the backup resumes. This also may happen even when someone is not using the zrepl, onle zfs send and zfs recieve.
If this repeats more often, I will try to make the backup filesystem mount read only to avoid the "changes", if it helps. It looks like the backup filesystem mounts itself automatically and possibly make the changes on it's own. So I thought that possibly more people have to deal with it. Doing the rollback always is not a good solution as if run the jails on a backup server and make real changes I want to keep them.
 
Andriy said:
Why do you have backup datasets mounted at all?
That does not happen for me.
Click to expand...

It seems that I have solved it by mounting it read only. Read only is good that I can have a look what I have.

YAML: 
      recv:
        placeholder:
          encryption: inherit
        properties: 
          override: {
                           readonly: on
       #                  "canmount": "noauto"
       #                  "canmount": "off"
                      }

I also have added compressed: true, behind send_properties: true to save network bandwidth .
 
Running backup is possible to interrupt using command zrepl signal reset JOB or with GUI zrepl status and Shirt+S (signal). However the job gets resumed with the next go. If I want to cancel the transfer, so it does not resume, I have to clean the zfs-abstraction . The zfs-abstraction is made of holds. Holds are zfs property which prevent deletion of snapshots and has got a name called tag.
The following ruby script is able to destroy dataset including all its snapshots and holds. It is convenient if you want to play with zrepl - you can repeat a operation with different parameters or if you make a mistake and a backup of huge dataset starts, which you did not want to backup.

I did not find this mentioned anywhere. Apparently everyone is expected to be clever enough to figure it out by himself. But it took me a while. Apart from this and too few working examples provided, I found zrepl working fine.

Ruby: 
#!/usr/bin/env ruby
# Force destroy a ZFS dataset or snapshot (ignoring holds)

target = ARGV[0] or abort "Usage: #{$0} <dataset|snapshot>\nIt deletes including all holds."

is_snapshot = target.include?('@')
puts "_  This will permanently delete #{target}#{' (and its snapshots)' unless is_snapshot}!"
print "Type 'yes' to continue: "
exit unless STDIN.gets.strip == 'yes'

def release_holds(obj)
  holds = `zfs holds -H #{obj} 2>/dev/null`.lines.map { |l| l.split[1] }.compact.uniq
  holds.each { |tag| system("zfs release -r #{tag} #{obj}") }
end

if is_snapshot
  release_holds(target)
  system("zfs destroy -r #{target}")
else
  snaps = `zfs list -H -t snapshot -o name -r #{target}`.lines.map(&:strip)
  snaps.each { |s| release_holds(s); system("zfs destroy -r #{s}") }
  system("zfs destroy -r #{target}")
end

puts "✅ Destroyed #{target}"
 
Today I was trying to make work the hooks. One of my jails has got the a database. I do not want to backup all data as they are huge and not that valuable.
So the /var/db/mysql directory is placed in different dataset and I do mysqldump before taking snapshots to dump what I need to keep.

The resulting sql dump is placed in a home directory of user root so it is included in the snapshot, which is taken at night each day.

I use ruby script again. To make it work, the full path has to be specified in almost all places. To the ruby interpreter, to jexec to mysqldump .



Ruby: 
#!/usr/local/bin/ruby
#!/usr/bin/env ruby     This does not work, the first line does
require 'open3'

$output="#{Time.now.strftime("%d/%m/%Y %H:%M")}|#{ENV['ZREPL_HOOKTYPE']}|#{ENV['ZREPL_FS']}|#{ENV['ZREPL_SNAPNAME']}|#{ENV['ZREPL_DRYRUN']}\n"


def runc(command)
stdout, stderr, status = Open3.capture3(command)
  $output<<"Output: #{stdout}\n"
  $output<<"Error: #{stderr}\n" if stderr
  $output<<"Exit Status: #{status.exitstatus}\n"
end

if ENV['ZREPL_FS']=='zroot/jails/homer' and ENV['ZREPL_HOOKTYPE']=="pre_snapshot" then
  runc("/usr/sbin/jexec homer sh -c '/usr/local/bin/mysqldump --no-data homer_db > /root/homer_db.sql'")
  runc("/usr/sbin/jexec homer sh -c '/usr/local/bin/mysqldump mysql user > /root/mysql_user.sql'")
  runc("/usr/sbin/jexec homer sh -c '/usr/local/bin/mysqldump reglog | gzip > /root/reglog.sql.gz'")
  File.write('/usr/local/etc/zrepl/zreplhooks.log',$output,mode: 'a')
end

To make the hook work I also have to place inside the zrepl yaml config.

YAML: 
    snapshotting:
      type: cron   
      prefix: zreplD_
      cron: "19 02 * * *"     
      hooks:
      - type: command
        path:  /usr/local/etc/zrepl/zreplhooks.rb
        timeout: 60s
        err_is_fatal: true       
        filesystems: {
          "zroot/jails/homer": true
        }
 
You must log in or register to reply here.
Back
Top