Other Hastd failing to attach device on FreeBSD 15.0

D

dsagra

Hi all, I am following the handbook to replicate data on a partition between 2 hosts. This is what I did:
* I created the partitions on both hosts:
Code: 
   [root@hosta ~]# gpart show=>        40  1953525088  nda0  GPT  (932G)
          40      532480     1  efi  (260M)
      532520        2008        - free -  (1.0M)
      534528   104857600     2  freebsd-zfs  (50G)
   105392128    16777216     3  freebsd-swap  (8.0G)
   122169344   104857600     4  freebsd-zfs  (50G)
   227026944     1024000     5  freebsd-zfs  (500M)
   228050944   209715200     6  freebsd-ufs  (100G)
   437766144  1515758984        - free -  (723G)
  
   [root@mattaneo ~]# gpart show
=>        40  3907029088  ada0  GPT  (1.8T)
          40      532480     1  efi  (260M)
      532520        2008        - free -  (1.0M)
      534528   104857600     2  freebsd-zfs  (50G)
   105392128    16777216     3  freebsd-swap  (8.0G)
   122169344   104857600     4  freebsd-zfs  (50G)
   227026944     1024000     5  freebsd-zfs  (500M)
   228050944   209715200     6  freebsd-ufs  (100G)
   437766144  3469262984        - free -  (1.6T)

* set up `/etc/hast.conf` on both hosts:
Code: 
listen 0.0.0.0

resource shared {
    on hostb {
        local /dev/ada0p6
        remote 10.255.255.1
    }
    on hosta {
        local /dev/nda0p6
        remote 10.255.255.2
    }
}

* ran the following commands on hosta, the to be primary node:
Code: 
hastctl create shared
service hastd start
hastctl role primary shared

* ran the following commands on hostb, the secondary one:
Code: 
hastctl create shared
service hastd start
hastctl role secondary shared

However, I keep seeing messages like this in /var/log/messages:
Code: 
/var/log/messages:{"time":"20260104171853","host":"nattaneo","facility":"[daemon.err]","msg":"hastd[88863]: Unable to receive header from tcp://10.255.255.2:50495: Operation timed out."
/var/log/messages:{"time":"20260104172241","host":"nattaneo","facility":"[daemon.err]","msg":"hastd[19854]: Unable to receive header from tcp://10.255.255.2:30507: Operation timed out."
/var/log/messages:{"time":"20260104172448","host":"nattaneo","facility":"[daemon.err]","msg":"hastd[19854]: Unable to receive header from tcp://10.255.255.2:33236: Operation timed out."
/var/log/messages:{"time":"20260104180418","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[51913]: Error processing log message: <30>1 2026-01-04T18:04:18.009047+00:00 >@<hastd: shared (primary) hastd 89635 - - [shared] (primary) Privileges successfully dropped."
/var/log/messages:{"time":"20260104182829","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[51913]: Error processing log message: <30>1 2026-01-04T18:28:29.316456+00:00 >@<hastd: shared (primary) hastd 89635 - - [shared] (primary) Termination signal received, exiting."
/var/log/messages:{"time":"20260104182829","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [1480] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260104182829","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [1480] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260104183010","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[51913]: Error processing log message: <30>1 2026-01-04T18:30:10.288291+00:00 >@<hastd: shared (primary) hastd 53598 - - [shared] (primary) Privileges successfully dropped."
/var/log/messages:{"time":"20260104184108","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [2240] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260104184108","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [2240] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260104184719","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[26434]: Error processing log message: <30>1 2026-01-04T18:47:19.334420+00:00 >@<hastd: shared (primary) hastd 75226 - - [shared] (primary) Privileges successfully dropped."
/var/log/messages:{"time":"20260104190514","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[26434]: Error processing log message: <30>1 2026-01-04T19:05:14.365850+00:00 >@<hastd: shared (primary) hastd 75226 - - [shared] (primary) Termination signal received, exiting."
/var/log/messages:{"time":"20260104190514","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [1104] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260104190514","host":"nattaneo","facility":"[kern.notice]","msg":"kernel: : [1104] g_dev_taste: g_dev_taste(hast/shared) failed to g_attach, error=6"
/var/log/messages:{"time":"20260105063137","host":"nattaneo","facility":"[syslog.err]","msg":"syslog-ng[26434]: Error processing log message: <30>1 2026-01-05T06:31:37.242261+00:00 >@<hastd: shared (primary) hastd 61114 - - [shared] (primary) Privileges successfully dropped."

This is confusing since both hosts can connect to each other on port 8457 tcp
Code: 
[ale@hosta ~]$ nc -vz 10.255.255.2 8457
Connection to 10.255.255.2 8457 port [tcp/*] succeeded!

[ale@hostb ~]$ nc -vz 10.255.255.1 8457
Connection to 10.255.255.1 8457 port [tcp/*] succeeded!

I am seeing the same behaviour regardless of what host I am trying to set up as primary: any thoughts what might be going on? Thank you
 
I did some more troubleshooting on this and the same thing happens when PF is not enabled.
  • hastctl dump show the partitions are the same size on both nodes:
    Code: 
    [root@hosta ~]# hastctl dump allresource: shared
    datasize: 107374171648 (100GB)
    extentsize: 2097152 (2.0MB)
    keepdirty: 64
    localoff: 10752
    resuid: 0
    localcnt: 0
    remotecnt: 0
    prevrole: primary

[root@hostb ~]# hastctl dump all
resource: shared
    datasize: 107374171648 (100GB)
    extentsize: 2097152 (2.0MB)
    keepdirty: 64
    localoff: 10752
    resuid: 0
    localcnt: 0
    remotecnt: 0
    prevrole: init
  • I keep seeing messages like the following in /var/log/messages
Code: 
Unable to receive header from tcp://10.255.255.2:33236: Operation timed out.
The same exact set up used to work as expected with FreeBSD 14.3, so probably something changed with FreeBSD 15.0 release?
 
You must log in or register to reply here.
Back
Top