FreeBSD 10.0-RELEASE-p7
Current pkgs all up to date.
I'm attempting to troubleshoot and discover why during load testing that I'm getting blocks of errors from HAST. The errors are as follows:
The times as you can see don't quite match up, however all these errors are very concerning. The hosts are connected via a single QSFP+ cable providing 4x10gbe paths that are bonded in a L3+L4 lagg hash. HOST3 is 10.10.10.11, and HOST4 is 10.10.10.12 and this is a host to host connection (no switch).
/etc/hast.conf looks like this (small snip):
Wondering if this could be caused by missing /etc/hosts entries for the host itself (though the config file does say what port to bind to, and has worked fine for a while on FreeBSD 9.1-RELEASE).
/etc/hosts looks like this:
There are of course HOST3 / HOST4 hostnames assigned to the system via /etc/rc.conf with the management IPs associated.
These manifest during times of stress testing (20 clients connected to ZVOL shares over istgt running lots of IO benchmark tests). Poor performance aside, I'd like to rule out HAST connectivity issues first. As you can see, I get giant blocks of these errors, and this is just over 2 days. The 'no route to host' bit really scares me...
Wondering if there's a problem with load as well. Help troubleshooting this would be awesome. Thanks.
Current pkgs all up to date.
I'm attempting to troubleshoot and discover why during load testing that I'm getting blocks of errors from HAST. The errors are as follows:
Code:
Sep 9 15:37:24 HOST4 hastd[3829]: [jbod1-slot00] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:26 HOST4 hastd[3888]: [jbod3-slot02] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:27 HOST4 hastd[4711]: [jbod1-slot04] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:27 HOST4 hastd[3835]: [jbod1-slot05] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:27 HOST4 hastd[3838]: [jbod1-slot06] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:27 HOST4 hastd[3841]: [jbod1-slot07] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[3846]: [jbod1-slot08] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[23243]: [jbod1-slot09] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[17407]: [jbod1-slot10] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[3853]: [jbod1-slot11] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[65416]: [jbod1-slot12] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[9227]: [jbod1-slot13] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[3865]: [jbod1-slot14] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[60652]: [jbod1-slot15] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:28 HOST4 hastd[3869]: [jbod1-slot16] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:29 HOST4 hastd[3879]: [jbod1-slot17] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:29 HOST4 hastd[3882]: [jbod1-slot18] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:29 HOST4 hastd[3883]: [jbod1-slot19] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:29 HOST4 hastd[3886]: [jbod1-slot20] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:29 HOST4 hastd[3887]: [jbod1-slot21] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:37:51 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:37:53 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:37:56 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:37:58 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:03 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:06 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:08 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:10 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:16 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:18 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:21 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:22 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:28 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 15:38:29 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:31 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:33 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:38 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:39 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:41 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:43 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:48 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:49 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:51 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:53 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:38:59 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:39:00 HOST4 hastd[3895]: [jbod3-slot41] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:39:00 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:39:01 HOST4 hastd[3903]: [jbod3-slot44] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:39:01 HOST4 hastd[3897]: [jbod3-slot43] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:39:01 HOST4 hastd[3896]: [jbod3-slot42] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:39:01 HOST4 hastd[3889]: [jbod3-slot40] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 15:39:01 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:39:03 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Connection reset by peer.
Sep 9 15:39:09 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:10 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:10 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:11 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:13 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:13 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:13 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:14 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:14 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:17 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:18 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:18 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:18 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:19 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:19 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:39:20 HOST4 hastd[3564]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:20 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:20 HOST4 hastd[3564]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:20 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:20 HOST4 hastd[3564]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:20 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:21 HOST4 hastd[3564]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:21 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:22 HOST4 hastd[3564]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:39:22 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: No route to host.
... 1500 lines removed ...
Sep 9 15:45:59 HOST4 hastd[3564]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:45:59 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[3564]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[3564]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[3564]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[3564]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:00 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:01 HOST4 hastd[3564]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:01 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:01 HOST4 hastd[3564]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:01 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:04 HOST4 hastd[3564]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:04 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:04 HOST4 hastd[3564]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:04 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:05 HOST4 hastd[3564]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:05 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:05 HOST4 hastd[3564]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:05 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3564]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3564]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3564]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3564]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:06 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:07 HOST4 hastd[3564]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:07 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: No route to host.
Sep 9 15:46:08 HOST4 hastd[3564]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:08 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:09 HOST4 hastd[3564]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:09 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:09 HOST4 hastd[3564]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:09 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3564]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3564]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3564]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[3564]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:10 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:11 HOST4 hastd[3564]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:11 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:12 HOST4 hastd[3564]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:12 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:12 HOST4 hastd[3564]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:12 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:14 HOST4 hastd[3564]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:14 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:14 HOST4 hastd[3564]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:14 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Network is down.
Sep 9 15:46:15 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:15 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:16 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:16 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:16 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:16 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:17 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:18 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:19 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:19 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:20 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:20 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:20 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:20 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:21 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:22 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:22 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:24 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:24 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:25 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:25 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:26 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:26 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:26 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:26 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:27 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:28 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:29 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:30 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:30 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:30 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:30 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:31 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:31 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:32 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:32 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:35 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:35 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:35 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:35 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:36 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:36 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:36 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:36 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:38 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:39 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:39 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:40 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:40 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:40 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:40 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:41 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:41 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:42 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:42 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:45 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:45 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:45 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:46 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:46 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:46 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:46 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:46 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:48 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:49 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:49 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:50 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:50 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:50 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:51 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:51 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:51 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:52 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:53 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:55 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:55 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:56 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:56 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:56 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:56 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:56 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:57 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:58 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:59 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:46:59 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:00 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:00 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:01 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:01 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:01 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:02 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:03 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:03 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:05 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:05 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:06 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:06 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:06 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:07 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:07 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:07 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:08 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:09 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:10 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:10 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:10 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:11 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:11 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:12 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:12 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:13 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:13 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:15 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:15 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:16 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:16 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:17 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:17 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:17 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:17 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:18 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:19 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:20 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:20 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:20 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:21 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:21 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:22 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:22 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:23 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:23 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:26 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:26 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:26 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:26 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:27 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:27 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:27 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:27 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:28 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:29 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:30 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:30 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:30 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:31 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:32 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:32 HOST4 hastd[3717]: [jbod2-slot33] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:32 HOST4 hastd[12829]: [jbod2-slot34] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:33 HOST4 hastd[89414]: [jbod2-slot32] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:33 HOST4 hastd[44330]: [jbod2-slot23] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:36 HOST4 hastd[3711]: [jbod2-slot31] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:36 HOST4 hastd[28403]: [jbod2-slot25] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:36 HOST4 hastd[3684]: [jbod2-slot22] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:36 HOST4 hastd[3786]: [jbod2-slot39] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:37 HOST4 hastd[18640]: [jbod2-slot30] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:37 HOST4 hastd[3699]: [jbod2-slot27] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:37 HOST4 hastd[3734]: [jbod2-slot38] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:38 HOST4 hastd[3723]: [jbod2-slot35] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:39 HOST4 hastd[3681]: [jbod2-slot01] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:39 HOST4 hastd[3729]: [jbod2-slot37] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:40 HOST4 hastd[3705]: [jbod2-slot29] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:40 HOST4 hastd[6190]: [jbod2-slot28] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:40 HOST4 hastd[3696]: [jbod2-slot26] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:42 HOST4 hastd[13446]: [jbod2-slot36] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 15:47:42 HOST4 hastd[3690]: [jbod2-slot24] (primary) Unable to connect to HOST3-REP: Connection refused.
Sep 9 20:35:17 HOST4 hastd[33794]: [jbod2-slot23] (primary) Unable to receive reply header: Operation timed out.
Sep 9 20:35:26 HOST4 hastd[33794]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 20:36:14 HOST4 hastd[33794]: [jbod2-slot23] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 9 21:49:38 HOST4 hastd[11563]: [jbod1-slot00] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:39 HOST4 hastd[11587]: [jbod3-slot02] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11564]: [jbod1-slot04] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11565]: [jbod1-slot05] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11566]: [jbod1-slot06] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11567]: [jbod1-slot07] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11568]: [jbod1-slot08] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:40 HOST4 hastd[11569]: [jbod1-slot09] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11570]: [jbod1-slot10] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11571]: [jbod1-slot11] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11572]: [jbod1-slot12] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11573]: [jbod1-slot13] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11574]: [jbod1-slot14] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11575]: [jbod1-slot15] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11576]: [jbod1-slot16] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11577]: [jbod1-slot17] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:41 HOST4 hastd[11583]: [jbod1-slot18] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11584]: [jbod1-slot19] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11586]: [jbod1-slot20] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11585]: [jbod1-slot21] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11588]: [jbod3-slot40] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11589]: [jbod3-slot41] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11590]: [jbod3-slot42] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11591]: [jbod3-slot43] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:42 HOST4 hastd[11592]: [jbod3-slot44] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 22:13:44 HOST4 hastd[4117]: [jbod1-slot00] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 22:14:10 HOST4 hastd[4042]: [jbod2-slot01] (primary) Unable to receive reply header: Operation timed out.
Sep 9 22:16:20 HOST4 hastd[4184]: [jbod3-slot02] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST4 hastd[4087]: [jbod2-slot36] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4093]: [jbod2-slot38] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4057]: [jbod2-slot26] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4060]: [jbod2-slot27] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4096]: [jbod2-slot39] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4078]: [jbod2-slot33] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4075]: [jbod2-slot32] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST4 hastd[4063]: [jbod2-slot28] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4063]: [jbod2-slot28] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4060]: [jbod2-slot27] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4075]: [jbod2-slot32] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4093]: [jbod2-slot38] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4096]: [jbod2-slot39] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4078]: [jbod2-slot33] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4087]: [jbod2-slot36] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4057]: [jbod2-slot26] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4081]: [jbod2-slot34] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4051]: [jbod2-slot24] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4066]: [jbod2-slot29] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:47 HOST4 hastd[4072]: [jbod2-slot31] (primary) Unable to receive reply header: Operation timed out.
Sep 11 16:58:53 HOST4 hastd[4063]: [jbod2-slot28] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 11 16:58:55 HOST4 hastd[4075]: [jbod2-slot32] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 11 16:58:55 HOST4 hastd[4087]: [jbod2-slot36] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 11 16:58:56 HOST4 hastd[4069]: [jbod2-slot30] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Sep 11 16:58:56 HOST4 hastd[4093]: [jbod2-slot38] (primary) Unable to receive handshake header from HOST3-REP: Operation timed out.
Code:
Sep 9 19:00:18 HOST3 hastd[3763]: [jbod2-slot31] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 19:00:18 HOST3 hastd[3763]: [jbod2-slot31] (secondary) Unable to send reply: Broken pipe.
Sep 9 19:14:11 HOST3 hastd[12113]: [jbod2-slot31] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 19:36:42 HOST3 hastd[3785]: [jbod2-slot38] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 19:36:42 HOST3 hastd[3785]: [jbod2-slot38] (secondary) Unable to send reply: Broken pipe.
Sep 9 20:16:55 HOST3 hastd[3762]: [jbod2-slot22] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 20:22:46 HOST3 hastd[3764]: [jbod2-slot39] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 20:27:36 HOST3 hastd[3761]: [jbod2-slot23] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 20:35:17 HOST3 hastd[15648]: [jbod2-slot23] (secondary) Unable to send reply: Broken pipe.
Sep 9 20:35:26 HOST3 hastd[15944]: [jbod2-slot23] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 9 20:35:38 HOST3 hastd[15958]: [jbod2-slot23] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 9 20:35:50 HOST3 hastd[15965]: [jbod2-slot23] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 9 20:36:02 HOST3 hastd[15989]: [jbod2-slot23] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 9 20:36:14 HOST3 hastd[16017]: [jbod2-slot23] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 9 21:48:59 HOST3 hastd[3786]: [jbod2-slot01] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[15238]: [jbod2-slot22] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[16041]: [jbod2-slot23] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3760]: [jbod2-slot24] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3804]: [jbod2-slot25] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3800]: [jbod2-slot26] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3765]: [jbod2-slot27] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3799]: [jbod2-slot28] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 21:49:00 HOST3 hastd[3801]: [jbod2-slot29] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 22:13:44 HOST3 hastd[18601]: [jbod1-slot00] (primary) Unable to receive reply header: Operation timed out.
Sep 9 22:14:10 HOST3 hastd[18699]: [jbod2-slot01] (secondary) Unable to receive request header: Socket is not connected.
Sep 9 22:16:20 HOST3 hastd[18658]: [jbod3-slot02] (primary) Unable to receive reply header: Operation timed out.
Sep 10 20:12:48 HOST3 hastd[18743]: [jbod2-slot33] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18736]: [jbod2-slot32] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18748]: [jbod2-slot38] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18715]: [jbod2-slot27] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18746]: [jbod2-slot36] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18718]: [jbod2-slot28] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18749]: [jbod2-slot39] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18714]: [jbod2-slot26] (secondary) Unable to receive request header: Socket is not connected.
Sep 10 20:12:48 HOST3 hastd[18714]: [jbod2-slot26] (secondary) Unable to send reply: Broken pipe.
Sep 11 16:58:47 HOST3 hastd[72174]: [jbod2-slot33] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72178]: [jbod2-slot28] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72180]: [jbod2-slot32] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72175]: [jbod2-slot38] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72176]: [jbod2-slot27] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72181]: [jbod2-slot36] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72179]: [jbod2-slot39] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[18709]: [jbod2-slot25] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[18727]: [jbod2-slot31] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[72177]: [jbod2-slot26] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:47 HOST3 hastd[18724]: [jbod2-slot30] (secondary) Unable to receive request header: Socket is not connected.
Sep 11 16:58:54 HOST3 hastd[22698]: [jbod2-slot28] (secondary) Unable to send activemap to HOST4-REP: Connection reset by peer.
Sep 11 16:58:55 HOST3 hastd[22702]: [jbod2-slot32] (secondary) Unable to send activemap to HOST4-REP: Connection reset by peer.
Sep 11 16:58:55 HOST3 hastd[22703]: [jbod2-slot36] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 11 16:58:57 HOST3 hastd[22706]: [jbod2-slot30] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.
Sep 11 16:58:57 HOST3 hastd[22707]: [jbod2-slot38] (secondary) Unable to send activemap to HOST4-REP: Broken pipe.The times as you can see don't quite match up, however all these errors are very concerning. The hosts are connected via a single QSFP+ cable providing 4x10gbe paths that are bonded in a L3+L4 lagg hash. HOST3 is 10.10.10.11, and HOST4 is 10.10.10.12 and this is a host to host connection (no switch).
/etc/hast.conf looks like this (small snip):
Code:
resource jbod2-slot39 {
on HOST3 {
local /dev/gpt/jbod2-slot39
local /dev/gpt/jbod2-slot39
remote HOST4-REP
}
on HOST4 {
local /dev/gpt/jbod2-slot39
local /dev/gpt/jbod2-slot39
remote HOST3-REP
}
}Wondering if this could be caused by missing /etc/hosts entries for the host itself (though the config file does say what port to bind to, and has worked fine for a while on FreeBSD 9.1-RELEASE).
/etc/hosts looks like this:
Code:
10.10.10.11 HOST3-REP
10.10.10.12 HOST4-REPThere are of course HOST3 / HOST4 hostnames assigned to the system via /etc/rc.conf with the management IPs associated.
These manifest during times of stress testing (20 clients connected to ZVOL shares over istgt running lots of IO benchmark tests). Poor performance aside, I'd like to rule out HAST connectivity issues first. As you can see, I get giant blocks of these errors, and this is just over 2 days. The 'no route to host' bit really scares me...
Wondering if there's a problem with load as well. Help troubleshooting this would be awesome. Thanks.