Offlate I'm seeing many clients failing with exit statys '58' though configurations looks good for me. Primary error message from the master server is
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd
Verified the logs and it mostly looks for me a client level firewall issue but I disabled all the firewall with no luck and same error. Any help is much appreciated.
Here are the test bpcd logs from master and corresponding logs from the client.
TEST-BPCD from master results:
D:\Veritas\NetBackup\bin\admincmd>bptestbpcd -client lcspepas56rnb -verbose -deb
ug
11:30:28.281 [4868.9624] <2> bptestbpcd: VERBOSE = 0
11:30:28.294 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:30:28.296 [4868.9624] <2> logconnections: BPCD CONNECT FROM 10.186.240.20.586
37 TO 10.186.242.56.1556 fd = 608
11:30:28.309 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded
11:35:28.323 [4868.9624] <8> vnet_sock_ready: [vnet.c:1485] max_time=300 sock=63
11:35:28.340 [4868.9624] <16> connect_to_service: connect failed STATUS (18) CON
NECT_FAILED
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx
status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu
rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd
11:35:28.341 [4868.9624] <8> vnet_connect_to_vnetd_ipi: [vnet_connect.c:258] con
nect_to_service() failed 18 0x12
11:35:28.341 [4868.9624] <16> bpcr_vnetd_connect_forward_socket_begin: vnet_conn
ect_to_vnetd_ipi(10.186.242.56) failed: 18
11:35:28.341 [4868.9624] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_sock
et_begin failed: 25
11:35:28.342 [4868.9624] <2> ConnectToBPCD: bpcd_connect_and_verify(lcspepas56rn
b, lcspepas56rnb) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(lcspepas56rnb) failed: 25
11:35:28.343 [4868.9624] <16> bptestbpcd main: Function ConnectToBPCD(lcspepas56
rnb) failed: 25
<2>bptestbpcd: cannot connect on socket
11:35:28.343 [4868.9624] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
11:35:28.343 [4868.9624] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
Client BPCD logs for the above test:
14:10:12.282 [2452.7104] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeBackupPrivilege)
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeDebugPrivilege)
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeRestorePrivilege)
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeSecurityPrivilege)
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTakeOwnershipPrivilege)
14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTcbPrivilege)
14:10:12.282 [2452.7104] <2> setup_debug_log: switching debug log file for bpcd
14:10:12.282 [2452.7104] <2> setup_debug_log: switched debug log file for bpcd
14:10:12.298 [2452.7104] <2> daemon_startup_listeners: 4 listening for legacy service bpcd
14:10:12.298 [2452.7104] <2> daemon_startup_listeners: listening for vnetd service bpcd
14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.132: pbxRegisterEx successful at 10.186.242.56:5126/bpcd, returns with 1 alt_addrs
14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.144: alt_addr: 10.186.163.91:5126
14:15:31.581 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied. )
14:30:34.868 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied. )
14:32:59.291 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied. )
14:35:35.057 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied. )
Client VNETD logs for above test:
12:58:33.645 [984.852] <2> main: VERBOSE = 0
12:58:33.645 [984.852] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\vnetd.exe -standalone
12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:635] msg VNETD ACCEPT FROM 10.186.240.20.59478 TO 10.186.242.56.13724 fd = 520
12:58:33.661 [984.852] <8> vnet_pop_byte: [vnet.c:1168] errno 0 0x0
12:58:33.661 [984.852] <2> vnet_pop_byte: vnet.c.1170: 0: Function failed: 9 0x00000009
12:58:33.661 [984.852] <2> vnet_pop_string: vnet.c.1250: 0: Function failed: 9 0x00000009
12:58:33.661 [984.852] <2> vnet_pop_signed: vnet.c.1293: 0: Function failed: 9 0x00000009
12:58:33.661 [984.852] <2> vnet_version_accept: .\vnetd.c.1089: 0: Function failed: 9 0x00000009
12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:640] version_accept failed 9 0x9
12:58:33.661 [984.852] <8> main: [.\vnetd.c:405] ProcessRequests failed 9 0x9
12:58:33.661 [984.852] <8> main: [.\vnetd.c:406] ProcessRequests returned 9 0x9
13:06:33.804 [5624.684] <2> setup_debug_log: switched debug log file for vnetd
13:06:33.804 [5624.684] <2> main: VERBOSE = 0