I need a solution
Backup failing with error 84 for multiple clients for huge Data during Multiplexing, but successful if using small data. Errors: Media write error (84), FTL - socket write failed, 24: socket write failed, INF - Server status = 24 & cannot write data to network
Errors: Media write error(84), FTL - socket write failed, 24: socket write failed, INF - Server status = 24 & cannot write data to network: An existing connection was forcibly closed by the remote host.
My environment:
NetBackup 7.1
All the systems are in 1 VMware ESXi 5.1 Server-
S002-VM2: Master Server (Windows 2008 Ent x86)
S003: Media Server (Windows 2008 Stn x86) – Firestreamer Virtual Tape Library
S001: Client (Windows 2008 Ent x86)
S004: Client (Windows 2008 Ent x86)
C007: Client (Windows 7 x86)
Policy: Clients – S001, C00-VM5 & S004. Multi streaming and multiple data streams enabled. Backup selections “C:\”.
Issue details:
Activity Monitor Job Error: media write error (84)
Job details: Critical bpbrm(pid=2360) from client C002-VM5.flab.com: FTL - socket write failed
Bpbrm logs on Media Server:
07:04:12.648 [2064.3400] <32> bpbrm handle_backup: from client C002-VM5.flab.com: FTL - socket write failed
07:04:12.648 [2064.3400] <2> ConnectionCache::connectAndCache: Acquiring new connection for host s002-vm2.flab.com, query type 1
07:04:12.648 [2064.3400] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:04:12.648 [2064.3400] <2> logconnections: BPDBM CONNECT FROM 10.0.0.8.49534 TO 10.0.0.4.1556 fd = 604
07:04:12.710 [2532.2328] <32> bpbrm handle_backup: from client S001.flab.com: FTL - socket write failed
07:04:12.710 [2532.2328] <2> ConnectionCache::connectAndCache: Acquiring new connection for host s002-vm2.flab.com, query type 1
07:04:12.710 [2532.2328] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:04:12.710 [2532.2328] <2> logconnections: BPDBM CONNECT FROM 10.0.0.8.49535 TO 10.0.0.4.1556 fd = 604
07:04:12.851 [3852.2368] <2> bpcr_get_platform_rqst: Server client platform length = 7
07:04:12.851 [3852.2368] <2> bpcr_check_for_use_ofb_support: bpcd platform win_x86
07:04:12.851 [3852.2368] <2> MNG: backup_cmd = /usr/openv/netbackup/bin/bpbkar bpbkar32 -r 604800 -ru root -dt 0 -to 0 -clnt S004.flab.com -class S-1-2-3-C-Drive_Big_MPX-VTL-onS003 -sched Full -st FULL -bpstart_to 300 -bpend_to 300 -read_to 9600 -blks_per_buffer 127 -stream_count 1 -stream_number 1 -jobgrpid 477 -use_otm -use_ofb -b S004.flab.com_1364088814 -kl 28 -fso -WOFB_enabled -WOFB_fim 1 -WOFB_usage 0 -WOFB_error 0 -ct 13
07:04:12.851 [3852.2368] <2> bpbrm handle_backup: forking client backup
07:04:12.851 [3852.2368] <2> bpbrm send_bpsched_connected_msg: sending bpsched msg: CONNECTED TO CLIENT FOR S004.flab.com_1364088814
07:04:12.897 [2064.3400] <2> db_end: Need to collect reply
07:04:12.897 [2064.3400] <2> bpbrm wait_for_mm_events: unexpected terminate
07:04:12.897 [2064.3400] <2> inform_client_of_status: INF - Server status = 150
07:04:12.897 [2064.3400] <2> put_string: cannot write data to network: An existing connection was forcibly closed by the remote host.
07:04:12.897 [2064.3400] <16> inform_client_of_status: could not send server status message
07:04:12.897 [2064.3400] <2> ConnectionCache::connectAndCache: Acquiring new connection for host s002-vm2.flab.com, query type 1
07:04:12.913 [2064.3400] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:04:12.913 [2064.3400] <2> logconnections: BPDBM CONNECT FROM 10.0.0.8.49536 TO 10.0.0.4.1556 fd = 604
07:04:12.913 [2532.2328] <2> db_end: Need to collect reply
07:04:12.913 [2532.2328] <2> bpbrm handle_backup: client S001.flab.com EXIT STATUS = 24: socket write failed
07:04:12.913 [2532.2328] <2> inform_client_of_status: INF - Server status = 24
07:04:12.913 [2532.2328] <2> put_string: cannot write data to network: An existing connection was forcibly closed by the remote host.
Bpbkar on client C002-VM5:
[100.1852] <16> tar_tfi::processException:
An Exception of type [SocketWriteException] has occured at:
Module: @(#) $Source: src/ncf/tfi/lib/TransporterRemote.cpp,v $ $Revision: 1.54 $ , Function: TransporterRemote::write[2](), Line: 321
Module: @(#) $Source: src/ncf/tfi/lib/Packer.cpp,v $ $Revision: 1.89 $ , Function: Packer::getBuffer(), Line: 656
Module: tar_tfi::getBuffer, Function: H:\71\src\cl\clientpc\util\tar_tfi.cpp, Line: 312
Local Address: [::]:0
Remote Address: [::]:0
OS Error: 10054 (An existing connection was forcibly closed by the remote host.
)
Attempted Solution:
I referred to Article “Status code 24 - Socket write failed” URL http://www.symantec.com/docs/TECH150369
Made the appropriate changes as advised in the document.
1) Changed client read timeout parameter from 300 to 9600.
2) Changed Communication buffer size from 32 Kb to 128 KB. Go to Host Properties > Clients > Client Properties > Windows Client > ClientSettings > Communication buffer size = 128
3) In case there is an Antivirus running, turn it off for troubleshooting proposes. (All Systems)
4) Disabled autotuning and chimney features, from command prompt run: (On Master Server)
netsh int tcp set global autotuning=disabled
netsh int tcp set global chimney=disabled
5) Created TcpTimedWaitDelay entry(as “RED_DWORD”) in HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and set the value to 30 seconds. (On Master Server)
6) Rebooted the Master server.
7) Started the Backup again, but same errors.
I have also attached my error logs files for bpbkar, bpbrm & bptm.
Note: If I remove a client from Policy and run backup it completes successfully. Or, if I use less amount of data then as well backup completes. Moreoever, for some reason I cannot use the same MPX Media for backups again and get error 86 on it, but that is not a priority right now.