Sap Backup Failed

Hi @Ayman_Seyam 

could you connect to the SAP HANA server and run the following command

su - <SID>adm 

(replace <SID>adm with the OS level db user example pypadm)

hdbsql -U BACKUP “select * from dummy”

additionally, review the ClHanaAgent.log on the SAP HANA server

Regards,

and here is the log 
2865588 2bb9b4 05/17 00:56:00 10327429 HdbSqlHelper::setSSLConnectString() - SSL not configured.
2865588 2bb9b4 05/17 00:56:00 10327429 HdbSqlHelper::setTraceLogDir() - Got CommCellId as 0, using default value of 2.
2865588 2bb9b4 05/17 00:56:00 10327429 HdbSqlHelper::setTraceLogDir() - jobResultDir was empty. Setting it to [/opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/10327429].
2865588 2bb9b4 05/17 00:56:00 10327429 HdbSqlHelper::setTraceLogDir() - traceLogDir was empty. Setting it to [/opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/10327429/HANATrace].
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::createHdbStoreKey() - ClientID list = <6373>
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/sapcontrol -nr 0 -function GetSystemInstanceList 2>&1].
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::runPopenCmd() - Command output [17.05.2023 00:56:00
GetSystemInstanceList
OK
hostname, instanceNr, httpPort, httpsPort, startPriority, features, dispstatus
et-s4db-pre, 0, 50013, 50014, 0.3, HDB|HDB_WORKER, GREEN
]
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::getHANAHosts() - et-s4db-pre  HDB|HDB_WORKER  GREEN
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::getHanaPythonVersion() - Command to read python version file = [cat /usr/sap/PYP/HDB00/exe/.python_version]
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::getHanaPythonVersion() - Python version found = [python3]
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::addPathsToEnvironmentVariableIfNotPresent() - Found environment variable [LD_LIBRARY_PATH] = [/usr/sap/PYP/HDB00/exe]
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::addPathsToEnvironmentVariableIfNotPresent() - Setting environment variable [LD_LIBRARY_PATH] = [/usr/sap/PYP/HDB00/exe:/usr/sap/PYP/HDB00/exe/Python3/lib:/usr/sap/PYP/HDB00/exe/Py3]
2865588 2bb9b4 05/17 00:56:00 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - hdbnsutil command to execute [/usr/sap/PYP/HDB00/exe/hdbnsutil -sr_state --sapcontrol=1 2>&1].
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - [SAPCONTROL-OK: <begin>]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - [online=true]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - [mode=none]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - [SAPCONTROL-OK: <end>]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::executeCommandToGetReplicationInfo() - [done.]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::getHanaNodeListForStoreKey() - Nodelist for creating HDB store key is [et-s4db-pre:30013,et-s4db-pre:30015].
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - /usr/sap/PYP/HDB00/exe/hdbuserstore List BACKUP 2>&1
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - Command completed with rc=0
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteDataBackup() - <HDB version info:
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteDataBackup() - <  version:             2.00.059.05.1662044871
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteDataBackup() - hdbStoreKey=<BACKUP>
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - /usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME, VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'multidb' AND KEY = 'mode'" 2>&1
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - Command completed with rc=0
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteDataBackup() - GUI Backup initiated for [PYP]. [Multi Tenant Database].
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - m_clientName=SAP_BACKEND_PreProd
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - instanceId=1139
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::refresh() - Refreshing HdbSqlHelper object.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setHdbUserStoreKeyInfo() - User store key not set by user. Will use [BACKUP] as key.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setSSLConnectString() - SSL not configured.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setTraceLogDir() - Got CommCellId as 0, using default value of 2.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setTraceLogDir() - jobResultDir was empty. Setting it to [/opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/10327429].
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setTraceLogDir() - traceLogDir was empty. Setting it to [/opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/10327429/HANATrace].
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - ClientNameList=<6373>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - instanceName=<PYP> id=1139
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - osUserName=<pypadm>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - db Node Number=<00>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbCmd=<ls -1d /usr/sap/PYP/HDB??>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - </usr/sap/PYP/HDB00
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - </usr/sap/PYP/HDB00></usr/sap/PYP/HDB>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - chkNodeNum<00>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbsqldir=</usr/sap/PYP/HDB00/exe>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::SwitchUser() - Change user=pypadm, gid=79, uid=1004
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::SwitchUser() - Current uid = [1004]gid = [79]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::SwitchUser() - Not calling initgroups, setuid, setgid for Non Root User
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::getSecuDirPath() - Command to find SECUDIR=[ls -d  /usr/sap/PYP/HDB00/*/sec]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::getSecuDirPath() - Command to find SECUDIR returned a single value. Using it as SECUDIR path: [/usr/sap/PYP/HDB00/et-s4db-pre/sec]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::setEnvironment() - Not setting HANA environment variables as nSetHANAEnv is not set
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbCmd=</usr/sap/PYP/HDB00/HDB -version>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <HDB version info:
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <  version:             2.00.059.05.1662044871
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hanaVersion=[2.00.059.05.1662044871]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - verNum=[59]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - SPS Version = [SPS05]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbStoreKey=<BACKUP>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbCmd=<cd /usr/sap/PYP/HDB00/exe; /usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME, VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'multidb' AND KEY = 'mode'" 2>&1>
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - cd /usr/sap/PYP/HDB00/exe; /usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME, VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'multidb' AND KEY = 'mode'" 2>&1
2865588 2bb9b4 05/17 00:56:01 10327429 CvProcess::system() - Command completed with rc=0
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::refresh() - Refreshing HdbSqlHelper object.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setHdbUserStoreKeyInfo() - User store key not set by user. Will use [BACKUP] as key.
2865588 2bb9b4 05/17 00:56:01 10327429 HdbSqlHelper::setSSLConnectString() - SSL not configured.
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - multiDb==true
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbCmd=</usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME,VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'multidb' AND KEY = 'database_isolation'">
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <LAYER_NAME,VALUE
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <DEFAULT,low
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <SYSTEM,low
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <2 rows selected (overall time 2997 usec; server time 2206 usec)
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <
>
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - IsolationLevel=[low]
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hanaCmd for finding the log backup frequency = </usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME,VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND KEY = 'log_backup_timeout_s' AND section = 'persistence' AND layer_name = 'SYSTEM'">
2865588 2bb9b4 05/17 00:56:01 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT LAYER_NAME,VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND KEY = 'log_backup_timeout_s' AND section = 'persistence' AND layer_name = 'SYSTEM'"].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [LAYER_NAME,VALUE
0 rows selected (overall time 1144 usec; server time 497 usec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hanaCmd for finding scale out param = </usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'persistence' and KEY = 'basepath_shared' and LAYER_NAME = 'SYSTEM'">
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT VALUE FROM M_INIFILE_CONTENTS WHERE file_name = 'global.ini' AND SECTION = 'persistence' and KEY = 'basepath_shared' and LAYER_NAME = 'SYSTEM'"].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [VALUE
0 rows selected (overall time 1364 usec; server time 565 usec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - Scale Out = [No]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hanaCmd for finding topology = </usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT HOST,PATH,SUBPATH from M_DISKS where USAGE_TYPE='DATA' order by HOST" 2>&1>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT HOST,PATH,SUBPATH from M_DISKS where USAGE_TYPE='DATA' order by HOST" 2>&1].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [HOST,PATH,SUBPATH
et-s4db-pre,/hana/data/PYP/,mnt00001
1 row selected (overall time 4512 usec; server time 3764 usec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - Processing entry=[et-s4db-pre,/hana/data/PYP/,mnt00001]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - hdbCmd=</usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "Select PATH,SUBPATH from M_DISKS where USAGE_TYPE='DATA'">
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <PATH,SUBPATH
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - </hana/data/PYP/,mnt00001
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - <1 row selected (overall time 3521 usec; server time 2866 usec)
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteBrowseDiscoverDatabases() - numNodes=[1]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - hanaCmd=</usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "Select DATABASE_NAME,ACTIVE_STATUS,OS_USER,OS_GROUP from M_DATABASES">
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <DATABASE_NAME,ACTIVE_STATUS,OS_USER,OS_GROUP
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <SYSTEMDB,YES,,
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - tenDb=[SYSTEMDB][YES][][]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <PYP,YES,,
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - tenDb=[PYP][YES][][]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <2 rows selected (overall time 961 usec; server time 252 usec)
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - numRec=[2]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getStoragePoliciesFromConfiguredInstance() - Got Application ID. AppNum [2:16298]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getStoragePoliciesFromConfiguredInstance() - isDefault [1], isCommandline [0], isLogCommandline [0]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getStoragePoliciesFromConfiguredInstance() - Got Application ID. AppNum [2:16299]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getStoragePoliciesFromConfiguredInstance() - isDefault [0], isCommandline [1], isLogCommandline [0]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getStoragePoliciesFromConfiguredInstance() - Data SP [209], logSP [209], commandline SP [209]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::CreateBackupsetsForInstance() - Got data SP [209], log SP [209], commandline SP [209], bSetStoragePolicies [1]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::CreateBackupsetsForInstance() - Backupset SYSTEMDB for instance  already exists. Continue with next database.
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::CreateBackupsetsForInstance() - Backupset PYP for instance  already exists. Continue with next database.
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Got Commvault installation directory=[/opt/commvault] Commvault backint path=[/opt/commvault/iDataAgent/backint]
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Skip creating the symbolic link. Required symbolic link already exists [/usr/sap/PYP/SYS/global/hdb/opt/hdbbackint] ---> [/opt/commvault/iDataAgent/backint].
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Skip creating the symbolic link. Required symbolic link already exists [/usr/sap/PYP/SYS/global/hdb/opt/hdbconfig/param] ---> [/opt/commvault/iDataAgent/jobResults/SapHana/PYP/param].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - hanaCmd=</usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "Select DATABASE_NAME,ACTIVE_STATUS,OS_USER,OS_GROUP from M_DATABASES">
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <DATABASE_NAME,ACTIVE_STATUS,OS_USER,OS_GROUP
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <SYSTEMDB,YES,,
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - tenDb=[SYSTEMDB][YES][][]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <PYP,YES,,
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - tenDb=[PYP][YES][][]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - <2 rows selected (overall time 989 usec; server time 304 usec)
>
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::getTenantDBList() - numRec=[2]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::setNumberOfStreamsInGlobalIni() - Setting [parallel_data_backup_backint_channels] = [4] in global.ini
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'SYSTEM') SET ('backup', 'parallel_data_backup_backint_channels') = '4' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 135.947 msec; server time 134.008 msec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::setNumberOfStreamsInGlobalIni() - Setting [parallel_data_backup_backint_channels] = [4] in global.ini
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'parallel_data_backup_backint_channels') = '4' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 112.678 msec; server time 110.800 msec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteDataBackup() - RetCode from the Discover Databases = [0]
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Got Commvault installation directory=[/opt/commvault] Commvault backint path=[/opt/commvault/iDataAgent/backint]
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Skip creating the symbolic link. Required symbolic link already exists [/usr/sap/PYP/SYS/global/hdb/opt/hdbbackint] ---> [/opt/commvault/iDataAgent/backint].
2865588 2bb9b4 05/17 00:56:02 10327429 ::checkSymbolicLinksForBackint() - Skip creating the symbolic link. Required symbolic link already exists [/usr/sap/PYP/SYS/global/hdb/opt/hdbconfig/param] ---> [/opt/commvault/iDataAgent/jobResults/SapHana/PYP/param].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::ExecuteDataBackup() - global.ini file=[/usr/sap/PYP/SYS/global/hdb/custom/config/DB_PYP/global.ini].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'log_backup_using_backint') = 'true' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 131.111 msec; server time 129.488 msec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'log_backup_parameter_file') = '/usr/sap/PYP/SYS/global/hdb/opt/hdbconfig/param' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 119.609 msec; server time 118.089 msec)
]
2865588 2bb9b4 05/17 00:56:02 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'data_backup_parameter_file') = '/usr/sap/PYP/SYS/global/hdb/opt/hdbconfig/param' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 122.086 msec; server time 120.094 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'backint_response_timeout') = '1800' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 114.720 msec; server time 112.690 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'catalog_backup_using_backint') = 'true' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 125.624 msec; server time 123.759 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'catalog_backup_parameter_file') = '/usr/sap/PYP/SYS/global/hdb/opt/hdbconfig/param' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 118.958 msec; server time 117.373 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::setNumberOfStreamsInGlobalIni() - Setting [parallel_data_backup_backint_channels] = [4] in global.ini
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'parallel_data_backup_backint_channels') = '4' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 118.768 msec; server time 117.182 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Cmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "ALTER SYSTEM ALTER CONFIGURATION ('global.ini', 'DATABASE', 'PYP') SET ('backup', 'parallel_data_backup_backint_size_threshold') = '32' WITH RECONFIGURE " 2>&1].
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::runPopenCmd() - Command output [0 rows affected (overall time 112.528 msec; server time 110.504 msec)
]
2865588 2bb9b4 05/17 00:56:03 10327429 JM Client  CVJobClient::initialize(): Got remote host [site_a_commserv.efinance.com.eg].
2865588 2bb9b4 05/17 00:56:03 10327429 JM Client  CVBkpJobClient::init(): Initializing job object with token [10327429:4:7:0:0:12556].
2865588 2bb9b4 05/17 00:56:03 10327429 InvalidatePreviousAttempts() - Invalidated previous backup attempts with Job Manager
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::setLastBackupSize() - Get SUM(ALLOCATED_PAGE_SIZE) cmd  = [/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "SELECT SUM(ALLOCATED_PAGE_SIZE) FROM SYS_DATABASES.M_CONVERTER_STATISTICS WHERE DATABASE_NAME='PYP'"]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::setLastBackupSize() - SUM(ALLOCATED_PAGE_SIZE)=[140320882688]
2865588 2bb9b4 05/17 00:56:03 10327429 ClHanaAgent::ExecuteDataBackup() - hanaBkpCmd=[/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "backup data for PYP using backint ('10327429_COMPLETE_DATA_BACKUP') TOOLOPTION 'BACKUP -j 10327429 -a 2:16304 -t 1 -jt 10327429:4:7:0:0:12556 -data -at 135 -d site-a-ma-03.efinance.com.eg*Site-A-MA-03*8400 -vm Instance001 -cn SAP_BACKEND_PreProd'"]
2865588 2bb9b4 05/17 00:56:03 10327429 ::RunHdbSqlCommand() - enter
2865588 2bb9b4 05/17 00:56:03 10327429 ::RunHdbSqlCommand() - hdbSqlCmdAddNull=[/bin/sh /opt/commvault/Base/Temp//hanaBkp_2865588 > /opt/commvault/Base/Temp//hanaBkp_2865588.out 2>&1 NULL]
2865588 2bb9b4 05/17 00:56:03 10327429 ::RunHdbSqlCommand() - start HdbSqlCmd:path=[/bin/sh]
2865588 2bb9b4 05/17 00:56:03 10327429 OraChildProcess::SetPostForkParam() - Parent path = /bin/sh
2865588 2bb9b4 05/17 00:56:03 10327429 OraChildProcess::SetPostForkParam() - Parent oraUser = pypadm
2865588 2bb9b4 05/17 00:56:03 10327429 OraChildProcess::InitializeUnix() - Parent: child process pid= 2865801
2865588 2bba8a 05/17 00:56:03 10327429 OraChildProcess::WaitThread() - start: thread Id=2865588
2865588 2bba8a 05/17 00:56:03 10327429 OraChildProcess::Wait() - Child exited with error: 3
2865588 2bba8a 05/17 00:56:03 10327429 OraChildProcess::WaitThread() - Wait failed; lastErr=3
2865588 2bb9b4 05/17 00:56:06 10327429 ::NotifyCommServeJobComplete() - Notify CommServe: jobObject.complete(FAIL,PHASE)
2865588 2bb9b4 05/17 00:56:06 10327429 BKP CALLED COMPLETE (PHASE Status::FAIL), 10327429. Token [10327429:4:7:0:0:12556]
2865588 2bb9b4 05/17 00:56:07 10327429 ::RunHdbSqlCommand() - HDBSql command Failed.
Please, check the client log files for details.
2865588 2bb9b4 05/17 00:56:07 10327429 ::ExitHere() - enter: phase=10
2865588 2bb9b4 05/17 00:56:07 10327429 GetChannelParamFileName() - Channel Parameter File : /opt/commvault/Base/Temp/cv_channel_info10327429
2865588 2bb9b4 05/17 00:56:07 10327429 ::NotifyCommServeJobComplete() - Notify CommServe: jobObject.complete(FAIL,PHASE)
2865588 2bb9b4 05/17 00:56:07 10327429 BKP CALLED COMPLETE (PHASE Status::FAIL), 10327429. Token [10327429:4:7:0:0:12556]
2865588 2bb9b4 05/17 00:56:07 10327429 JM Client  JobClient: Jobmanager replied with an error code of [3] in the acknowledgement message. Error [Invalid token [10327429:4:7:0:0:12556] received for the job complete message. Error:[The job token is for an older attempt than that of the job. AttemptNo in client token [7]. Current AttemptNo of job [8].].]
2865588 2bb9b4 05/17 00:56:07 10327429 ::complete() - m_jobObjectB->complete failed
2865588 2bb9b4 05/17 00:56:07 10327429 ::NotifyCommServeJobComplete() - jobObject.complete(FAIL,PHASE) failed.
2865588 2bb9b4 05/17 00:56:07 10327429 ::NotifyCommServeJobComplete() - First error is already set to [HDBSql command Failed.
Please, check the client log files for details.]
2865588 2bb9b4 05/17 00:56:07 10327429 ::ExitHere() - m_firstErrDescr = [HDBSql command Failed.
Please, check the client log files for details.]
2865588 2bb9b4 05/17 00:56:07 10327429 ::Finalize() - enter Parent=true
2865588 2bb9b4 05/17 00:56:07 10327429 ::Finalize() - m_jobControl->finishedWorkWithoutExit() ...
2865588 2bb9b4 05/17 00:56:07 10327429 finishedWorkWithOutExit() - Release control semaphore 10327429_2865588
2865588 2bb9b4 05/17 00:56:07 10327429 finishedWorkWithOutExit() - Waiting for control thread to Quit, 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 controlThread() - Normal termination, Job Id 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 controlThread() - Send unregister message to CVD, Job Id 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 unRegisterProcess() - Send Message to CVD 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 unRegisterProcess() - Started Session 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 unRegisterProcess() - Send Message Successfully 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 controlThread() - Close control semaphore, Job Id 10327429
2865588 2bb9b6 05/17 00:56:07 10327429 controlThread() - Control thread done, Job Id 10327429
2865588 2bb9b4 05/17 00:56:07 10327429 ::Finalize() - delete m_jobControl ...
2865588 2bb9b4 05/17 00:56:07 10327429 ::CloseJobResultsFile() - Writing the partial RMAN log [/usr/sap/PYP/HDB00/exe/hdbsql   -t -U BACKUP -d SYSTEMDB -C "backup data for PYP using backint ('10327429_COMPLETE_DATA_BACKUP') TOOLOPTION 'BACKUP -j 10327429 -a 2:16304 -t 1 -jt 10327429:4:7:0:0:12556 -data -at 135 -d site-a-ma-03.efinance.com.eg*Site-A-MA-03*8400 -vm Instance001 -cn SAP_BACKEND_PreProd'"]
2865588 2bb9b4 05/17 00:56:07 10327429 ::CloseJobResultsFile() - [10327429] going to m_buResults.closeFile...
2865588 2bb9b4 05/17 00:56:07 10327429 ::CloseJobResultsFile() - [10327429] m_buResults.closeFile() & cleanUp() succeeded.
2865588 2bb9b4 05/17 00:56:07 10327429 ::Finalize() - [10327429] delete m_jobObject.
2865588 2bb9b4 05/17 00:56:07 10327429 ::~DbJobObject() - m_jobObjectB->terminateAllSessions()...
2865588 2bb9b4 05/17 00:56:07 10327429 ::~DbJobObject() - After m_jobObject deletion.
2865588 2bb9b4 05/17 00:56:07 10327429 ::Finalize() - delete m_event ...
 

not solved yet