Hello,
Commvault version: v11.20.46
Source PostgreSQL OS: RHEL 7.7
Destination PostgreSQL OS: RHEL 7.7
Source PostgreSQL version: 12.4
Destination PostgreSQL version: 12.6
We are trying to out-of-place PostgreSQL individual db restore with our customer but we takes error;
Error Code: u94:19]
Description: PostgreSQL Database: a~ Command failed pg_ctl: could not start server Examine the log output. ~] Restore Failed with PostgreSQL Error: r~Starting of the aux server failed:: ~].
Source: <destination_server>, Process: POSTGRESBLKRESTORE
POSTGRESBLKRESTORE output;
28213 6e35 06/10 13:50:47 325986 CvProcess::system() - chown -R postgres /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170; su - postgres -c "chmod -R 700 /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170"; su - postgres -c "/usr/pgsql-12/bin/pg_ctl start -w -t 43200 -D /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170 -l /opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/325986/pg_logfile_325986 -o --port=50792"
28213 6e35 06/10 13:50:56 325986 CvProcess::system() - Command completed with rc=1
28213 6e35 06/10 13:50:56 325986 PostgresBlockRestore::CreateNewPostGresServer() - Command failed chown -R postgres /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170; su - postgres -c "chmod -R 700 /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170"; su - postgres -c "/usr/pgsql-12/bin/pg_ctl start -w -t 43200 -D /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170 -l /opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/325986/pg_logfile_325986 -o --port=50792"]
28213 6e35 06/10 13:50:56 325986 PostgresBlockRestore::CreateNewPostGresServer() - system out = waiting for server to start........... stopped waiting
err = pg_ctl: could not start server
Examine the log output.
ret = 1
28213 6e35 06/10 13:50:56 325986 PostgresBlockRestore::CreateNewPostGresServer() - Command failed mchown -R postgres /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170; su - postgres -c "chmod -R 700 /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170"; su - postgres -c "/usr/pgsql-12/bin/pg_ctl start -w -t 43200 -D /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_632602_18_18170 -l /opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/325986/pg_logfile_325986 -o --port=50792"]
28213 6e35 06/10 13:50:56 325986 PostgresBlockRestore::CreateNewPostGresServer() - system out = waiting for server to start........... stopped waiting
err = pg_ctl: could not start server
Examine the log output.
ret = 1
/opt/commvault/iDataAgent/jobResults/CV_JobResults/2/0/325986/pg_logfile_325986 output;
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:49.552 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: starting PostgreSQL 12.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:49.553 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: listening on IPv4 address "0.0.0.0", port 50792
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:49.553 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: listening on IPv6 address "::", port 50792
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:49.553 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.50792"
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:49.588 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: listening on Unix socket "/tmp/.s.PGSQL.50792"
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:52.103 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: redirecting log output to logging collector process
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:52.103 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >HINT: Future log output will appear in directory "/Log/log".
/Log/log/postgresql-10.log output;
user=,db=,app=,client=,user=,process=31695,time=2021-06-10 13:50:52.108 +03, commandtag=, statement=00000, session=60c1ee8c.7bcf, sessionstart=2021-06-10 13:50:52 +03, transactionid=0 >LOG: database system was interrupted; last known up at 2021-06-10 11:37:14 +03
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000037.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000038.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000039.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003A.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003B.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003C.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003D.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003E.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003F.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000040.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000041.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000042.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000043.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000044.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000045.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000046.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000047.history’: No such file or directory
user=,db=,app=,client=,user=,process=31695,time=2021-06-10 13:50:56.368 +03, commandtag=, statement=00000, session=60c1ee8c.7bcf, sessionstart=2021-06-10 13:50:52 +03, transactionid=0 >LOG: starting archive recovery
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/00000046.history’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000460000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000450000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000440000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000430000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000420000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000410000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000400000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003F0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003E0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003D0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003C0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003B0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/0000003A0000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000390000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000380000018D0000006A’: No such file or directory
cp: cannot stat ‘/Restores/commvault/staging//325986//WAL/000000370000018D0000006A’: No such file or directory
user=,db=,app=,client=,user=,process=31695,time=2021-06-10 13:50:56.578 +03, commandtag=, statement=00000, session=60c1ee8c.7bcf, sessionstart=2021-06-10 13:50:52 +03, transactionid=0 >LOG: restored log file "000000360000018D0000006A" from archive
user=,db=,app=,client=,user=,process=31695,time=2021-06-10 13:50:56.616 +03, commandtag=, statement=XX000, session=60c1ee8c.7bcf, sessionstart=2021-06-10 13:50:52 +03, transactionid=0 >FATAL: requested timeline 70 is not a child of this server's history
user=,db=,app=,client=,user=,process=31695,time=2021-06-10 13:50:56.616 +03, commandtag=, statement=XX000, session=60c1ee8c.7bcf, sessionstart=2021-06-10 13:50:52 +03, transactionid=0 >DETAIL: Latest checkpoint is at 18D/6A000110 on timeline 54, but in the history of the requested timeline, the server forked off from that timeline at 183/DF000000.
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:56.619 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: startup process (PID 31695) exited with exit code 1
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:56.619 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: aborting startup due to startup process failure
user=,db=,app=,client=,user=,process=31657,time=2021-06-10 13:50:56.647 +03, commandtag=, statement=00000, session=60c1ee89.7ba9, sessionstart=2021-06-10 13:50:49 +03, transactionid=0 >LOG: database system is shut down
They was using script for delete old logs. Script is;
#!/bin/bash
find /dbbackups/PGBackups/$HOSTNAME/xlogarchive/ -mtime +5 -exec mv {} /dbbackups/PGBackups/$HOSTNAME/silinecekler/xlogarchive/ \;
rm -rf /dbbackups/PGBackups/$HOSTNAME/silinecekler/xlogarchive/*
I think this script deleted the history files so we are taking error now. I reinstalled Commvault agent from source and destination PostgreSQL servers and I took full backups and log backups. Why am I still seeing this error? How do I solve this, do we must put history file there?
I would be appreciated for your helps.
BR,
İbrahim