Hello,
is there anyonone who can tell me which packages have to be installed on the DB linux machine when setting up a postgres SQL DB backup with Commvault? Any documentation could also help.
THX
Solved
POSTGRES SQL backup setup question
- Apprentice
Best answer by Alex Deaconu
The ‘fail to bind’ port is normal when you restart the service, because there is already another cvfwd and cvd hogging 8403 and 8400. In a second terminal, you can ‘netstat -tulnp | grep 84’ or ‘ss -tulnp | grep 84’ to see the bound ports and the processes that are using them. Maybe you can even do something like `watch -n 1 ‘netstat -tulnp | grep 84’` to not have to repeatedly run the command manually.
You do not have to do it, but if you notice other processes binding on 8400 and 8403, that would not be good. I am pretty sure it is just the previous PID of Commvault.
Sometimes the error for failing to bind on the IP address is when /etc/hosts does not contain the loopback entry ‘127.0.0.1 localhost’. Commvault uses ‘localhost’ when binding, and with it absent, you might get weird behavior. Typically you see no connections until localhost is resolved, but sometimes Commvault might manage to bind on some address and have intermittent connectivity.
From the next set of logs, something seems a bit wrong to me.
3188466 30a6f2 03/14 10:53:20 PT:00002 CN:00006 [local:cvd<-COMMSERVE] Completed client connection {a205fec6-ae6e---c04a1f3659eb} to 127.0.0.1:8403
I missed it when you initially configured it, but now it makes more sense.
commvault status
Version = 11.32.29
Media Revision = 1191
Cvd Port Number = 8403
CVD should be 8400, not 8403, and CVFWD should be 8403. I suspect CVFWD fails to bind 8403 because CVD is binding it. I would reinstall Commvault and not configure any custom ports, just leave the port empty when asked for it.
Otherwise, you can edit a few ports manually here and there, with the CV Services stopped, ‘commvault stop’. I would suggest you reinstall, but you can try changing the ports. The reason for reinstalling is that it is possible the CommServe might not get updated with the new registered ports for the client.
grep nCVDPORT /etc/CommVaultRegistry/Galaxy/Instance001/Session/.properties
grep nCVFWD_TUNNEL_PORT4 /etc/CommVaultRegistry/Galaxy/Instance001/Runtime/.properties
grep PORT /etc/CommVaultRegistry/Galaxy/Instance001/Firewall/.properties
grep tunnel_ports /opt/commvault/Base/FwConfig.txt
The nCVDPORT should be 8400 in Session, and nCVFWD_TUNNEL_PORT4 8403 in Runtime. 8403 for tunnel_ports in FwConfig.txt. The PORT in Firewall registry might not show you anything, like 8403, you might only get BOOLEAN values when you grep for PORT in Firewall/.properties.
+4- Vaulter
Hi Ladislav,
For PostgreSQL, you need Commvault File System Core, File System Agent, and PostgreSQL Agent.
https://documentation.commvault.com/v11/essential/getting_started_with_postgresql.html
You will then have two methods of backing up PostgreSQL instances, by using the DumpBasedBackupSet, which uses pg_dump, or by using FileSystemBasedBackupSet, which leverages file system backups of the PostgreSQL Data Directory, and then backup up PG WAL files, which need to be enabled on the DB level.
Both have their own advantages and use cases. Let me know if you have any question on that.
Kind regards,
Alex
- Apprentice
Hello,
thank you for your answer. I am mostly interested in firewall rules. What ports need to be enabled vica versa when setting up this backup? We are using a common proxy, which serves as FW between the commserve and the DB client. I have installed the FS , FS core and postgres agent, the client appeared in commserve but is not ready. Pushing installation also fails that credentials are bad, or there is no connection between the FREL (where are the packages) and the client. Do you have any commands to check if the connection is ok between the involved machines?
Thank you
Ladislav
+4- Vaulter
I assume your network looks like this.
CS and MA → Proxy Gateway ← Client.
The only port open above is 8403 on the Proxy.
https://documentation.commvault.com/2023e/expert/network_gateway_in_perimeter_network.html
Create two groups, Proxied_Clients and Proxies. Then go to topologies, either in the Commcell Console, or Manage → Network → Network Topologies and create a new Network Gateway Topology.
The top group should be Proxied_Clients, the middle group should be Proxies, and the third should be MyCommServe Computer and Media Agents.
When Push Installing via a Remote Software Cache, the remote cache client, in this case I believe your FREL, will need to be able to connect to the target client via SSH for the original FS Core install.
root@frel# /opt/commvault/Base/cvping client 22
This should work. And when you push install to the client, you need to add the client to the Proxied_Clients group during the push install wizard.
Then, you need the following commands to work.
root@client# /opt/commvault/Base/cvping frel 8403
user@CS> ‘C:\Program Files\Commvault\ContentStore\Base\cvping frel 8403
user@MA> ‘C:\Program Files\Commvault\ContentStore\Base\cvping frel 8403
You can view the Port requirements here.
https://documentation.commvault.com/v11/essential/port_requirements_for_commvault.html
For PostgreSQL communication, everything happens locally on the client, on Unix Sockets, which means, if you have any custom pg_hba.conf settings, you need to add a local md5 connection for the CV user.
- Apprentice
Hello
thank you very much. I understand it much better now. I will try to configure it that way. Will let you know.
Ladislav
- Apprentice
UPDATE: We have managed to install the Core package and the client is “ready” in the commserve, but we are unable to push the FS and postgres packages from the commserve. The installation just fails after 10 minutes of attempting, so there is also some trouble..we have to look at it deeper.
+4- Vaulter
Can you transfer the package to the client and install it locally? You can copy from CVInstallDir\SW\CVMedia, the whole Unix folder, to the client.
From there, ‘chmod -R 775 Unix’, ‘cd Unix/’ and ‘./cvpkgadd’. That way you will know the agent is there and is fine.
Not sure why the timeout, typically you get timeouts when the installation process takes too long, or when the CV Agent fails to start after an installation.
You can run ‘commvault status’ to check the packages installed, and ‘commvault list’ to check the PIDs of Commvault processes, which, with the FS Core, FS and Postgres, should be cvd, ClMgrS, cvlaunchd and cvfwd.
- Apprentice
I made a package on FREL for FS core and postgres agent. Copied to the client and installed it. Then I tried to push FS agent but the push hanged.. the strange thing is that the agent from the client side can be listed with commvault list and its there. On the other hand the check readiness fails for the client. I am confused now as form the client side everything looks good, processes are up, but from the commserve I am unable to restart its services or check its readiness. From the commserve I can see installed softwares are FS core and postgres. It makes no sense at all. I will try to setup the postgres backup with the DB team and we will see if it will work or not.
+4- Vaulter
If the client does not show ready, it either will fail to backup, or there is a cache somewhere that shows you the wrong status.
You can run a backup and see if that works. Given that you are unable to see the FS iDA in the client properties, I think the client really is offline. You might want to open a case to troubleshoot, if that is the issue.
- Apprentice
Yes, its strange, but from the client it looks ok.
commvault status
Version = 11.32.29
Media Revision = 1191
Cvd Port Number = 8403
[ Package ]
1002/CVGxBase = File System Core
1101/CVGxIDA = File System
1209/CVGxPostGres = PostGres
[ Physical Machine/Cluster Groups ]
Name = dxy1
- Client Hostname = dxy1.com
- Job Results Directory = /opt/commvault/iDataAgent/jobResults
[root@dxy1]# ps -ef | grep comm
root 1623028 1 0 Mar12 ? 00:00:02 /opt/commvault/Base/cvlaunchd
root 1623029 1 0 Mar12 ? 00:03:06 /opt/commvault/Base/cvd
root 1623030 1 0 Mar12 ? 00:00:07 /opt/commvault/Base/cvfwd
root 1623032 1 0 Mar12 ? 00:00:31 /opt/commvault/Base/ClMgrS
+4- Vaulter
That looks fine on the client, but on the CommServe, is it not reported as it should?
If you are using the CommCell Java Console, on the properties of the client, under the Version tab → Update Status, you would see Base0 Module, File System Core, File System and PostgreSQL modules.
In the Command Center, when you go to either Manage → Servers, or Protect → File Servers, navigate to the client → Add Software. There you would only see File System and PostgreSQL, which is normal.
On the client, you can check the Log Location, by default /var/log/commvault/Log_Files, for cvfwd.log if you see ‘CS if OFFLINE’. You can also check ApplicationMgr.log, and if you do not see something containing ‘getSessionPoolInstance’ and the CommServe Client Name, then the client is most likely not communicating with the CommServe.
- Apprentice
I have checked from Comcell java and its showing all 4 modules you listed.
[redacted data]
+4- Vaulter
You might not want to post your UUIDs publicly, maybe remove your comment. I saw what I needed to see.
Please make sure that /opt/commvault/Base/FwConfig.txt has the same rules as in the Commvault Java Console under client Properties → Network → Network Summary tab. Just copy and paste inside that file, replacing all the contents of FwConfig.txt with the ones from the Console.
Edit the client’s Instance001/CommServe/.properties and change to the CommServe Host Name. Using the Proxy’s Host Name during registration is fine.
Make sure that the CS Host Name is resolvable in DNS, even if the client will not attempt a direct connection to the CS. If it is not, add it to /etc/hosts, both the normal host name and the Fully Qualified Domain Name.
I saw that CommServe eventually got ONLINE on the client, meaning it should now have connectivity, but just as a precaution, ensure these steps are followed.
- Apprentice
Hi, yes I missed that. I couldnt find the delete button or anything like that :/
- Apprentice
Support helped me to remove the message :) I will let you know the output from your answer. Thank you so much for your help so far. :)
- Apprentice
Hello
I did as you wrote. Copied the FW congig from the comcell console and added the commserve to /etc/hosts. Also edited the properties to use the commserve hostname. In the log I can see this error now in the fw.log on the client.
3179128 308278 03/14 10:40:27 ######## ########
3179128 308278 03/14 10:40:27 ######## ######## ERROR: cvfwd_process_tunnel_ports(): Unable to bind IPv4 socket for listening to tunnel connections on port 8403: Address already in use
3179128 308278 03/14 10:40:27 ######## ######## ERROR: cvfwd_process_tunnel_ports(): Will retry in 60s
3179128 308278 03/14 10:40:33 PT:00002 CN:00012 [local->COMMSERV:8400] Successfully completed client connection {06bdf3c2-69e4-be07-ee75-} via proxy COMWISP
3179128 308278 03/14 10:40:33 PT:00001 CN:00013 [local->COMMSERV:8400] Successfully completed client connection {049c81b8-8570-6a1e-6dc5-} via proxy COMWISP2
3179128 308278 03/14 10:40:45 ######## ######## Reporting the client ONLINE
3179128 308278 03/14 10:40:45 ######## ######## CS is ONLINE
3179128 308278 03/14 10:41:28 ######## ######## Trying to bind to tunnel ports again
3179128 308278 03/14 10:41:28 ######## ######## ERROR: cvfwd_process_tunnel_ports(): Unable to bind IPv4 socket for listening to tunnel connections on port 8403: Address already in use
3179128 308278 03/14 10:41:28 ######## ######## ERROR: cvfwd_process_tunnel_ports(): Will retry in 60s
3179128 308278 03/14 10:42:27 ######## ######## Trying to bind to tunnel ports again
3179128 308278 03/14 10:42:27 ######## ######## ERROR: cvfwd_process_tunnel_ports(): Unable to bind IPv4 socket for listening to tunnel connections on port 8403: Address already in use
- Apprentice
THE client was ready for 5 minutes. I have restarted it services and got this in the log and got Not Ready again. After 15 minutes its still not ready and I am getting the connection refused error below..
3188466 30a6f2 03/14 10:52:28 PT:00001 ######## [COMWISP] Initialized new PERSISTENT tunnel from "POSTGRES_client" to "COMWISP" via (100.64..123, 100.64..37)
3188466 30a6f2 03/14 10:52:28 PT:00001 CN:00005 [local->COMMSERVE:AppManager netw] Successfully completed client connection {eadcc138-e78b---9d0b5d042b2b} via proxy COMWISP
3188466 30a6f2 03/14 10:52:56 ######## ######## Detected a change in IP configuration. Sending KEEP_ALIVEs through all tunnels
3188466 30a6f2 03/14 10:52:56 ######## ######## Detected a change in IP configuration. Re-reading config files
3188466 30a6f2 03/14 10:52:56 PT:00002 ######## [COMWISP2] Initialized new PERSISTENT tunnel from "POSTGRES_client" to "COMWISP2" via (100.64.2.123, 100.64.50.36)
3188466 30a6f2 03/14 10:53:20 PT:00002 CN:00006 [local:cvd<-COMMSERVE] ERROR: cvfwd_connect_to_afunix_client1(): Failed to connect() fd=14 to /opt/commvault/Base/Sockets/cvd: Connection refused
3188466 30a6f2 03/14 10:53:20 PT:00002 CN:00006 [local:cvd<-COMMSERVE] Completed client connection {a205fec6-ae6e---c04a1f3659eb} to 127.0.0.1:8403
3188466 30a6f2 03/14 10:53:20 ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x00
3188466 30a6f2 03/14 10:53:20 DT:00004 ######## ERROR: cvfwd_dispatch_one_tunnel(): Invalid command received. Drop the tunnel
3188466 30a6f2 03/14 10:53:20 ######## ######## 00 00 00 38 00 00 00 10 00 00 00 10 00 00 00 0f |...8............|
3188466 30a6f2 03/14 10:53:20 ######## ######## 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
3188466 30a6f2 03/14 10:53:20 ######## ######## 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 |................|
3188466 30a6f2 03/14 10:53:20 ######## ######## 00 00 00 00 00 00 00 00 |........|
3188466 30a6f2 03/14 10:53:20 PT:00002 CN:00006 [local:cvd<-COMMSERVE] ERROR: Client connection failed on POSTGRES_client: Failed to connect() fd=14 to /opt/commvault/Base/Sockets/cvd: Connection refused
3188466 30a6f2 03/14 10:53:25 ######## ######## Reporting the client ONLINE
3188466 30a6f2 03/14 10:53:25 ######## ######## CS is ONLINE
+4- Vaulter
- Answer
The ‘fail to bind’ port is normal when you restart the service, because there is already another cvfwd and cvd hogging 8403 and 8400. In a second terminal, you can ‘netstat -tulnp | grep 84’ or ‘ss -tulnp | grep 84’ to see the bound ports and the processes that are using them. Maybe you can even do something like `watch -n 1 ‘netstat -tulnp | grep 84’` to not have to repeatedly run the command manually.
You do not have to do it, but if you notice other processes binding on 8400 and 8403, that would not be good. I am pretty sure it is just the previous PID of Commvault.
Sometimes the error for failing to bind on the IP address is when /etc/hosts does not contain the loopback entry ‘127.0.0.1 localhost’. Commvault uses ‘localhost’ when binding, and with it absent, you might get weird behavior. Typically you see no connections until localhost is resolved, but sometimes Commvault might manage to bind on some address and have intermittent connectivity.
From the next set of logs, something seems a bit wrong to me.
3188466 30a6f2 03/14 10:53:20 PT:00002 CN:00006 [local:cvd<-COMMSERVE] Completed client connection {a205fec6-ae6e---c04a1f3659eb} to 127.0.0.1:8403
I missed it when you initially configured it, but now it makes more sense.
commvault status
Version = 11.32.29
Media Revision = 1191
Cvd Port Number = 8403
CVD should be 8400, not 8403, and CVFWD should be 8403. I suspect CVFWD fails to bind 8403 because CVD is binding it. I would reinstall Commvault and not configure any custom ports, just leave the port empty when asked for it.
Otherwise, you can edit a few ports manually here and there, with the CV Services stopped, ‘commvault stop’. I would suggest you reinstall, but you can try changing the ports. The reason for reinstalling is that it is possible the CommServe might not get updated with the new registered ports for the client.
grep nCVDPORT /etc/CommVaultRegistry/Galaxy/Instance001/Session/.properties
grep nCVFWD_TUNNEL_PORT4 /etc/CommVaultRegistry/Galaxy/Instance001/Runtime/.properties
grep PORT /etc/CommVaultRegistry/Galaxy/Instance001/Firewall/.properties
grep tunnel_ports /opt/commvault/Base/FwConfig.txt
The nCVDPORT should be 8400 in Session, and nCVFWD_TUNNEL_PORT4 8403 in Runtime. 8403 for tunnel_ports in FwConfig.txt. The PORT in Firewall registry might not show you anything, like 8403, you might only get BOOLEAN values when you grep for PORT in Firewall/.properties.
- Apprentice
Hello,
after editing those ports the client is ready.
“Successfully completed client connection:
I have created a package with the right port number. We have made a mistake when creating a custom package, where we added the CVD port as 8403.
Thank you for helping with this , I really appreciate it :)
Ladislav
Readiverse Academy Certifications
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.