I couldn't install Commserve on Linux

Thanks @Mike London UK. I’ll be putting together a KB article for this.

I believe the cause here is the creation of the filesystems for Commvault. By default they have the unlabeled_t type which prevented systemd from interacting with their contents, among other problems most likely.

New directories automatically inherit the SELinux context of their parent, but that doesn’t apply to mount points. We saw in /etc/selinux/targeted/contexts/files/file_contexts that everything under /opt would get context system_u:object_r:usr_t:s0, but that needs to be applied with restorecon.

Hi @Carl Manzi yes doing the restorecon recursively on the mounted files systems works. I did check the mount points and as expected they were correct (as created under root file system). A better way forward than the mount options I think but both methods probably need to go into the documentation.

Hi Carl I’ll try that and report back later, thanks.

@Mike London UK, could you try one more thing?

Revert to snapshot again and then run restorecon on the directories:

restorecon -Rv /opt/commvault /cvlt

Now retry install and see if it behaves correctly.

Thanks Carl, I wound VM back to snapshot and modified fstab and installed CV. All services up and running immediately!

The very first line from /etc/selinux/targeted/contexts/files/file_contexts is:

/opt/.*	system_u:object_r:usr_t:s0

I don’t see any more specific rules that would match /opt/commvault and provide different defaults. There’s also no context options in this fstab. They would look like this example from RHEL docs:

/dev/sda2   /test   xfs   context="system_u:object_r:samba_share_t:s0"   0 0

I have a suspicion that the mountpoint is playing a part in this but I’m not sure what that is yet. What I would suggest for the moment is to update your fstab file like this:

/dev/cvlt/cvbin	/opt/commvault	  xfs defaults,defcontext="unconfined_u:object_r:usr_t:s0" 0 0
/dev/cvlt/csdb	/cvlt/commvaultDB xfs defaults,defcontext="unconfined_u:object_r:usr_t:s0" 0 0
/dev/cvlt/ddb	/cvlt/ddb	      xfs defaults,defcontext="unconfined_u:object_r:usr_t:s0" 0 0
/dev/cvlt/idxch	/cvlt/ic	      xfs defaults,defcontext="unconfined_u:object_r:usr_t:s0" 0 0

That should ensure that anything created in those directories without a label gets the proper labeling. I’m going to discuss this internally and see if we can piece together the details.

Yes /opt/commvault has the unlabeled_t string as well. Output before running restorecon command is attached. Running restorecon on /opt/commvault allows all the services to start :-)

There are context entries in /etc/fstab, also attached for reference as I have other mount paths (for CS DB, and IndexCache.

Looks like we’re getting there, thanks for your help here.

You’re right I missed the different type label (usr_t vs unlabeled_t). Does /opt/commvault have the unlabeled_t type as well?

ls -ldZ /opt/commvault

If so, try running restorecon -R /opt/commvault. See if services start ok after that.

Does /etc/fstab specify a context or defcontext option for this mount? I’d also be curious to see the results of this command:

grep -rE '^/opt' /etc/selinux/targeted/contexts/

I have something extra for /opt/commvault/Base and uname -r differs:

[root@lnxcs02 Desktop]# ls -lZ /opt/commvault/Base
lrwxrwxrwx. 1 root commvault unconfined_u:object_r:unlabeled_t:s0 6 Nov 29 16:09 /opt/commvault/Base -> Base64
[root@lnxcs02 Desktop]# uname -r
4.18.0-348.2.1.el8_5.x86_64
[root@lnxcs02 Desktop]# 

I do have a mounted file system at /opt/commvault. Could that need some treatment?

@Mike London UK, everything I see in that output looks like the system is configured correctly. I have a RHEL8.4 CS running in the lab here and everything matches up.

systemd labels match yours:

# ps -fZ 1
LABEL                           UID          PID    PPID  C STIME TTY      STAT   TIME CMD
system_u:system_r:init_t:s0     root           1       0  0 Nov24 ?        Ss     0:37 /usr/lib/systemd/systemd

Labeling on the Base symlink matches:

# ls -lZ /opt/commvault/Base
lrwxrwxrwx. 1 root commvault unconfined_u:object_r:usr_t:s0 6 Sep 28 15:53 /opt/commvault/Base -> Base64

Can we compare kernel and SELinux versions?

# uname -r
4.18.0-305.17.1.el8_4.x86_64

# rpm -q selinux-policy selinux-policy-targeted
selinux-policy-3.14.3-67.el8_4.1.noarch
selinux-policy-targeted-3.14.3-67.el8_4.1.noarch

@Carl Manzi  not much but has two of this alert “sealert -l 8e151386-961c-4345-9c8f-6d0a2d000183”

So, output of that is attached as quite long.

@Mike London UK 

Let’s see if we can get more detail. Try running the following to see SELinux denial entries in the system journal:

journalctl -g 'SELinux is preventing'

There will probably be a number of entries but look for those that appear toward the bottom, around 16:28 server time. Part of those messages should include something like this which we can run to get a detailed report of the issue:

For complete SELinux messages run: sealert -l 97a1c0df-81ed-4c08-ba27-41c5067b713b

Run those sealert commands as shown on your system -- they will have a different GUID than shown above -- and let’s see what they tell us.

@Carl Manzi  the output is:

----
type=AVC msg=audit(29/11/21 16:28:15.046:320) : avc:  denied  { read } for  pid=1 comm=systemd name=Base dev="dm-2" ino=4136693 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=lnk_file permissive=0
----
type=PROCTITLE msg=audit(29/11/21 16:28:15.047:321) : proctitle=(Galaxy)
type=SYSCALL msg=audit(29/11/21 16:28:15.047:321) : arch=x86_64 syscall=execve success=no exit=EACCES(Permission denied) a0=0x5568d21bd2c0 a1=0x5568d1ff2bd0 a2=0x5568d1ff0450 a3=0x7f7b37559bc0 items=0 ppid=1 pid=17655 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=(Galaxy) exe=/usr/lib/systemd/systemd subj=system_u:system_r:init_t:s0 key=(null)
type=AVC msg=audit(29/11/21 16:28:15.047:321) : avc:  denied  { read } for  pid=17655 comm=(Galaxy) name=Base dev="dm-2" ino=4136693 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=lnk_file permissive=0
 

Thanks @Mike London UK. These look like RHEL defaults so no concerns there.

After install but before you relabel, can you run the following command to see what sort of denials are occurring?

ausearch -ts recent -m avc -i

@Carl Manzi here’s the output:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33
(root@lnxcs02 Desktop]# 
Hope that helps.

Hi @Mike London UK ,

Could you run sestatus and share the output?

Email me your document, I will take a peek.   You said this didnt happen on 7.x correct, only 8.4?

@MFasulo I’ve just re-run the install with the same result, “re-label” the system and then Commvault starts. There was no prompt for permissions, maybe not displayed with my install options. I chose 

1. Create new CommCell
2. All in one
3. /opt for install directory
4. /cvlt/ic for index cache
5. /cvlt for web cache, database installation path (/cvlt/commvaultDB is a mounted file system) and disaster recovery path

Happy to share doc with screenshots and other info you’re interested. Is here the best place for that or somewhere separate.?

I don’t recall seeing that screen but I would have just taken the defaults if it was displayed. I’ll re-run the installer and let you know.

When you did the install did you leave the default permission set?  

Have got this installed on Centos 8.4 as well after a strange problem with Commvault not being able to run any services. /var/log/message showed permission errors and mentioned “re-labelling” so I ran “touch /.autorelabel” and rebooted and CV services came online after the boot.

@MFasulo have managed to download the image thanks and that has installed successfully into my Centos 7.9 VM. It did install in my 8.3 VM but nothing runs for some reason but that is something for another day/thread (once I go through it again).

Thanks for all your help.

Hi,

Installation is done.

FYI @MFasulo @Mike Struening 

Hi,

I downloaded “11.25 Linux MediaKit (Early Release)” again and I can see Commserve option now.

Thank you @MFasulo  @Mike Struening  

I’m going to try install Commserve.

This is the software kit needed (it should show 11.25.6)

Store (commvault.com)

I’m in active communication with Mike.