Issue with Ingesting events, audit, alerts to splunk - TCP connector - Syslog configuration

@alligator -  Did you have any issues setting up the secure communication between the commserve and splunk server? Is there a firewall in place that could be blocking communication?  

Commvault document for configuring syslog. 

https://documentation.commvault.com/2022e/essential/114237_configuring_syslog_server.html

@NVFD411 there is no firewall enabled, also we are not using the secure messaging in our case. 

Yes,we followed the same document  https://documentation.commvault.com/2022e/essential/114237_configuring_syslog_server.html, some how we are  not able to identify the issue.

Please take a look at the EvmgrS and cvd logs for any errors.  Please post if any errors are found.

@NVFD411 

No 😔

I don’t see any errors reported on EvmgrS and cvd logs related to syslog 😓

I couldn’t find anything that relates to a syslog failure like

data sent was rejected by syslog or anything of that sort.

 @alligator 

is this a Linux box and does it have SELinux enabled?

I ask because I faced an issue where SELinux blocked cv to read the log file. Upon further investigation, I discovered that SELinux was blocking the "LogMonitoring" . After adding "LogMonitoring" to the whitelist, the issue was resolved.

***audit.log cut***
type=AVC msg=audit(1679073616.118:7508): avc:  denied  { read } for  pid=13076 comm="LogMonitoring" name="audit.log" dev="dm-6" ino=5392 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:auditd_log_t:s0 tclass=file permissive=0
 

@DanC @NVFD411 

Atlast we were able to find the issue 😓

we were trying to get the data on TCP port , but this would only work with UDP.

after re-configuring the rsyslog with UDP its working, same when we configured UDP port on our splunk , we are able to see the events directly on splunk.

I hope the documentation would be modified accordingly.

@DanC yes there in the documentation its mentioned either UDP or TLS, i missed that first sentence :(

Also while working with support we were given with some sample screenshots on how to configure the ports on splunk, in that it was shown as TCP

so didn’t give much thought to it in the beginning itself :-|

Though we did notice that the UDP lines were commented on the rsyslog.conf file, and it was configured on TCP port 514

i think it might be good if the documentation can be added with a note: that it wont won’t work on TCP

reason being  

“Since the CS is only sending the alerts and not expecting any communications back from the other end, we use UDP as a connectionless protocol, this allow to avoid the ACK required for TCP communications, thus reducing the network load.” updated by support team #230301-212

Thanks a lot for response on this thread @DanC @NVFD411 @JavierB @Manzar Ali @Shafi @Tommaso Mauri 
 

@DanC , i have one more doubt with regards to this, to make a TLS connection, can we just configure the CSR from the commserve (which is on windows) and then generate the pem file and upload it in the syslog configuration in commandcenter.
Is there anything that we would have to do on the splunk server once we enable this as here we don’t have a syslog server , rather it just the splunk server with UDP port configured.

In documentation, it only says this

• To enable secure messaging between the Commserve and the syslog server, obtain the certificate authority file that is used to sign Syslog Server certificate. The certificate authority file should be in .pem format only. Also, perform required configurations in the syslog server to accept encrypted messages from the Commserve.

@alligator 

It's important to note that Splunk has not been certified by Commvault as a syslog server, and may not work in all circumstances. though it can act as a syslog server and  provides two options for receiving syslog data: the preconfigured Splunk syslog server virtual appliance or the Universal Forwarder feature.

Commvault has tested and certified several other syslog servers, including Rsyslog, nsyslog, ArcSight Syslog Server, Kiwi Syslog Server, and syslog-ng.

Since Splunk is not certified by Commvault as a syslog server, there is no guarantee that it will work.

However, the process for configuring syslog TLS should be similar across different syslog servers, including Splunk.

For example, to enable encrypted communication channel using TLS between a client (e.g. Commserve) and a (r)syslog server (e.g. RHEL,Splunk), you can:

#1 Configure the syslog server to support encrypted communication using TLS.

#2 Use certtool (or other certificate management tool) to generate a CA and self-signed client certificates.

#3 Set the certificate files in the syslog server and import the client certificate to the Commvault Command Center.