Information extraction from log files for SIEM/SOC

@Onno van den Berg,

Have you checked out this documentation in of our BOL;

https://documentation.commvault.com/11.26/expert/5639_log_monitoring.html

https://documentation.commvault.com/11.26/expert/5853_system_monitoring.html

Hi @HolowEd,

This is not what I'm looking for because I'm related to the actual information that is logged in the current log files. In additional most companies using SOC/SIEM will not use Commvault Log Monitoring for that but will us a specific solution for it or leverage solutions like Splunk or Azure Sentinel so their security departments can scan all access logs for specific threats. 

The log monitoring feature I see being very helpful for troubleshooting purposes. 

So to be clear I'm looking for a guide that explains how to retrieve the information around the use cases from the Commvault logs.

Onno

@Onno van den Berg 

I can check with our products team to see if they have anything that they can supply that can be used as a guide for setting this up.

@Onno van den Berg

If Splunk is your SIEM of choice, we have a plugin in the splunkbase that will collect log data.  At that point you can query the log data for whatever you want.

You can also send any triggered Commvault event  to a SIEM via Syslog or Webhook (via event alert).  But in cases where there are specific events in the logs that you want to monitor for; create an Event Raiser Log Monitoring policy.  This is a very cool feature - it allows you to generate your own events and alerts when certain log conditions are detected based on simple search criteria or advanced regular expressions.  Event raiser does not require an index server.  When the condition is detected, it will generate an event and/or alert that you can send off to your event platform using Syslog or webhook.  Additionally this feature is not just limited to Commvault logs - you can also monitor external system logs and generate events in Commvault.  Its a very powerful feature, and provides a ton of flexibility, but its a great way to monitor for log conditions without having to send all the logs to your event system.  At the moment you can only create LM policies in java gui, but we are working on bringing it over to Command Center.

Here are some demo videos:

• Splunk plugin
• Webhook alerts
• Syslog
• Event Raiser (monitor linux audit log for ransomware activity example)

Here are examples for your criteria:

• ​​​​​Activity via REST API, Command Center, CommCell console, Apps, etc.

You would want to monitor the webserver.log

• Activity towards (and through) network gateways.

You can monitor connections in CVD and CVFWD

• Activity to/from functionality with external components like external REST API's, Key Vaults. 

This really depends on the specifics.  But if you are using a KMIP KMS you can monitor the KMIPClient.log for example.

• Communication between client computers.

You can monitor connections in CVD and CVFWD

• Data retrieval e.g. who is retrieving data from the platform via a restore, download functionality in Command Center. etc. 

Actually we audit this in the audit trail by default in 1125+ You can also make an alert for this, and send it via webhook or syslog.

Thanks for the extensive response @DMCVault !!

Splunk plugin
The Splunk plugin looks nice and I see a benefit for a large amount of customers, however I'm not allowed to use it like this because I'm only targeting the backend infrastructure not the client computers of our customers. In additional we have automation in place to distribute the Splunk forwarding agent. Only thing I'm interested are the dashboard. I'll see if I can export them after installing the plugin on our Splunk island.

Syslog
If I recall correctly syslog only gathers and forwards the logs of the CommServe, right? One comment on it as I have been playing with it, there is no delete/remove configuration button on the syslog page. From a UX and consistency point of view it would be nice to add it.

Webhook alerts
This is especially nice to send out alerts to Slack/Teams. Maybe also to have the data ingested for SOC/Siem but it depends on the use-case because it doesn't send lower level information like for example network communication attempts.

Event Raiser
I have been playing around with it for some time a few years ago. I do not see a lot of new development around it anymore but it definitely helps-out during troubleshooting. I would consider even configuring and turning it on by default and have some gxtail magic being added to it and make it embedded in Command Center. Unfortunalty for SOC/SIEM it is less interesting. 

As for the overview on logs files:

Activity via REST API, Command Center, CommCell console, Apps, etc.
You would want to monitor the webserver.log
Thanks! Does it log specific codes which can be used to filter the webserver.log for these kind of events?

Activity towards (and through) network gateways.
You can monitor connections in CVD and CVFWD
Thanks! Does it log specific codes which can be used as a filter?

Activity to/from functionality with external components like external REST API's, Key Vaults. 
This really depends on the specifics.  But if you are using a KMIP KMS you can monitor the KMIPClient.log for example.
Ok!

Communication between client computers.
You can monitor connections in CVD and CVFWD
Thanks! Does it log specific codes which can be used as a filter?

Data retrieval e.g. who is retrieving data from the platform via a restore, download functionality in Command Center. etc. 
Actually we audit this in the audit trail by default in 1125+ You can also make an alert for this, and send it via webhook or syslog.
Can you point me to the correct log file? I have been testing but was not really sure which log file captures these events.