Log4j Vulnerability - Please Post All Questions Here

Summary
 

The following thread describes the potential exposure to the Apache Log4j vulnerability and steps to update Commvault software.

It has been confirmed that a small subset of Commvault agents are impacted. 

Update as of 1st February: Maintenance Release to bring Log4j version to 2.17.1 across Commvault software platform has been released. This release includes the upgrade of components that previously used Log4j 1.x. 

Update as of 20th December: Maintenance Release including relevant hotfixes now available for Commvault software, see section “Maintenance Releases”. Please note customers who have already applied hotfixes for Log4j 2.16, do not need to install.

For customers with Commvault Hyperscale X and Distributed storage, please see section new Community article here: 

Apache Log4j information

• CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. From log4j 2.15.0, this behavior has been disabled by default
• CVE-2021-45046: Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default
• CVE-2021-45105: Apache Log4j2 did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted. 

• CVE-2021-4104: JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data.

• CVE-2021-44832: Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.

Note: Log4j version 1.x is NOT affected.

There’s a great blog article that covers the potential impact.

Updates:

CVE-2021-45105: Commvault can confirm that the affected Log4j2 function is NOT leveraged by Commvault software and thus there is no immediate need to update Commvault to use Log4j 2.17. 

CVE-2021-4104: The Commvault software does not use the JMSAppender module and, therefore, the vulnerability about log4j 1.x versions does not affect any Commvault products.

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

Identifying and Updating Commvault

Exposure:

Note: check FAQ at the bottom of this post for specific version questions.

The exposure impacts the below Commvault product features:

• Cloud Apps package
• Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

• Commvault Distributed Storage

• Commvault Hyperscale X

Identifying affected servers using the Commvault Log4j report

Please use the below direct link to download the Commvault Log4J Affected Servers report then follow step 4 to import and run

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789  

Alternatively, follow steps 1-3 to manually download the Commvault report:

1. Log into cloud.commvault.com and click the Software Store tile icon 
2. Search the Store for Log4j and click the Download button for the “Log4J affected servers” report 
3. Log into Command Center and navigate to Reports
4. At the top right of the Reports page, click Actions and Import report. Proceed to select the downloaded report file to import into Command Center. 
5. Now you can run the report.

This report will show you any servers with Cloud Apps, SQL Server, and Oracle Database packages installed that may be affected by Log4j vulnerability.
Note: If the resulting report shows No Data to Display, then there are no affected clients in this CommCell

The easiest course of action is to upgrade all servers listed in this report (Oracle, Cloud Apps, and SQL) – that would be the recommendation. However, at a minimum, servers with Database Archiving, Data Masking, or Extent based backups (SQL table level restore) features enabled should have highest priority as the vulnerable log4j package is actively used, while otherwise the packages are dormant.

Maintenance Releases

The table below outlines the specific Maintenance Releases that will both address Log4j 2.x vulnerabilities as well as update Log4j 1.x components to 2.17.1 (the latest release of Log4j)

Note: If the previous Log4j 2.16 hotfixes has already been applied, then this latest Maintenance Release is optional 

How to deploy Maintenance Release:

1. First perform a disaster recovery backup using steps HERE .
2. OPTIONAL: create a server group containing all the affected servers using instructions HERE.  This can make it easier to select servers for the upgrade process.
3. Go to documentation to find the list of updates: https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html
4. Download the maintenance pack for the version the CommServe is running on. 
   
   If you do not know the CommServe version, in Command Center search for About and click the About search result to bring up the version popup. 
5. Extract the Maintenance Pack
6. Follow instructions HERE to copy the software packages to the cache using Command Center.  
7. Proceed to Install updates following instructions HERE .  You can update only the affected clients to avoid the CommServe services stopping, however it is recommended to update the CommServe and all affected servers as shown on the report for completeness.
8. Once completed re-rerun the report to show that the servers have the appropriate fixes

Note: For instructions on how to apply Log4j 2.16 hotfixes on older Maintenance Release, please see FAQ

See Commvault Online Documentation for additional information:

https://documentation.commvault.com/11.25/essential/146231_security_vulnerability_and_reporting.html

HyperScale X and CDS (Hedvig)

For all detailed information on how to update HyperScale X and Commvault Distributed Storage (CDS) to address Log4j vulnerabilities, please see article here:

Log4j FAQ 

Q:   There is a new vulnerability in 2.15 is Commvault addressing this? 

A:   The LOG4J 2.15 version (GA Dec. 06, 2021) disabled the essential exploit functions by default was released last week on Dec 6, 2021. This was considered the market-acceptable, non-vulnerable upgrade package up to today.  

 The Apache organization released a new version, 2.16, on Monday, Dec.13, 2021, which physically removes the vulnerable functions. 

 This evening, the security groups issued a new vulnerability CVE-2021-45046 targeting concerns with the 2.15 version and recommending the shift to 2.16.   

  This significant change affects all client remediation methods, requiring an upgrade to version 2.16.  Log4j 2.16 hotfixes have now been released, please see table above

Q: When will new hotfixes be available for 2.16 log4j? 

A: Log4j 2.16 hotfixes have now been released, please see table above

Q:  I've noticed older 1.x versions of log4j being used in the platform.  Are these vulnerable? 

A:   We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action.  

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”.  That version is not vulnerable to the current respective CVEs, but it would clear up the scanning concerns.

Q:  I noticed HyperScale 1.5 is using end of life versions of Log4j.  Is this being resolved? 

A:  The 1.x versions of log4j bundled with HyperScale 1.5 are maintained and supported by Redhat.  These versions are not affected by this CVE. 

Q: Are older versions like v10 and v9 affected?

A: These versions are not affected

Q: Why are some updates showing skipped during Copy to Cache?

A: These are updates for Operating Systems your CommCell does not have.  It’s more informational than error related.

Q: Why does the report show No Records Available or No Items to Display?

A: This means there are no affected clients in this CommCell

Q: What order should I apply the updates?

A: The Maintenance Release needs to be installed first, then the Hotfix Pack.  The best option is to use Copy to Cache, followed by pushing the updates out from the GUI as per the instructions.  This will ensure everything is applied as needed in the correct order.

Q: Can I remove versions manually?

A: No, removing anything manually will potentially cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

Q: Is Anti-virus a concern?

A: It is possible that an AV service may lock the affected files out of concern and cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

Q: How do I download Maintenance Release using CommCell Console?

A: Please follow Commvault Online Documentation steps below

https://documentation.commvault.com/11.25/expert/2705_downloading_commvault_software_using_commcell_console.html

Q: Is Metallic vulnerable to the vulnerability?

A: We have found that the Log4j vulnerability has no impact on Metallic or the security and privacy of your data backups. Metallic does not use the impacted libraries as per the Apache Log4j advisory.

We will continue to proactively monitor and provide any further updates, while customers with questions can reach out to Metallic.io/support.

Q: Log4j scanner is still picking up DbArchiveEngine.jar as potentially vulnerable?

A: Some Log4j scanners are actually incorrectly picking up DbArchiveEngine.jar as potentially vulnerable when in fact it is already patched. This is because the scanner was unable to determine the version of Log4j used and ending up marking it as “potentially vulnerable”. Please note that if you have patched Commvault clients with either the 2.16 hotfix or the latest Maintenance Release, then this DbArchiveEngine.jar binary is also patched and will not have the Log4j 0-day vulnerability. 

Q: I have updated to latest Maintenance Release but Log4j Affected Servers report is still showing my clients as not fixed?

A: There has been a new Log4j Affected Servers report released on December 22nd that has updated the checks to correctly report fixed for clients on the latest Maintenance Release. This new report is version 1.1.2.3 whereas the previous report is 1.1.2.2.

Q: How do I apply the Log4j hotfixes if I am already on the older minimum required Maintenance Release?

A: Please follow below steps:

1. Ensure the Commserve and affected clients are on the minimum required Maintenance Release pack. 
   1. If not, please download and install using the CommCell Console 
   2. Alternatively, you may download the minimum required Maintenance Release from the links in the table below
2. Download the Log4J-Fix pack for your version
3. Unzip the contents of the download
4. Run Copy To Cache and point to the folder created by the unzip to add the new updates to your software cache
5. Push out updates to the clients
6. Verify client status by checking the Log4j Affected Servers report or Client Details report or viewing the client properties

Log4j 2.16 fixes (CVE-2021-44228, CVE-2021-45046)