Skip to main content

Log4j Vulnerability - Please Post All Questions Here


Mike Struening
Vaulter
Forum|alt.badge.img+23

Summary
 

The following thread describes the potential exposure to the Apache Log4j vulnerability and steps to update Commvault software.

It has been confirmed that a small subset of Commvault agents are impacted. 

 

Update as of 1st February: Maintenance Release to bring Log4j version to 2.17.1 across Commvault software platform has been released. This release includes the upgrade of components that previously used Log4j 1.x. 

 

Update as of 20th December: Maintenance Release including relevant hotfixes now available for Commvault software, see section “Maintenance Releases”. Please note customers who have already applied hotfixes for Log4j 2.16, do not need to install.

For customers with Commvault Hyperscale X and Distributed storage, please see section new Community article here: 

https://community.commvault.com/technical-blogs-and-articles-39/hyperscale-x-update-procedure-to-address-log4j-vulnerability-2045

 

Apache Log4j information

  • CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. From log4j 2.15.0, this behavior has been disabled by default
  • CVE-2021-45046: Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default
  • CVE-2021-45105: Apache Log4j2 did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted. 
  • CVE-2021-4104: JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data.

  • CVE-2021-44832: Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.

Note: Log4j version 1.x is NOT affected.

There’s a great blog article that covers the potential impact.

 

Updates:

CVE-2021-45105: Commvault can confirm that the affected Log4j2 function is NOT leveraged by Commvault software and thus there is no immediate need to update Commvault to use Log4j 2.17. 

CVE-2021-4104: The Commvault software does not use the JMSAppender module and, therefore, the vulnerability about log4j 1.x versions does not affect any Commvault products.

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

 

Identifying and Updating Commvault

 

Exposure:

Note: check FAQ at the bottom of this post for specific version questions.

The exposure impacts the below Commvault product features:

Cloud Apps package
Oracle agent - Database archiving, data masking, and logical dump backup
Microsoft SQL Server agent - Database archiving, data masking, and table level restore

Commvault Distributed Storage

Commvault Hyperscale X

 

Identifying affected servers using the Commvault Log4j report

 

Please use the below direct link to download the Commvault Log4J Affected Servers report then follow step 4 to import and run

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789  

 

Alternatively, follow steps 1-3 to manually download the Commvault report:

  1. Log into cloud.commvault.com and click the Software Store tile icon 
  2. Search the Store for Log4j and click the Download button for the “Log4J affected servers” report 
  3. Log into Command Center and navigate to Reports
  4. At the top right of the Reports page, click Actions and Import report. Proceed to select the downloaded report file to import into Command Center. 
  5. Now you can run the report.

This report will show you any servers with Cloud Apps, SQL Server, and Oracle Database packages installed that may be affected by Log4j vulnerability.
Note: If the resulting report shows No Data to Display, then there are no affected clients in this CommCell

The easiest course of action is to upgrade all servers listed in this report (Oracle, Cloud Apps, and SQL) – that would be the recommendation. However, at a minimum, servers with Database Archiving, Data Masking, or Extent based backups (SQL table level restore) features enabled should have highest priority as the vulnerable log4j package is actively used, while otherwise the packages are dormant.

 

Maintenance Releases

 

The table below outlines the specific Maintenance Releases that will both address Log4j 2.x vulnerabilities as well as update Log4j 1.x components to 2.17.1 (the latest release of Log4j)

Note: If the previous Log4j 2.16 hotfixes has already been applied, then this latest Maintenance Release is optional 

Feature Release

Maintenance Release

11.26

11.26.21

11.25

11.25.32

11.24

11.24.48

11.23

11.23.47

11.20

11.20.90

SP16

SP16.153

 

How to deploy Maintenance Release:

 

  1. First perform a disaster recovery backup using steps HERE .
  2. OPTIONAL: create a server group containing all the affected servers using instructions HERE.  This can make it easier to select servers for the upgrade process.
  3. Go to documentation to find the list of updates: https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html
  4. Download the maintenance pack for the version the CommServe is running on. 

    If you do not know the CommServe version, in Command Center search for About and click the About search result to bring up the version popup. 
  5. Extract the Maintenance Pack
  6. Follow instructions HERE to copy the software packages to the cache using Command Center.  
  7. Proceed to Install updates following instructions HERE .  You can update only the affected clients to avoid the CommServe services stopping, however it is recommended to update the CommServe and all affected servers as shown on the report for completeness.
  8. Once completed re-rerun the report to show that the servers have the appropriate fixes

Note: For instructions on how to apply Log4j 2.16 hotfixes on older Maintenance Release, please see FAQ

 

See Commvault Online Documentation for additional information:

https://documentation.commvault.com/11.25/essential/146231_security_vulnerability_and_reporting.html

 

HyperScale X and CDS (Hedvig)

 

For all detailed information on how to update HyperScale X and Commvault Distributed Storage (CDS) to address Log4j vulnerabilities, please see article here:

https://community.commvault.com/technical-blogs-and-articles-39/hyperscale-x-update-procedure-to-address-log4j-vulnerability-2045

 

 

Log4j FAQ 

 

Q:   There is a new vulnerability in 2.15 is Commvault addressing this? 

A:   The LOG4J 2.15 version (GA Dec. 06, 2021) disabled the essential exploit functions by default was released last week on Dec 6, 2021. This was considered the market-acceptable, non-vulnerable upgrade package up to today.  

 The Apache organization released a new version, 2.16, on Monday, Dec.13, 2021, which physically removes the vulnerable functions. 

 This evening, the security groups issued a new vulnerability CVE-2021-45046 targeting concerns with the 2.15 version and recommending the shift to 2.16.   

  This significant change affects all client remediation methods, requiring an upgrade to version 2.16.  Log4j 2.16 hotfixes have now been released, please see table above

 

Q: When will new hotfixes be available for 2.16 log4j? 

A: Log4j 2.16 hotfixes have now been released, please see table above

 

Q:  I've noticed older 1.x versions of log4j being used in the platform.  Are these vulnerable? 

A:   We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action.  

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”.  That version is not vulnerable to the current respective CVEs, but it would clear up the scanning concerns.

 

Q:  I noticed HyperScale 1.5 is using end of life versions of Log4j.  Is this being resolved? 

A:  The 1.x versions of log4j bundled with HyperScale 1.5 are maintained and supported by Redhat.  These versions are not affected by this CVE. 

 

Q: Are older versions like v10 and v9 affected?

A: These versions are not affected

 

Q: Why are some updates showing skipped during Copy to Cache?

A: These are updates for Operating Systems your CommCell does not have.  It’s more informational than error related.

 

Q: Why does the report show No Records Available or No Items to Display?

A: This means there are no affected clients in this CommCell

 

Q: What order should I apply the updates?

A: The Maintenance Release needs to be installed first, then the Hotfix Pack.  The best option is to use Copy to Cache, followed by pushing the updates out from the GUI as per the instructions.  This will ensure everything is applied as needed in the correct order.

 

Q: Can I remove versions manually?

A: No, removing anything manually will potentially cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

 

Q: Is Anti-virus a concern?

A: It is possible that an AV service may lock the affected files out of concern and cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

 

Q: How do I download Maintenance Release using CommCell Console?

A: Please follow Commvault Online Documentation steps below

https://documentation.commvault.com/11.25/expert/2705_downloading_commvault_software_using_commcell_console.html

 

Q: Is Metallic vulnerable to the vulnerability?

A: We have found that the Log4j vulnerability has no impact on Metallic or the security and privacy of your data backups. Metallic does not use the impacted libraries as per the Apache Log4j advisory.

We will continue to proactively monitor and provide any further updates, while customers with questions can reach out to Metallic.io/support.

 

Q: Log4j scanner is still picking up DbArchiveEngine.jar as potentially vulnerable?

A: Some Log4j scanners are actually incorrectly picking up DbArchiveEngine.jar as potentially vulnerable when in fact it is already patched. This is because the scanner was unable to determine the version of Log4j used and ending up marking it as “potentially vulnerable”. Please note that if you have patched Commvault clients with either the 2.16 hotfix or the latest Maintenance Release, then this DbArchiveEngine.jar binary is also patched and will not have the Log4j 0-day vulnerability. 

 

Q: I have updated to latest Maintenance Release but Log4j Affected Servers report is still showing my clients as not fixed?

A: There has been a new Log4j Affected Servers report released on December 22nd that has updated the checks to correctly report fixed for clients on the latest Maintenance Release. This new report is version 1.1.2.3 whereas the previous report is 1.1.2.2.

 

Q: How do I apply the Log4j hotfixes if I am already on the older minimum required Maintenance Release?

A: Please follow below steps:

  1. Ensure the Commserve and affected clients are on the minimum required Maintenance Release pack. 
    1. If not, please download and install using the CommCell Console 
    2. Alternatively, you may download the minimum required Maintenance Release from the links in the table below
  2. Download the Log4J-Fix pack for your version
  3. Unzip the contents of the download
  4. Run Copy To Cache and point to the folder created by the unzip to add the new updates to your software cache
  5. Push out updates to the clients
  6. Verify client status by checking the Log4j Affected Servers report or Client Details report or viewing the client properties

Log4j 2.16 fixes (CVE-2021-44228, CVE-2021-45046)

Feature Release

Minimum Required
Maintenance Release

Update Link (includes 2.16 fix)

Installed Windows
Updates

Installed
Unix Updates

11.26

11.26.2

11.26 Log4J-2.16 Fix

1755

1755

11.25

11.25.9

11.25 Log4J-2.16 Fix

2763

2779

2763

2779

11.24

11.24.23

11.24 Log4J-2.16 Fix

4552

4564

4551

4564

11.23

11.23.37

11.23 Log4J-2.16 Fix

4160

4178

4161

4178

11.22

11.22.50

11.22 Log4J-2.16 Fix

3911

3920

3912

3920

11.21

11.21.66

11.21 Log4J-2.16 Fix

3587

3599

3588

3599

11.20

11.20.77

11.20 Log4J-2.16 Fix

4562

4574

4561

4574

SP16

SP16.128

SP16 Log4J-2.16 Fix

2943

2946

2942

2946

344 replies

Forum|alt.badge.img+2
  • Byte
  • 8 replies
  • December 13, 2021

Hey Mike

Hope you are keeping well

Over the last couple of days this has been causing some ripples. By the way, Commvault says it only affects the cloud apps packages / SQL agent / Oracle agent

But our internal vulnerability scan system was able to find this “Log4J” vulnerability on multiple server that didnt have any SQL/oracle agents installed. And know what when checking those servers it turned out to be the commvault folder that had this log4j file 
path : D:\program files\Commvault\ContentStore\Base\DBMinerTool

1. Can you help to know what is “DBminerTool” . What is it being used / installed for ?

2.  Checked some properties of this log4j file and the version is 1.2.17. Is it still vulnerable ?
     This may have been installed in any of the commvaults old MR / FR packages

3. Can we have these files removed from all the agents that had it detected ? Are these file here for a specific purpose or can they be deleted ?

 

 


Forum|alt.badge.img+6
  • Commvault Certified Expert
  • 40 replies
  • December 13, 2021

@wstbackupNo that version is not an issue for this vulnerability.
 

 


Forum|alt.badge.img+8
  • Vaulter
  • 53 replies
  • December 13, 2021

@wstbackup We are upgrading remaining older versions of log4j, but as of our current analysis they can be ignored since they are not affected.


Mike Struening
Vaulter
Forum|alt.badge.img+23

@wstbackup to answer your other questions:

DBMinertool.exe is used for pseudo mounts and browse of tables for table level restore of SQL

Regarding those files, the hotfixes, once applied, should remove any affected files.  As noted above, there are some versions that are not vulnerable.


Forum|alt.badge.img
  • Bit
  • 1 reply
  • December 13, 2021

In what order should the hotfixes contained in the log4j_fix zip file be applied for a Windows environment? 4562 then 4563? In my certain case, its regarding version 11.20.

 

Thanks.


Mike Struening
Vaulter
Forum|alt.badge.img+23
NTCbrad wrote:

In what order should the hotfixes contained in the log4j_fix zip file be applied for a Windows environment? 4562 then 4563? In my certain case, its regarding version 11.20.

 

Thanks.

@NTCbrad , why not use Copy To Cache and push them, out that way?

 

Otherwise, manually install them in numerical order (though I don’t think it makes a difference).


d3nz
Forum|alt.badge.img+1
  • 4 replies
  • December 13, 2021

Hello,

Is this Log4j Vulnerability affect Web Console/Command Center server?


Mike Struening
Vaulter
Forum|alt.badge.img+23
d3nz wrote:

Hello,

Is this Log4j Vulnerability affect Web Console/Command Center server?

Not specifically; only if those the server running the console also has the affected agents installed.


d3nz
Forum|alt.badge.img+1
  • 4 replies
  • December 13, 2021

Thanks Mike,

Seems like there is no agents at server:

[root@commandcenter ~]# find / -name "apache"
/var/lib/selinux/targeted/active/modules/100/apache
/usr/share/selinux/targeted/default/active/modules/100/apache
/opt/commvault/Apache/work/Catalina/localhost/ROOT/org/apache
/opt/commvault/Apache/work/Catalina/localhost/adminconsole/org/apache
/opt/commvault/Apache/work/Catalina/localhost/console/org/apache
/opt/commvault/Apache/work/Catalina/localhost/webconsole/org/apache
/opt/commvault/Apache/work/Catalina/localhost/manager/org/apache
/opt/commvault/Apache/work/Catalina/localhost/global/org/apache
[root@commandcenter ~]# find / -name "Log4j"
[root@commandcenter ~]#


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 13, 2021
d3nz wrote:

Thanks Mike,

Seems like there is no agents at server:

[root@commandcenter ~]# find / -name "apache"
/var/lib/selinux/targeted/active/modules/100/apache
/usr/share/selinux/targeted/default/active/modules/100/apache
/opt/commvault/Apache/work/Catalina/localhost/ROOT/org/apache
/opt/commvault/Apache/work/Catalina/localhost/adminconsole/org/apache
/opt/commvault/Apache/work/Catalina/localhost/console/org/apache
/opt/commvault/Apache/work/Catalina/localhost/webconsole/org/apache
/opt/commvault/Apache/work/Catalina/localhost/manager/org/apache
/opt/commvault/Apache/work/Catalina/localhost/global/org/apache
[root@commandcenter ~]# find / -name "Log4j"
[root@commandcenter ~]#

Web console / command center does use Apache to service web pages however Log4j v2 won’t be there as it’s not a package that would be needed there.


Forum|alt.badge.img
  • Byte
  • 3 replies
  • December 14, 2021

Would it be beneficial to push out the log4j fix to all Oracle and SQL clients (we don’t use the cloud app) as a precautionary action?

I am unsure if Oracle ArchiveLog subclients fall under ‘Archiving’, so would rather be safe than sorry.

 

 

 


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021
Gazza wrote:

Would it be beneficial to push out the log4j fix to all Oracle and SQL clients (we don’t use the cloud app) as a precautionary action?

I am unsure if Oracle ArchiveLog subclients fall under ‘Archiving’, so would rather be safe than sorry.

 

 

 

Absolutely, deploy to your CS and all Oracle and SQL clients. ArchiveLog does not fall under “Archiving” but better to be safe than sorry as well in case your DBA decides to do something :)


Michael Woodward
Commvault Certified Expert
Forum|alt.badge.img+11
  • Commvault Certified Expert
  • 78 replies
  • December 14, 2021

Thanks Mike,

 

Can you clarify with the SQL and Oracle agents are they only vulnerable if the functions listed are used? or are all SQL / Oracle agents vulnerable.

 Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

 

We’re trying to prioritize our response to our customers at present.

 


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021
Michael Woodward wrote:

Thanks Mike,

 

Can you clarify with the SQL and Oracle agents are they only vulnerable if the functions listed are used? or are all SQL / Oracle agents vulnerable.

 Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

 

We’re trying to prioritize our response to our customers at present.

 

@Michael Woodward 

Only when used. So priority would be those that are used, followed by unused. 


Michael Woodward
Commvault Certified Expert
Forum|alt.badge.img+11
  • Commvault Certified Expert
  • 78 replies
  • December 14, 2021
Jordan wrote:
Michael Woodward wrote:

Thanks Mike,

 

Can you clarify with the SQL and Oracle agents are they only vulnerable if the functions listed are used? or are all SQL / Oracle agents vulnerable.

 Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

 

We’re trying to prioritize our response to our customers at present.

 

@Michael Woodward

Only when used. So priority would be those that are used, followed by unused. 

Thanks @Jordan we’ll focus our energies on the cloud apps users (which is the majority) then move onto SQL / Oracle clients  Appreciate the quick reply


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021

No problems at all @Michael Woodward 


Forum|alt.badge.img
  • Byte
  • 3 replies
  • December 14, 2021
Jordan wrote:
Gazza wrote:

Would it be beneficial to push out the log4j fix to all Oracle and SQL clients (we don’t use the cloud app) as a precautionary action?

I am unsure if Oracle ArchiveLog subclients fall under ‘Archiving’, so would rather be safe than sorry.

 

 

 

Absolutely, deploy to your CS and all Oracle and SQL clients. ArchiveLog does not fall under “Archiving” but better to be safe than sorry as well in case your DBA decides to do something :)

Thanks very much @Jordan, I’ll work on a rollout of the patch.

Do you know if this also affects MySQL, PostgresSQL and SAP agents as well? Might have to include these in the rollout based on your recommendation.


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021

@Gazza 

If you use the copy to cache feature and push to clients, the ones not impacted will have nothing to install anyway. But MySQL, PostgeSQL and SAP at not impacted.


Forum|alt.badge.img
  • Byte
  • 3 replies
  • December 14, 2021
Jordan wrote:

@Gazza

If you use the copy to cache feature and push to clients, the ones not impacted will have nothing to install anyway. But MySQL, PostgeSQL and SAP at not impacted.

Well noted, thanks again @Jordan


Forum|alt.badge.img+5
  • Byte
  • 15 replies
  • December 14, 2021

Hello everyone, I’d like some clarification on fixing this vulnerability and the table that is presented as there’s something that’s confusing. Specificailly, does upgrading to the version mentioned in the Minimum Maintenance Release Required automatically install the Log4j fix? 

 

The reason why it’s confusing is in the past, all that was required was to upgrade to the latest MR.

As an example, on this site:

https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html#security-advisories

There are two listed security vulnerabilities. The bottom one: “CV_2021_08_1: Authentication Bypass Vulnerabilities on CVWebService Endpoint”, clearly shows that the resolution is to simply download and install the following maintenance release (or a more recent release), for your Feature Release on the CommServe and Web Server mentioned in the table.

 

But for the Log4j, the resolution seems to indicate that an upgrade Minimum Maintenance Release Required value is NOT a fix in itself, and instead an installation to the link to the Log4J Fix is required as well.


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021

Hi @JSNOPUD 

 

Correct, in this case, you will need both the minimum maintenance release version as well as the log4j hotfix on top.

 

This is due to the log4j fix being released out of band to normal Commvault patch release cycle.

 

These log4j hotfixes will later be rolled into a maintenance release as well per usual software development cycles. 

 

Hope that clears things up for you a bit.

 

Thank you


Forum|alt.badge.img+5
  • Byte
  • 15 replies
  • December 14, 2021
Jordan wrote:

Hi @JSNOPUD 

 

Correct, in this case, you will need both the minimum maintenance release version as well as the log4j hotfix on top.

 

This is due to the log4j fix being released out of band to normal Commvault patch release cycle.

 

These log4j hotfixes will later be rolled into a maintenance release as well per usual software development cycles. 

 

Hope that clears things up for you a bit.

 

Thank you

Hi @Jordan just for clarification, I am currently running 11.21.62. On my Commvault Commcell Console, on my Commserve machine I can select All Tasks → Add/Remove Software → Download Software on from there, I will download Feature Release 21, Maintenance Release 71. I then use that to upgrade to Commserve, Media Agents, and eventually clients. 

Is there a similar process on the Commcell Console to install log4j as well?

You had mentioned that it will be rolled into a maintenance window in the future. So should I continuously check:

https://documentation.commvault.com/11.24/assets/service_pack/updates/11_21_71.htm

and maybe:

https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html

to see if the table will be updated so that it’s rolled into a Maintenance Release so all that’s required to install the fix is to upgrade to the latest Maintenance Release as was the case with previous security fixes?

 

 


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021

Hi @JSNOPUD ,

 

Your steps are fine up to here getting 11.21.71 installed onto CS and MA.

 

To get the Log4j hotfix installed, please see the pinned article at the top of this thread. Copying relevant part below:

 

Applying HotFix:

To get the hotfix installed, you’ll need to:

  1. Download the relevant updates in the chart below (depending on what Maintenance Releases you have in your CommCell)
  2. Unzip the contents of the download
  3. Run Copy To Cache to add the new updates to your software cache
  4. Push out updates to the clients

 

Using these steps, you can push out the updates to your clients once CS and MA are on 11.21.71.

 


Forum|alt.badge.img+5
  • Byte
  • 15 replies
  • December 14, 2021

@Jordan Okay, thanks very much. That makes it more clear.

Do you think there will be a future Maintenance Release which will incorporate these separate installations, or will the log4j fix continued to be separate from a Maintenance Release upgrade?


Forum|alt.badge.img+11
  • Vaulter
  • 135 replies
  • December 14, 2021

@JSNOPUD 

 

The hotfixes will be rolling into a future maintenance release, I just can’t advise on which at this stage or when it will be released as such. 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings