commvault.com commvault.com

Log4j Vulnerability - Please Post All Questions Here

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Summary
 

The following thread describes the potential exposure to the Apache Log4j vulnerability and steps to update Commvault software.

It has been confirmed that a small subset of Commvault agents are impacted. 

 

Update as of 1st February: Maintenance Release to bring Log4j version to 2.17.1 across Commvault software platform has been released. This release includes the upgrade of components that previously used Log4j 1.x. 

 

Update as of 20th December: Maintenance Release including relevant hotfixes now available for Commvault software, see section “Maintenance Releases”. Please note customers who have already applied hotfixes for Log4j 2.16, do not need to install.

For customers with Commvault Hyperscale X and Distributed storage, please see section new Community article here: 

 

Apache Log4j information

  • CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. From log4j 2.15.0, this behavior has been disabled by default
  • CVE-2021-45046: Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default
  • CVE-2021-45105: Apache Log4j2 did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted. 

  • CVE-2021-4104: JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data.

  • CVE-2021-44832: Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.

Note: Log4j version 1.x is NOT affected.

There’s a great blog article that covers the potential impact.

 

Updates:

CVE-2021-45105: Commvault can confirm that the affected Log4j2 function is NOT leveraged by Commvault software and thus there is no immediate need to update Commvault to use Log4j 2.17. 

CVE-2021-4104: The Commvault software does not use the JMSAppender module and, therefore, the vulnerability about log4j 1.x versions does not affect any Commvault products.

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

 

Identifying and Updating Commvault

 

Exposure:

Note: check FAQ at the bottom of this post for specific version questions.

The exposure impacts the below Commvault product features:

• Cloud Apps package
• Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

• Commvault Distributed Storage

• Commvault Hyperscale X

 

Identifying affected servers using the Commvault Log4j report

 

Please use the below direct link to download the Commvault Log4J Affected Servers report then follow step 4 to import and run

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789  

 

Alternatively, follow steps 1-3 to manually download the Commvault report:

  1. Log into cloud.commvault.com and click the Software Store tile icon 
  2. Search the Store for Log4j and click the Download button for the “Log4J affected servers” report 
  3. Log into Command Center and navigate to Reports
  4. At the top right of the Reports page, click Actions and Import report. Proceed to select the downloaded report file to import into Command Center. 
  5. Now you can run the report.

This report will show you any servers with Cloud Apps, SQL Server, and Oracle Database packages installed that may be affected by Log4j vulnerability.
Note: If the resulting report shows No Data to Display, then there are no affected clients in this CommCell

The easiest course of action is to upgrade all servers listed in this report (Oracle, Cloud Apps, and SQL) – that would be the recommendation. However, at a minimum, servers with Database Archiving, Data Masking, or Extent based backups (SQL table level restore) features enabled should have highest priority as the vulnerable log4j package is actively used, while otherwise the packages are dormant.

 

Maintenance Releases

 

The table below outlines the specific Maintenance Releases that will both address Log4j 2.x vulnerabilities as well as update Log4j 1.x components to 2.17.1 (the latest release of Log4j)

Note: If the previous Log4j 2.16 hotfixes has already been applied, then this latest Maintenance Release is optional 

Feature Release

Maintenance Release

11.26

11.26.21

11.25

11.25.32

11.24

11.24.48

11.23

11.23.47

11.20

11.20.90

SP16

SP16.153

 

How to deploy Maintenance Release:

 

  1. First perform a disaster recovery backup using steps HERE .
  2. OPTIONAL: create a server group containing all the affected servers using instructions HERE.  This can make it easier to select servers for the upgrade process.
  3. Go to documentation to find the list of updates: https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html
  4. Download the maintenance pack for the version the CommServe is running on. 

    If you do not know the CommServe version, in Command Center search for About and click the About search result to bring up the version popup. 
  5. Extract the Maintenance Pack
  6. Follow instructions HERE to copy the software packages to the cache using Command Center.  
  7. Proceed to Install updates following instructions HERE .  You can update only the affected clients to avoid the CommServe services stopping, however it is recommended to update the CommServe and all affected servers as shown on the report for completeness.
  8. Once completed re-rerun the report to show that the servers have the appropriate fixes

Note: For instructions on how to apply Log4j 2.16 hotfixes on older Maintenance Release, please see FAQ

 

See Commvault Online Documentation for additional information:

https://documentation.commvault.com/11.25/essential/146231_security_vulnerability_and_reporting.html

 

HyperScale X and CDS (Hedvig)

 

For all detailed information on how to update HyperScale X and Commvault Distributed Storage (CDS) to address Log4j vulnerabilities, please see article here:

 

 

Log4j FAQ 

 

Q:   There is a new vulnerability in 2.15 is Commvault addressing this? 

A:   The LOG4J 2.15 version (GA Dec. 06, 2021) disabled the essential exploit functions by default was released last week on Dec 6, 2021. This was considered the market-acceptable, non-vulnerable upgrade package up to today.  

 The Apache organization released a new version, 2.16, on Monday, Dec.13, 2021, which physically removes the vulnerable functions. 

 This evening, the security groups issued a new vulnerability CVE-2021-45046 targeting concerns with the 2.15 version and recommending the shift to 2.16.   

  This significant change affects all client remediation methods, requiring an upgrade to version 2.16.  Log4j 2.16 hotfixes have now been released, please see table above

 

Q: When will new hotfixes be available for 2.16 log4j? 

A: Log4j 2.16 hotfixes have now been released, please see table above

 

Q:  I've noticed older 1.x versions of log4j being used in the platform.  Are these vulnerable? 

A:   We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action.  

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”.  That version is not vulnerable to the current respective CVEs, but it would clear up the scanning concerns.

 

Q:  I noticed HyperScale 1.5 is using end of life versions of Log4j.  Is this being resolved? 

A:  The 1.x versions of log4j bundled with HyperScale 1.5 are maintained and supported by Redhat.  These versions are not affected by this CVE. 

 

Q: Are older versions like v10 and v9 affected?

A: These versions are not affected

 

Q: Why are some updates showing skipped during Copy to Cache?

A: These are updates for Operating Systems your CommCell does not have.  It’s more informational than error related.

 

Q: Why does the report show No Records Available or No Items to Display?

A: This means there are no affected clients in this CommCell

 

Q: What order should I apply the updates?

A: The Maintenance Release needs to be installed first, then the Hotfix Pack.  The best option is to use Copy to Cache, followed by pushing the updates out from the GUI as per the instructions.  This will ensure everything is applied as needed in the correct order.

 

Q: Can I remove versions manually?

A: No, removing anything manually will potentially cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

 

Q: Is Anti-virus a concern?

A: It is possible that an AV service may lock the affected files out of concern and cause features to not work properly.  Use the Maintenance Releases and Hotfix packs to remediate.

 

Q: How do I download Maintenance Release using CommCell Console?

A: Please follow Commvault Online Documentation steps below

https://documentation.commvault.com/11.25/expert/2705_downloading_commvault_software_using_commcell_console.html

 

Q: Is Metallic vulnerable to the vulnerability?

A: We have found that the Log4j vulnerability has no impact on Metallic or the security and privacy of your data backups. Metallic does not use the impacted libraries as per the Apache Log4j advisory.

We will continue to proactively monitor and provide any further updates, while customers with questions can reach out to Metallic.io/support.

 

Q: Log4j scanner is still picking up DbArchiveEngine.jar as potentially vulnerable?

A: Some Log4j scanners are actually incorrectly picking up DbArchiveEngine.jar as potentially vulnerable when in fact it is already patched. This is because the scanner was unable to determine the version of Log4j used and ending up marking it as “potentially vulnerable”. Please note that if you have patched Commvault clients with either the 2.16 hotfix or the latest Maintenance Release, then this DbArchiveEngine.jar binary is also patched and will not have the Log4j 0-day vulnerability. 

 

Q: I have updated to latest Maintenance Release but Log4j Affected Servers report is still showing my clients as not fixed?

A: There has been a new Log4j Affected Servers report released on December 22nd that has updated the checks to correctly report fixed for clients on the latest Maintenance Release. This new report is version 1.1.2.3 whereas the previous report is 1.1.2.2.

 

Q: How do I apply the Log4j hotfixes if I am already on the older minimum required Maintenance Release?

A: Please follow below steps:

  1. Ensure the Commserve and affected clients are on the minimum required Maintenance Release pack. 
    1. If not, please download and install using the CommCell Console 
    2. Alternatively, you may download the minimum required Maintenance Release from the links in the table below
  2. Download the Log4J-Fix pack for your version
  3. Unzip the contents of the download
  4. Run Copy To Cache and point to the folder created by the unzip to add the new updates to your software cache
  5. Push out updates to the clients
  6. Verify client status by checking the Log4j Affected Servers report or Client Details report or viewing the client properties

Log4j 2.16 fixes (CVE-2021-44228, CVE-2021-45046)

Feature Release

Minimum Required
Maintenance Release

Update Link (includes 2.16 fix)

Installed Windows
Updates

Installed
Unix Updates

11.26

11.26.2

11.26 Log4J-2.16 Fix

1755

1755

11.25

11.25.9

11.25 Log4J-2.16 Fix

2763

2779

2763

2779

11.24

11.24.23

11.24 Log4J-2.16 Fix

4552

4564

4551

4564

11.23

11.23.37

11.23 Log4J-2.16 Fix

4160

4178

4161

4178

11.22

11.22.50

11.22 Log4J-2.16 Fix

3911

3920

3912

3920

11.21

11.21.66

11.21 Log4J-2.16 Fix

3587

3599

3588

3599

11.20

11.20.77

11.20 Log4J-2.16 Fix

4562

4574

4561

4574

SP16

SP16.128

SP16 Log4J-2.16 Fix

2943

2946

2942

2946
https://www.linkedin.com/in/michael-struening

344 replies

W
Rank
Userlevel 2
Badge +2

Hey Mike

Hope you are keeping well

Over the last couple of days this has been causing some ripples. By the way, Commvault says it only affects the cloud apps packages / SQL agent / Oracle agent

But our internal vulnerability scan system was able to find this “Log4J” vulnerability on multiple server that didnt have any SQL/oracle agents installed. And know what when checking those servers it turned out to be the commvault folder that had this log4j file 
path : D:\program files\Commvault\ContentStore\Base\DBMinerTool

1. Can you help to know what is “DBminerTool” . What is it being used / installed for ?

2.  Checked some properties of this log4j file and the version is 1.2.17. Is it still vulnerable ?
     This may have been installed in any of the commvaults old MR / FR packages

3. Can we have these files removed from all the agents that had it detected ? Are these file here for a specific purpose or can they be deleted ?

 

 

M
Rank
Userlevel 3
Badge +6

@wstbackupNo that version is not an issue for this vulnerability.
 

 

Chris Hollis
Rank
Userlevel 6
Badge +15

Just a quick tip for those who are using the Java Console and aren’t sure what version / feature release / maintenance release they have installed.. you can confirm by right clicking the client machine (or Commserve/MediaAgent) > properties > version tab 

 


In the above example, 11 = version, 20 = feature release (previously known as service pack), 77 = maintenance release.

You can also view multiple / all your clients from the ‘client computer’ list as per this example:
 

Official Link is here: https://documentation.commvault.com/11.24/expert/2669_viewing_feature_release_details_for_one_or_more_clients.html

 

L
Rank
Userlevel 6
Badge +15

The KB isn’t too clear to me.

If we don’t use these agents anywhere.

  • Cloud Apps package
  • Oracle agent - Database archiving, data masking, and logical dump backup
  • Microsoft SQL Server agent - Database archiving, data masking, and table level restore

Do we need to install the updates?

Hi @Paul Hutchings 

I had the same concern, but, after discussions with my Secops team, they explained that as long as we have a vulnerable version of the product on our servers, we had to patch them. So many other products are affected, so it’s possible to have it fixed just product by product. 

 

The report of affected servers, then download of fix, copy/sync and update, process has been generously detailed here :thumbsup: . So I decided to do it.

 

Now I’m pushing the updates little by little, starting with servers using the concerned agents, and later with the ones where it’s installed but not actively used. That’s also a good opportunity to remove it if unused…

Then, I feel happy to show this screenshot to my boss, thanks to this helpful community : :wink:

 

Onno van den Berg
Rank
Userlevel 7
Badge +19

Even though Commvault states it is not affected by the latest finding in version Log4j2 2.16, I still urge Commvault to make sure 2.16 is replaced as soon as possible with Log4j2 2.17. Reason because it removes a lot of noise for many IT departments who will have to clarify towards their internal security teams, customers (think of MSPs) that Commvault is not using the specific Log4j2 function. 

 

 

J
Rank
Userlevel 5
Badge +11

Would it be beneficial to push out the log4j fix to all Oracle and SQL clients (we don’t use the cloud app) as a precautionary action?

I am unsure if Oracle ArchiveLog subclients fall under ‘Archiving’, so would rather be safe than sorry.

 

 

 

Absolutely, deploy to your CS and all Oracle and SQL clients. ArchiveLog does not fall under “Archiving” but better to be safe than sorry as well in case your DBA decides to do something :)

S
Rank
Userlevel 7
Badge +15

Hi @Bloopa , @Mohit Srivastava 

The minimum MR level needs to be deployed to the Commserve to bring the Commserve up to that level, the the log4j hotfixes deployed to affected clients.

If the hotfix is deployed to all clients or some client that isn’t exposed, the hotfix will simply execute, determine no updates apply and exit. So there’s no impact if this fix is deployed to unaffected clients.

In fact this is true of all hotfixes, if they don’t apply to the targeted client, the hotfix installer will tell you no updates apply to this system and then exit.

The Commserve and Media Agents aren’t affected by log4j as they do not have those packages present in their installations. Commserve simply needs to be updated to the minimum level to facilitate push updates to all other clients.

Thanks,

Stuart

Customer Support
M
Rank
Userlevel 1
Badge +2

Apparently, unless you specifically have those options for SQL configured on your agent, nothing shows up, correct? You guys should get some credit for cooking up the report but the feedback could be a little clearer from the report. Just saying. Thanks for all you are doing. 

 

Onno van den Berg
Rank
Userlevel 7
Badge +19

@Laurent @Sisinfra_DGTIC this unfortunately is one of the downsides of the decision that Commvault made to reduce the amount of documented updates by not documenting the downstream hotfixes/updates in the release notes of a new maintenance release. I have not validated it myself but if you examine the hotfix numbers and you cross-reference them against the updates mentioned in the log4j article than you notice the numbers are part of the series that is covered by the maintenance release. 

So for FR24.29 the starting number is 4549 and it stops at 4570 and the specific log4j updates are numbered with Windows (4552/4564) and for Linux (4551/4564).

C
Rank
Userlevel 1
Badge

Even though Commvault states it is not affected by the latest finding in version Log4j2 2.16, I still urge Commvault to make sure 2.16 is replaced as soon as possible with Log4j2 2.17. Reason because it removes a lot of noise for many IT departments who will have to clarify towards their internal security teams, customers (think of MSPs) that Commvault is not using the specific Log4j2 function.

I'm going to agree with this.

 

We support fed and dod organizations and this will be a battle to keep the solution in place with such a glaring and active vulnerability out there, regardless of cv saying they aren't affected. Coupled with the lack of clear documentation on all of the patches and documentation the impression is that this is not an important item.

 

And it's clear that dev and support don't deal with classified or government systems or consider them important.

As an FYI, the dhs directive gives wide latitude for security teams to offline and isolate affected systems without input from the system owners.

 

Maintaining and pushing cv where we can is an uphill battle on the best of days so it It would behoove cv to take the government seriously on vulnerability response.

Onno van den Berg
Rank
Userlevel 7
Badge +19

@Mike Struening

 

Do I just skip the 11.22.50 and migrate to 11.22.57 then? Do I update CommServe, then Media Agents then all other clients in that order?

Just skip 11.22.50 and go directly to 11.22.57, and indeed update in the order as you specified!  

S
Rank
Userlevel 2
Badge +2

@Deepk Mathew , you look to be on version 11.20.77 which is the required version for 11.20 Log4J-2.16 Fix, but that version does not remove the log4j versions that are copied to cache for recovery purposes.  MR 11.20.85 is now available for download that updates and I believe removes the older binaries from cache.

J
Rank
Userlevel 5
Badge +11

hi @MathBob ,

 

Sticky post at the top has been updated. MR pack rolls up the Log4j 2.16 patches for easy deployment, it does not address Log4j 1.x at this time. Once Log4j 1.x update roadmap is complete, it will be announced.

 

Thank you

Onno van den Berg
Rank
Userlevel 7
Badge +19

@Mohit Chordia  I would strongly recommend to update all clients to the latest version. That way you are sure you are fully up-to-date and that there are no vulnerable version left in the Updates folder that can be picked up by scanners.

@0ber0n There is no new update for the report so the report most likely will not have the logic to take notice of the version that has the fixes embedded. The current version of the report only looks for the separate hotfixes. 

M
Rank
Userlevel 1
Badge +2

hi @MathBob ,

 

Sticky post at the top has been updated. MR pack rolls up the Log4j 2.16 patches for easy deployment, it does not address Log4j 1.x at this time. Once Log4j 1.x update roadmap is complete, it will be announced.

 

Thank you

Thanks for the update. I thank you for all your work. I have a few observations/suggestions, and would like you to take them in the spirit of having come from someone who knows how hard you are all working. 

 

  1. We need a 2.17 fix. We are a government client and they just want this. And with good reason. This one should not be that hard. 
  2. We need a fix for the 1.x libraries to be removed. They have several CVEs all of their very own and, again, the feds do not want to hear it. They want it gone. So - we need to know- is there a way to safely remove these manually? 
  3. Communication there is a little spotty. It’s hard to extract facts from you guys right now. I would suggest that you need to be a little clearer and a little more reliable on what’s being released in terms of info. 

Once again, thanks. 

D
Rank
Userlevel 5
Badge +8

@wstbackup We are upgrading remaining older versions of log4j, but as of our current analysis they can be ignored since they are not affected.

S
Rank
Userlevel 7
Badge +15

Hi @Laurent 

Thanks for your perfectly reasonable question :nerd:

SQL Agents not using the identified features may not be as exposed to the log4j vulnerability, but the packages may still be found on those systems. These hotfixes will help clean those up by removing affected packages.

So, the best advice right now is to apply the hotfixes on any potentially affected clients.

Thanks,

Stuart

Customer Support
Scott Moseman
Rank
Userlevel 6
Badge +18

This CVE pertains to the vulnerability causing the release of 2.17.1

https://documentation.commvault.com/v11/essential/146231_security_vulnerability_and_reporting.html

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

Thanks,
Scott
 

Commvault Enterprise Success
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@bc1410 you as well!

I just heard back from our dev team and I can share this bit:

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”.  That version is not vulnerable to the current respective CVEs, but it would clear up the scanning for you.

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

I added a FAQ into the original post which covers many of the recent questions.  If there’s anything the FAQ and post don’t cover, please don’t hesitate to ask.  As this evolves, I’ll keep collecting information, and answers.

Appreciate seeing our community coming together to assist and advise, ask and share :sunglasses:   you are all the best!

https://www.linkedin.com/in/michael-struening
S
Rank
Userlevel 7
Badge +15

Hello! I am on FR 11.24 MR 23. Is the fix included in the MR 25 if I update to it ? Or do I still need to apply the fix.

Hi @mciobanu 

The Log4J Fixes have not yet been rolled up into an MR, this is happening very soon and should be released by Dec 24th.

Until MR is provided containing these hotfixes, you will need to apply the hotfixes from 11.24 Log4J-2.16 Fix additionally.

Thanks,

Stuart

Customer Support
S
Rank
Userlevel 7
Badge +15

Hi @Fernando Souza 

Thanks for highlighting this one, we have updated the FAQ section at the top of the page - please keep checking back regularly as we’re updating the page with all the latest info!

Q: Why does the report show No Data Available or No Items to Display?

A: This means there are no affected clients in this CommCell

Thanks,

Stuart

Customer Support
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

I have applied the log4j-2.16 hotfixes to my Customer’s site on top of 11.24.25. The Log4j affected servers report shows that there is one client affected and also shows that the corrective fix is installed. However the Customer is reporting that their scans still show that vulnerable software is installed on the server --> /opt/commvault/Base32/DbJars/log4j-core-2.3.jar

  1. Are the hotfixes supposed to uninstall the older, vulnerable software?
  2. If not, can the Customer manually remove the file above themselves to clear the positive scan they are getting?

Thanks!

 

@Scott Hall , the old versions will be in the uninstall folder, though an upcoming Maintenance Release will clear everything out.

It’s recommended to leave it (it’s not active) and let the upcoming MR remove it.

https://www.linkedin.com/in/michael-struening
J
Rank
Userlevel 5
Badge +11

What about .17 that was released earlier today to address another issue that was found?

As a fed msp, we are getting requests for information and updates on following apache recommendations as systems are being scanned even with the .16 update in place.

The likelihood of being able to obtain waivers for CV is low as everyone has visibility on this (way outside the normal chain of command).

Hi @chrisp 

 

Commvault engineering team has determined that Log4j 2.17’s vulnerability is only when a specific feature of Log4j is used and this feature can be confirmed to NOT be in use across Commvault’s code base. 

 

This means Commvault cannot be vulnerable to 2.16’s vulnerability and it is secure to stay on 2.16 for Commvault products. 

 

Thank you

S
Rank
Userlevel 3
Badge +8

Hi All, sorry for beating a dead horse, but Log4J v1.

I’ve seen a few entries above stating that v1 is not vulnerable, or a bit cryptically “These versions are not affected by this CVE.”

Since Nessus and Microsoft Threat Intelligence are reporting that ..\Base\vmheartbeatmon\zookeeper\lib\log4j-1.2.16.jar is vulnerable, is there a non-Commvault site that confirms we do not have to patch that? Customers are a bit wary at the moment and quite paranoid.

Many thanks, and keep up the hard work during this difficult time.

Reply


Terms & ConditionsCookie settings