Log4j Vulnerability - Please Post All Questions Here

2 years ago
13 December 2021
344 replies
19588 views

Userlevel 7

+23

Mike Struening RETIRED
Vaulter
4998 replies

Summary

The following thread describes the potential exposure to the Apache Log4j vulnerability and steps to update Commvault software.

It has been confirmed that a small subset of Commvault agents are impacted.

Update as of 1st February: Maintenance Release to bring Log4j version to 2.17.1 across Commvault software platform has been released. This release includes the upgrade of components that previously used Log4j 1.x.

Update as of 20th December: Maintenance Release including relevant hotfixes now available for Commvault software, see section “Maintenance Releases”. Please note customers who have already applied hotfixes for Log4j 2.16, do not need to install.

For customers with Commvault Hyperscale X and Distributed storage, please see section new Community article here:

Apache Log4j information

CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. From log4j 2.15.0, this behavior has been disabled by default
CVE-2021-45046: Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default
CVE-2021-45105: Apache Log4j2 did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted.
CVE-2021-4104: JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data.
CVE-2021-44832: Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.

Note: Log4j version 1.x is NOT affected.

There’s a great blog article that covers the potential impact.

Updates:

CVE-2021-45105: Commvault can confirm that the affected Log4j2 function is NOT leveraged by Commvault software and thus there is no immediate need to update Commvault to use Log4j 2.17.

CVE-2021-4104: The Commvault software does not use the JMSAppender module and, therefore, the vulnerability about log4j 1.x versions does not affect any Commvault products.

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

Identifying and Updating Commvault

Exposure:

Note: check FAQ at the bottom of this post for specific version questions.

The exposure impacts the below Commvault product features:

• Cloud Apps package
• Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

• Commvault Distributed Storage

• Commvault Hyperscale X

Identifying affected servers using the Commvault Log4j report

Please use the below direct link to download the Commvault Log4J Affected Servers report then follow step 4 to import and run

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789

Alternatively, follow steps 1-3 to manually download the Commvault report:

Log into cloud.commvault.com and click the Software Store tile icon
Search the Store for Log4j and click the Download button for the “Log4J affected servers” report
Log into Command Center and navigate to Reports
At the top right of the Reports page, click Actions and Import report. Proceed to select the downloaded report file to import into Command Center.
Now you can run the report.

This report will show you any servers with Cloud Apps, SQL Server, and Oracle Database packages installed that may be affected by Log4j vulnerability.
Note: If the resulting report shows No Data to Display, then there are no affected clients in this CommCell

The easiest course of action is to upgrade all servers listed in this report (Oracle, Cloud Apps, and SQL) – that would be the recommendation. However, at a minimum, servers with Database Archiving, Data Masking, or Extent based backups (SQL table level restore) features enabled should have highest priority as the vulnerable log4j package is actively used, while otherwise the packages are dormant.

Maintenance Releases

The table below outlines the specific Maintenance Releases that will both address Log4j 2.x vulnerabilities as well as update Log4j 1.x components to 2.17.1 (the latest release of Log4j)

Note: If the previous Log4j 2.16 hotfixes has already been applied, then this latest Maintenance Release is optional

Feature Release	Maintenance Release
11.26	11.26.21
11.25	11.25.32
11.24	11.24.48
11.23	11.23.47
11.20	11.20.90
SP16	SP16.153

How to deploy Maintenance Release:

First perform a disaster recovery backup using steps HERE .
OPTIONAL: create a server group containing all the affected servers using instructions HERE. This can make it easier to select servers for the upgrade process.
Go to documentation to find the list of updates: https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html
Download the maintenance pack for the version the CommServe is running on.

If you do not know the CommServe version, in Command Center search for About and click the About search result to bring up the version popup.
Extract the Maintenance Pack
Follow instructions HERE to copy the software packages to the cache using Command Center.
Proceed to Install updates following instructions HERE . You can update only the affected clients to avoid the CommServe services stopping, however it is recommended to update the CommServe and all affected servers as shown on the report for completeness.
Once completed re-rerun the report to show that the servers have the appropriate fixes

Note: For instructions on how to apply Log4j 2.16 hotfixes on older Maintenance Release, please see FAQ

See Commvault Online Documentation for additional information:

https://documentation.commvault.com/11.25/essential/146231_security_vulnerability_and_reporting.html

HyperScale X and CDS (Hedvig)

For all detailed information on how to update HyperScale X and Commvault Distributed Storage (CDS) to address Log4j vulnerabilities, please see article here:

Log4j FAQ

Q:   There is a new vulnerability in 2.15 is Commvault addressing this?

A:   The LOG4J 2.15 version (GA Dec. 06, 2021) disabled the essential exploit functions by default was released last week on Dec 6, 2021. This was considered the market-acceptable, non-vulnerable upgrade package up to today. 

 The Apache organization released a new version, 2.16, on Monday, Dec.13, 2021, which physically removes the vulnerable functions.

 This evening, the security groups issued a new vulnerability CVE-2021-45046 targeting concerns with the 2.15 version and recommending the shift to 2.16.

  This significant change affects all client remediation methods, requiring an upgrade to version 2.16.  Log4j 2.16 hotfixes have now been released, please see table above

Q: When will new hotfixes be available for 2.16 log4j?

A: Log4j 2.16 hotfixes have now been released, please see table above

Q:  I've noticed older 1.x versions of log4j being used in the platform.  Are these vulnerable?

A:   We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action. 

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”. That version is not vulnerable to the current respective CVEs, but it would clear up the scanning concerns.

Q:  I noticed HyperScale 1.5 is using end of life versions of Log4j.  Is this being resolved?

A:  The 1.x versions of log4j bundled with HyperScale 1.5 are maintained and supported by Redhat.  These versions are not affected by this CVE.

Q: Are older versions like v10 and v9 affected?

A: These versions are not affected

Q: Why are some updates showing skipped during Copy to Cache?

A: These are updates for Operating Systems your CommCell does not have. It’s more informational than error related.

Q: Why does the report show No Records Available or No Items to Display?

A: This means there are no affected clients in this CommCell

Q: What order should I apply the updates?

A: The Maintenance Release needs to be installed first, then the Hotfix Pack. The best option is to use Copy to Cache, followed by pushing the updates out from the GUI as per the instructions. This will ensure everything is applied as needed in the correct order.

Q: Can I remove versions manually?

A: No, removing anything manually will potentially cause features to not work properly. Use the Maintenance Releases and Hotfix packs to remediate.

Q: Is Anti-virus a concern?

A: It is possible that an AV service may lock the affected files out of concern and cause features to not work properly. Use the Maintenance Releases and Hotfix packs to remediate.

Q: How do I download Maintenance Release using CommCell Console?

A: Please follow Commvault Online Documentation steps below

https://documentation.commvault.com/11.25/expert/2705_downloading_commvault_software_using_commcell_console.html

Q: Is Metallic vulnerable to the vulnerability?

A: We have found that the Log4j vulnerability has no impact on Metallic or the security and privacy of your data backups. Metallic does not use the impacted libraries as per the Apache Log4j advisory.

We will continue to proactively monitor and provide any further updates, while customers with questions can reach out to Metallic.io/support.

Q: Log4j scanner is still picking up DbArchiveEngine.jar as potentially vulnerable?

A: Some Log4j scanners are actually incorrectly picking up DbArchiveEngine.jar as potentially vulnerable when in fact it is already patched. This is because the scanner was unable to determine the version of Log4j used and ending up marking it as “potentially vulnerable”. Please note that if you have patched Commvault clients with either the 2.16 hotfix or the latest Maintenance Release, then this DbArchiveEngine.jar binary is also patched and will not have the Log4j 0-day vulnerability.

Q: I have updated to latest Maintenance Release but Log4j Affected Servers report is still showing my clients as not fixed?

A: There has been a new Log4j Affected Servers report released on December 22nd that has updated the checks to correctly report fixed for clients on the latest Maintenance Release. This new report is version 1.1.2.3 whereas the previous report is 1.1.2.2.

Q: How do I apply the Log4j hotfixes if I am already on the older minimum required Maintenance Release?

A: Please follow below steps:

Ensure the Commserve and affected clients are on the minimum required Maintenance Release pack.
1. If not, please download and install using the CommCell Console
2. Alternatively, you may download the minimum required Maintenance Release from the links in the table below
Download the Log4J-Fix pack for your version
Unzip the contents of the download
Run Copy To Cache and point to the folder created by the unzip to add the new updates to your software cache
Push out updates to the clients
Verify client status by checking the Log4j Affected Servers report or Client Details report or viewing the client properties

Log4j 2.16 fixes (CVE-2021-44228, CVE-2021-45046)

Feature Release	Minimum Required Maintenance Release	Update Link (includes 2.16 fix)	Installed Windows Updates	Installed Unix Updates
11.26	11.26.2	11.26 Log4J-2.16 Fix	1755	1755
11.25	11.25.9	11.25 Log4J-2.16 Fix	2763 2779	2763 2779
11.24	11.24.23	11.24 Log4J-2.16 Fix	4552 4564	4551 4564
11.23	11.23.37	11.23 Log4J-2.16 Fix	4160 4178	4161 4178
11.22	11.22.50	11.22 Log4J-2.16 Fix	3911 3920	3912 3920
11.21	11.21.66	11.21 Log4J-2.16 Fix	3587 3599	3588 3599
11.20	11.20.77	11.20 Log4J-2.16 Fix	4562 4574	4561 4574
SP16	SP16.128	SP16 Log4J-2.16 Fix	2943 2946	2942 2946

https://www.linkedin.com/in/michael-struening

Like
Quote
Share

Show first post Hide first post

344 replies

Page 3 / 14

Userlevel 7

+15

Hi @James_Goodison

Yes, you’re already above the minimum versions for 11.20, so (as described in the original post at the top) all you will need to do is run a Copy Software job for the hotfixes for 11.20 into the software cache and install updates on clients.

Thanks,

Stuart

Userlevel 7

+15

Hi @0ber0n

Yes, these hotfixes provided out of band now will be included in the next Maintenance Release, scheduled for release early in January.

Thanks,

Stuart

Userlevel 7

+15

@d3nz

Yes, if you are running Cloud Apps, please deploy the fixes to your access nodes.

Thanks,

Stuart

Userlevel 7

+15

Hi @LucM - BergerPeatMoss

Not all hotfixes are applicable to all clients, so if you manually install a hotfix on a client that doesn’t contain any affected packages, you may see the information alert “None of the updates are applicable for the packages installed on the machine” similar to this:

All that means is the affected packages were not found on that client and the hotfix was not needed for that client.

Please proceed with installation of the remaining hotfixes for your version.

Thanks,

Stuart

Userlevel 7

+15

Hi @Theja

Commserve and Media Agents are not affected by this vulnerability.

However, you will need to update your Commserve to the minimum version for your environment in order to push out updates from the software cache to clients.

Your first screenshot shows that SP22-HotFix-3913 was not applicable for that client.

Your second screenshot shows that SP22-HotFix-3911 was successfully applied to the client.

Thanks,

Stuart

Userlevel 7

+15

Hi @EPick

Yes, as you’re already above the minimum versions for 11.20, so (as described in the original post at the top) all you will need to do is run a Copy Software job for the hotfixes for 11.20 into the software cache and install updates on clients.

Thanks,

Stuart

Userlevel 1

The KB isn’t too clear to me.

If we don’t use these agents anywhere.

Cloud Apps package
Oracle agent - Database archiving, data masking, and logical dump backup
Microsoft SQL Server agent - Database archiving, data masking, and table level restore

Do we need to install the updates?

Userlevel 7

+15

Hi @Paul Hutchings

The only agents identified as affected are the ones listed, and the hotfixes provided are only applicable for those agents.

These hotfixes released to address this issue will be provided in the next Maintenance Release, scheduled for release in early January. So you can always catch up and apply the latest MR in January to ensure your environment is ultimately protected then.

Thanks,

Stuart

Userlevel 6

+15

The KB isn’t too clear to me.

If we don’t use these agents anywhere.

Cloud Apps package
Oracle agent - Database archiving, data masking, and logical dump backup
Microsoft SQL Server agent - Database archiving, data masking, and table level restore

Do we need to install the updates?

Hi @Paul Hutchings

I had the same concern, but, after discussions with my Secops team, they explained that as long as we have a vulnerable version of the product on our servers, we had to patch them. So many other products are affected, so it’s possible to have it fixed just product by product.

The report of affected servers, then download of fix, copy/sync and update, process has been generously detailed here :thumbsup: . So I decided to do it.

Now I’m pushing the updates little by little, starting with servers using the concerned agents, and later with the ones where it’s installed but not actively used. That’s also a good opportunity to remove it if unused…

Then, I feel happy to show this screenshot to my boss, thanks to this helpful community : :wink:

All, it still does not seem clear having opened up these hotfixes why the installupdates and removeupdates are being updated in this … unless CV dev have put some kind of wrapper around something similar to the jndi class / lib inside these binaries ?

Userlevel 7

+23

Hi @Libor

Support for V10 ended in December 2017, so no new hotfixes will be provided for V10 beyond SP15.

Log4j versions affected by this vulnerability are 2.0-2.14. Apache have provided a fix for this vulnerability in 2.15.

Thanks,

Stuart

Understand but i am not asking for hot fix or so. I am asking if V10 SP15 is impacted by

CVE-2021-44228.

Also i would like to know which log4j version is used in V10 SP15 ?

I would like to know this so i can be aware of possible risk and maybe to apply some workarounds because our environment does not allow update to higher version ATM.

Thank You!

@Libor , older versions are not impacted. I don’t know the specific versions we had in use on v10 SP15, though they are not affected.

Userlevel 4

+14

Hi @Bloopa , @Mohit Srivastava

The minimum MR level needs to be deployed to the Commserve to bring the Commserve up to that level, the the log4j hotfixes deployed to affected clients.

If the hotfix is deployed to all clients or some client that isn’t exposed, the hotfix will simply execute, determine no updates apply and exit. So there’s no impact if this fix is deployed to unaffected clients.

In fact this is true of all hotfixes, if they don’t apply to the targeted client, the hotfix installer will tell you no updates apply to this system and then exit.

The Commserve and Media Agents aren’t affected by log4j as they do not have those packages present in their installations. Commserve simply needs to be updated to the minimum level to facilitate push updates to all other clients.

Thanks,

Stuart

Thanks for all these important informations !

Hi all,

A customer has raised a concern with us regarding the eariler versions of Log4j2 They have run the scanner which shows another instance in the following path:

* \Program Files\Commvault\ContentStore\Base\vmheartbeatmon\zookeeper lib\log4/-1.2.16.jar

Also they refer to the UK governments National Cyber Security Centres alert which says the earlier versions should not be used.

https://www.ncsc.gov.uk/news/apache-log4j-vulnerability

“Version 1 of the Log4j library is no longer supported and is affected by multiple security vulnerabilities. Developers should migrate to the latest version of Log4j 2.”

Does Commvault have a resoponse to this I can pass on to my customer?

Hi all,

A customer has raised a concern with us regarding the eariler versions of Log4j2 They have run the scanner which shows another instance in the following path:

* \Program Files\Commvault\ContentStore\Base\vmheartbeatmon\zookeeper lib\log4/-1.2.16.jar

Also they refer to the UK governments National Cyber Security Centres alert which says the earlier versions should not be used.

https://www.ncsc.gov.uk/news/apache-log4j-vulnerability

“Version 1 of the Log4j library is no longer supported and is affected by multiple security vulnerabilities. Developers should migrate to the latest version of Log4j 2.”

Does Commvault have a resoponse to this I can pass on to my customer?

I see alot more. What’s up with those?

Hi All,

Does anyone know if this hotfix should be installed on clients where the backup is done through intellisnap?

Thanks

As per the article - Log4j – Apache Log4j Security Vulnerabilities, Log4j 1.x has reached end of life and is no longer supported. Vulnerabilities reported after August 2015 against Log4j 1.x were not checked and will not be fixed. Users should upgrade to Log4j 2 to obtain security fixes.

So kindly suggest if there is any use of log4j.1.x in Commvault as we can see multiple files in our installation directories. Please suggest how we can proceed on this.

Userlevel 7

+23

Hi All,

Does anyone know if this hotfix should be installed on clients where the backup is done through intellisnap?

Thanks

It depends on if those clients are using the archiving or masking features….or if they might be at some point. We suggest upgrading any that are possibly vulnerable to be on the safe side of things.

Userlevel 1

While Log4j versions prior to 2.0 are not subject to THIS vulnerability, they are well past EOL and there are critical other vulnerabilities to which they are subject. Some of my CommVault systems have as many as SIX versions of old Log4j binaries; what are you doing to address this?

Userlevel 7

+23

Hi @MathBob , @Rana , @C.Sudy , @Oriium

Edited: We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action.

Once I have more information on that discussion, I’ll reply here.

Do you know if this hotfix patch currently posted for commvault is also protection against this new

CVE - CVE-2021-45046 (mitre.org) “Apache Log4j2 Thread Context Message Pattern and Context Lookup Pattern vulnerable to a denial of service attack was released this afternoon.” vulnerability that was released today?

Userlevel 7

+23

@Dave S All of these are covered by the hotfix:

I’ll look into your CVE.

Updated CommCell to 11.20.77 no problem.

Log4J Fix contained 2 hotfixes, 4562 and 4563.

4562 installed on CommCell server, but 4563 said “None of the updates are applicable for the packages installed on the machine”

Reading the above comments, I think I understand that the CommCell just didn’t need that hotfix.

But, I’m having a problem doing the CommCell → Add/Remove Software → Copy Software to add the two patches to the cache.
Patches were unzipped, and I’ve tried pointing at top unzipped folder and all subfolders.

Where should I be pointing the Copy Software to in the unzipped folder structure ?

…..\Config ?

Adding Copy Job error:

Error Code: [68:481]
Description: The source folder [C:\Temp\hotfix4563] is empty or doesn't have any valid Commvault media.
Source: cnp033, Process: DownloadSoftware

Userlevel 5

+11

Jordan
Vaulter
134 replies
2 years ago
15 December 2021

hi @KGreer

Did you unzip the individual patches?

You should be able to just unzip the main download then point copy to cache to this folder (containing all the zipped individual patches) and let it copy everything.

Then it makes it easy to deploy to the respective clients that needs it which could be on different platforms etc.

Userlevel 3

+13

CS : V11.SP20.77 and 4562 applied

webserver installed on the Commserve and MS 365 defender discovers tomcat vulnerabilities - cve-2021-44228

Does Commserve/webserver uses log4j-1.2.16.jar because I see the file is in our Commserve installation directory (base\apache\lib)

Userlevel 5

+11

Jordan
Vaulter
134 replies
2 years ago
15 December 2021

hi @DanC

Looks like defender is incorrectly identifying Log4j 1.x files as part of cve-2021-44228. Lof4j 1.x is NOT impacted by this CVE.

Commvault does however understand that Log4j 1.x is quite old and may have other vulnerabilities so are currently putting together a plan to address this. Please note that Log4j 1.x does NOT have the most critical 0-day vulnerabilities in cve-2021-44228 like Log4j 2.x does.

Page 3 / 14

Summary

Apache Log4j information

Identifying and Updating Commvault

Exposure:

Identifying affected servers using the Commvault Log4j report

Maintenance Releases

How to deploy Maintenance Release:

HyperScale X and CDS (Hedvig)

Log4j FAQ

CVE-2021-44228.

Reply

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded