Log4j Vulnerability - Please Post All Questions Here

2 years ago
13 December 2021
344 replies
19614 views

Userlevel 7

+23

Mike Struening RETIRED
Vaulter
4998 replies

Summary

The following thread describes the potential exposure to the Apache Log4j vulnerability and steps to update Commvault software.

It has been confirmed that a small subset of Commvault agents are impacted.

Update as of 1st February: Maintenance Release to bring Log4j version to 2.17.1 across Commvault software platform has been released. This release includes the upgrade of components that previously used Log4j 1.x.

Update as of 20th December: Maintenance Release including relevant hotfixes now available for Commvault software, see section “Maintenance Releases”. Please note customers who have already applied hotfixes for Log4j 2.16, do not need to install.

For customers with Commvault Hyperscale X and Distributed storage, please see section new Community article here:

Apache Log4j information

CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. From log4j 2.15.0, this behavior has been disabled by default
CVE-2021-45046: Apache Log4j 2.15.0 was incomplete in certain non-default configurations. Log4j 2.16.0 fixes this issue by removing support for message lookup patterns and disabling JNDI functionality by default
CVE-2021-45105: Apache Log4j2 did not protect from uncontrolled recursion from self-referential lookups. This allows an attacker with control over Thread Context Map data to cause a denial of service when a crafted string is interpreted.
CVE-2021-4104: JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data.
CVE-2021-44832: Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code.

Note: Log4j version 1.x is NOT affected.

There’s a great blog article that covers the potential impact.

Updates:

CVE-2021-45105: Commvault can confirm that the affected Log4j2 function is NOT leveraged by Commvault software and thus there is no immediate need to update Commvault to use Log4j 2.17.

CVE-2021-4104: The Commvault software does not use the JMSAppender module and, therefore, the vulnerability about log4j 1.x versions does not affect any Commvault products.

CVE-2021-44832: The Commvault software does not use the JdbcAppender module and, therefore, the vulnerability about remote code execution attack does not affect any Commvault products.

Identifying and Updating Commvault

Exposure:

Note: check FAQ at the bottom of this post for specific version questions.

The exposure impacts the below Commvault product features:

• Cloud Apps package
• Oracle agent - Database archiving, data masking, and logical dump backup
• Microsoft SQL Server agent - Database archiving, data masking, and table level restore

• Commvault Distributed Storage

• Commvault Hyperscale X

Identifying affected servers using the Commvault Log4j report

Please use the below direct link to download the Commvault Log4J Affected Servers report then follow step 4 to import and run

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789

Alternatively, follow steps 1-3 to manually download the Commvault report:

Log into cloud.commvault.com and click the Software Store tile icon
Search the Store for Log4j and click the Download button for the “Log4J affected servers” report
Log into Command Center and navigate to Reports
At the top right of the Reports page, click Actions and Import report. Proceed to select the downloaded report file to import into Command Center.
Now you can run the report.

This report will show you any servers with Cloud Apps, SQL Server, and Oracle Database packages installed that may be affected by Log4j vulnerability.
Note: If the resulting report shows No Data to Display, then there are no affected clients in this CommCell

The easiest course of action is to upgrade all servers listed in this report (Oracle, Cloud Apps, and SQL) – that would be the recommendation. However, at a minimum, servers with Database Archiving, Data Masking, or Extent based backups (SQL table level restore) features enabled should have highest priority as the vulnerable log4j package is actively used, while otherwise the packages are dormant.

Maintenance Releases

The table below outlines the specific Maintenance Releases that will both address Log4j 2.x vulnerabilities as well as update Log4j 1.x components to 2.17.1 (the latest release of Log4j)

Note: If the previous Log4j 2.16 hotfixes has already been applied, then this latest Maintenance Release is optional

Feature Release	Maintenance Release
11.26	11.26.21
11.25	11.25.32
11.24	11.24.48
11.23	11.23.47
11.20	11.20.90
SP16	SP16.153

How to deploy Maintenance Release:

First perform a disaster recovery backup using steps HERE .
OPTIONAL: create a server group containing all the affected servers using instructions HERE. This can make it easier to select servers for the upgrade process.
Go to documentation to find the list of updates: https://documentation.commvault.com/11.24/essential/146231_security_vulnerability_and_reporting.html
Download the maintenance pack for the version the CommServe is running on.

If you do not know the CommServe version, in Command Center search for About and click the About search result to bring up the version popup.
Extract the Maintenance Pack
Follow instructions HERE to copy the software packages to the cache using Command Center.
Proceed to Install updates following instructions HERE . You can update only the affected clients to avoid the CommServe services stopping, however it is recommended to update the CommServe and all affected servers as shown on the report for completeness.
Once completed re-rerun the report to show that the servers have the appropriate fixes

Note: For instructions on how to apply Log4j 2.16 hotfixes on older Maintenance Release, please see FAQ

See Commvault Online Documentation for additional information:

https://documentation.commvault.com/11.25/essential/146231_security_vulnerability_and_reporting.html

HyperScale X and CDS (Hedvig)

For all detailed information on how to update HyperScale X and Commvault Distributed Storage (CDS) to address Log4j vulnerabilities, please see article here:

Log4j FAQ

Q:   There is a new vulnerability in 2.15 is Commvault addressing this?

A:   The LOG4J 2.15 version (GA Dec. 06, 2021) disabled the essential exploit functions by default was released last week on Dec 6, 2021. This was considered the market-acceptable, non-vulnerable upgrade package up to today. 

 The Apache organization released a new version, 2.16, on Monday, Dec.13, 2021, which physically removes the vulnerable functions.

 This evening, the security groups issued a new vulnerability CVE-2021-45046 targeting concerns with the 2.15 version and recommending the shift to 2.16.

  This significant change affects all client remediation methods, requiring an upgrade to version 2.16.  Log4j 2.16 hotfixes have now been released, please see table above

Q: When will new hotfixes be available for 2.16 log4j?

A: Log4j 2.16 hotfixes have now been released, please see table above

Q:  I've noticed older 1.x versions of log4j being used in the platform.  Are these vulnerable?

A:   We have some older instances in the installed component structure related to the older generation Log4J 1.x files which are not part of the current CVE Log4J 2.x vulnerability. We are doing further investigation on those conditions to determine a course of action. 

We do plan to remove all 1.x references in the Feb 1st maintenance release to prevent “false alarms”. That version is not vulnerable to the current respective CVEs, but it would clear up the scanning concerns.

Q:  I noticed HyperScale 1.5 is using end of life versions of Log4j.  Is this being resolved?

A:  The 1.x versions of log4j bundled with HyperScale 1.5 are maintained and supported by Redhat.  These versions are not affected by this CVE.

Q: Are older versions like v10 and v9 affected?

A: These versions are not affected

Q: Why are some updates showing skipped during Copy to Cache?

A: These are updates for Operating Systems your CommCell does not have. It’s more informational than error related.

Q: Why does the report show No Records Available or No Items to Display?

A: This means there are no affected clients in this CommCell

Q: What order should I apply the updates?

A: The Maintenance Release needs to be installed first, then the Hotfix Pack. The best option is to use Copy to Cache, followed by pushing the updates out from the GUI as per the instructions. This will ensure everything is applied as needed in the correct order.

Q: Can I remove versions manually?

A: No, removing anything manually will potentially cause features to not work properly. Use the Maintenance Releases and Hotfix packs to remediate.

Q: Is Anti-virus a concern?

A: It is possible that an AV service may lock the affected files out of concern and cause features to not work properly. Use the Maintenance Releases and Hotfix packs to remediate.

Q: How do I download Maintenance Release using CommCell Console?

A: Please follow Commvault Online Documentation steps below

https://documentation.commvault.com/11.25/expert/2705_downloading_commvault_software_using_commcell_console.html

Q: Is Metallic vulnerable to the vulnerability?

A: We have found that the Log4j vulnerability has no impact on Metallic or the security and privacy of your data backups. Metallic does not use the impacted libraries as per the Apache Log4j advisory.

We will continue to proactively monitor and provide any further updates, while customers with questions can reach out to Metallic.io/support.

Q: Log4j scanner is still picking up DbArchiveEngine.jar as potentially vulnerable?

A: Some Log4j scanners are actually incorrectly picking up DbArchiveEngine.jar as potentially vulnerable when in fact it is already patched. This is because the scanner was unable to determine the version of Log4j used and ending up marking it as “potentially vulnerable”. Please note that if you have patched Commvault clients with either the 2.16 hotfix or the latest Maintenance Release, then this DbArchiveEngine.jar binary is also patched and will not have the Log4j 0-day vulnerability.

Q: I have updated to latest Maintenance Release but Log4j Affected Servers report is still showing my clients as not fixed?

A: There has been a new Log4j Affected Servers report released on December 22nd that has updated the checks to correctly report fixed for clients on the latest Maintenance Release. This new report is version 1.1.2.3 whereas the previous report is 1.1.2.2.

Q: How do I apply the Log4j hotfixes if I am already on the older minimum required Maintenance Release?

A: Please follow below steps:

Ensure the Commserve and affected clients are on the minimum required Maintenance Release pack.
1. If not, please download and install using the CommCell Console
2. Alternatively, you may download the minimum required Maintenance Release from the links in the table below
Download the Log4J-Fix pack for your version
Unzip the contents of the download
Run Copy To Cache and point to the folder created by the unzip to add the new updates to your software cache
Push out updates to the clients
Verify client status by checking the Log4j Affected Servers report or Client Details report or viewing the client properties

Log4j 2.16 fixes (CVE-2021-44228, CVE-2021-45046)

Feature Release	Minimum Required Maintenance Release	Update Link (includes 2.16 fix)	Installed Windows Updates	Installed Unix Updates
11.26	11.26.2	11.26 Log4J-2.16 Fix	1755	1755
11.25	11.25.9	11.25 Log4J-2.16 Fix	2763 2779	2763 2779
11.24	11.24.23	11.24 Log4J-2.16 Fix	4552 4564	4551 4564
11.23	11.23.37	11.23 Log4J-2.16 Fix	4160 4178	4161 4178
11.22	11.22.50	11.22 Log4J-2.16 Fix	3911 3920	3912 3920
11.21	11.21.66	11.21 Log4J-2.16 Fix	3587 3599	3588 3599
11.20	11.20.77	11.20 Log4J-2.16 Fix	4562 4574	4561 4574
SP16	SP16.128	SP16 Log4J-2.16 Fix	2943 2946	2942 2946

https://www.linkedin.com/in/michael-struening

Like
Quote
Share

344 replies

Page 8 / 14

Userlevel 7

+15

Hi @zahni

You can register for Commvault Cloud access here:

https://cloud.commvault.com/commcellregistration/commcellregistration-form.jsp

This will enable to view lots of resources, including the Log4J fixes.

Thanks,

Stuart

Hi @zahni

You can register for Commvault Cloud access here:

https://cloud.commvault.com/commcellregistration/commcellregistration-form.jsp

This will enable to view lots of resources, including the Log4J fixes.

Thanks,

Stuart

Sorry, as I told you: no cloud. I will run a fresh scan with Nessus ( https://www.tenable.com/ ) later this day, to verify if any Log4J 2.xx exist.

hello,

is there a easy manual way to determine if a machine has the vulnerability? The report that CV published in the store only shows two machines in my environment that need the hotfix. We have other machines that have sql and oracle and I want to be sure everything that needs the patch gets it.

Thank you

hello,

Thank you

OK, I found nothing:

( the 2 unkown jar are also from 1.x)

Nessus detected 15 installs of Apache Log4j:  Path    : D:\Commvault\ContentStore\Base\WebApp\shared\tomcat-extras\log4j-1.2.17.jar  Version : 1.2.17  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\vmheartbeatmon\zookeeper\lib\log4j-1.2.16.jar  Version : 1.2.16  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CVAnalytics\DataCube\app\webapps\server\WEB-INF\lib\log4j.jar  Version : unknown  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CustomReportsEngine\WEB-INF\lib\log4j-1.2.16.jar  Version : 1.2.16  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\WebApp\shared\apache-ant-1.9.4\lib\log4j-1.2.14.jar  Version : 1.2.14  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CVAnalytics\CVSeaHome\app\webapps\server\WEB-INF\lib\log4j-1.2.16.jar  Version : 1.2.16  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\DBMinerTool\log4j.jar  Version : unknown  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CVCIEngine\CvPreviewHome\app\webapps\server\WEB-INF\lib\log4j.jar  Version : unknown  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\WebApp\shared\apache-ant-1.8.4\lib\log4j-1.2.14.jar  Version : 1.2.14  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CVAnalytics\CVSeaHome\app\webapps\server\WEB-INF\lib\log4j-1.2.17.jar  Version : 1.2.17  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Apache\lib\log4j-1.2.16.jar  Version : 1.2.16  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\CVCIEngine\CvPreviewHome\webapps\CvContentPreviewGenApp\WEB-INF\lib\log4j-1.2.17.jar  Version : 1.2.17  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\WebApp\shared\apache-ant-1.8.1\lib\log4j-1.2.14.jar  Version : 1.2.14  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\MessageQueue\lib\optional\log4j-1.2.17.jar  Version : 1.2.17  Method  : JAR filesystem search  Path    : D:\Commvault\ContentStore\Base\WebApp\shared\log4j-1.2.15.jar  Version : 1.2.15  Method  : JAR filesystem search

I have applied the log4j-2.16 hotfixes to my Customer’s site on top of 11.24.25. The Log4j affected servers report shows that there is one client affected and also shows that the corrective fix is installed. However the Customer is reporting that their scans still show that vulnerable software is installed on the server --> /opt/commvault/Base32/DbJars/log4j-core-2.3.jar

Are the hotfixes supposed to uninstall the older, vulnerable software?
If not, can the Customer manually remove the file above themselves to clear the positive scan they are getting?

Thanks!

Userlevel 7

+23

hello,

Thank you

@Cbcurry1 , then you only have 2 vulnerable. Update those 2 and you should be good.

Userlevel 7

+23

Are the hotfixes supposed to uninstall the older, vulnerable software?
If not, can the Customer manually remove the file above themselves to clear the positive scan they are getting?

Thanks!

@Scott Hall , the old versions will be in the uninstall folder, though an upcoming Maintenance Release will clear everything out.

It’s recommended to leave it (it’s not active) and let the upcoming MR remove it.

@Mike Struening and others:

I’ve seen references in this discussion to whether SQL Server clients and Oracle clients have database archiving or data masking “enabled,” but I can’t find where those features are enabled (or disabled). While our site seems to be clean from a Log4J standpoint for now, we’d like to make sure our DBAs don’t somehow enable Log4J before we have the fixes pushed out everywhere.

Can you clarify where those features are enabled or disabled, please?

Nick Laflamme

1)can oracle dba run backups via commvault (command line backups)and use datamasking and datarchiving feautures.We are asking this because we ran log4j clients report and we didnot see any clients effected in cv consoles but dba claims they use datamasking and archiving features.These backups are oracle command line backups.

2)How do we differentiate the logical dump subclient from other subclients(any configuration selection) and how do we know if we enabled datamasking on any of the sql and oracle clients.We are looking for configuration settings from cv console which can show whether these features are enabled/disabled on clients from commvault

Userlevel 7

+23

@Mike Struening and others:

Can you clarify where those features are enabled or disabled, please?

Nick Laflamme

@Nick Laflamme II sharing a screenshot someone posted a few posts back:

Userlevel 7

+23

Hi @bhbekkam If RMAN is doing the backups and not using our binaries, then you won’t be vulnerable.

Regarding which clients are affected/vulnerable, the report will tell you. The screenshot I just pasted above shows where the feature is enabled.

can rman go around using datamasking/archiving using our libobk.so file though the features are not enabled from commvault console

Also can you show me in commvault console where can we see if datamasking is enabled for a db?

Userlevel 1

Are the hotfixes supposed to uninstall the older, vulnerable software?
If not, can the Customer manually remove the file above themselves to clear the positive scan they are getting?

Thanks!

@Scott Hall , the old versions will be in the uninstall folder, though an upcoming Maintenance Release will clear everything out.

It’s recommended to leave it (it’s not active) and let the upcoming MR remove it.

The issue with this is that CISA is requiring that all versions of Log4j prior to 2.15 are removed by Friday, 12/24. If you are company who does business with the Feds this is a big issue for you. Please provide guidance.

Userlevel 5

+11

Jordan
Vaulter
134 replies
2 years ago
17 December 2021

can rman go around using datamasking/archiving using our libobk.so file though the features are not enabled from commvault console

Also can you show me in commvault console where can we see if datamasking is enabled for a db?

hi @bhbekkam

Not sure about what you are asking for the first question, but if you use the Log4j Affected Servers report, it will tell you which clients have these features enabled.

https://cloud.commvault.com/webconsole/softwarestore/store.do#!/135/663/21789

Please see top sticky post for all updated information:

Userlevel 5

+11

Jordan
Vaulter
134 replies
2 years ago
17 December 2021

hi @MathBob

Completely understand the predicament here. Commvault is targeting the rollup Maintenance Release to be available before December 24th. Once the rollup MR is installed, all temporary old binaries will also be completely removed.

Please track the top sticky post for updates

Thank you

Userlevel 7

+23

The original post has now been updated with the process Remediating affected CDS file system for HyperScale X (HSX) Nodes.

Note: These instructions only apply to Hyperscale X /CDS/Hedvig.

Hyperscale 1.5 uses the 1.x versions of log4j bundled which are maintained and supported by Redhat.  These versions are not affected by this CVE.

Hello there,

We have updated to latest MR with the recommended hotfix and patches recommended for log4j however some client servers still detected as vulnerability in internal VA scan and show the Log4j 2.15 binary though already updated as per Commvault recommendation. Please see the below screenshot for hotfixes installed in one of client servers which is still coming up in vulnerability scan.. please advise further

Userlevel 5

Hello there,

When you install the updates, the old 2.15 binaries are copied to a cache (updates folder) for rollback purposes. This is how installupdates works with Commvault - just incase something goes wrong during a hotfix install. They are not in use. In the upcoming Maintenance packs, we will be cleaning these up soe they arent picked up on scans.

We have installed the latest patch.

But if we scan the Oracle systems, we got always an vulnerability information.

# ./log4j2-scan /opt
Logpresso CVE-2021-44228 Vulnerability Scanner 2.3.1 (2021-12-19)
Scanning directory: /opt (without devtmpfs, tmpfs, 192.168.101.250:/vol_trace_archive, FSFS01O1-101:/ora_mesPDB_oraarch, FSFS01O1-101:/ora_mesPDB_crs3, FSFS01O1-101:/ora_mesPDB_redo1, FSFS01O1-101:/ora_mesPDB_crs2, 192.168.101.90:/pdb_quafos, FSFS01O1-101:/ora_mesPDB_oradata, FSFS01O1-101:/ora_mesPDB_crs1, FSFS01O1-101:/ora_mesPDB_redo2, 192.168.101.90:/sap2mds_data, fsfs0003:/vol/ora_backup)

Found CVE-2021-44228 (log4j 2.x) vulnerability in /opt/omni/bin/telemetry/log4j-core-2.6.2.jar, log4j 2.6.2 (mitigated)

Found CVE-2021-45046 (log4j 2.x) vulnerability in /opt/oracle.ahf/common/jlib/tfa.war (WEB-INF/lib/log4j-core-2.15.0.jar), log4j 2.15.0 (mitigated)

Found CVE-2021-45046 (log4j 2.x) vulnerability in /opt/oracle.ahf/common/jlib/log4j-core-2.15.0.jar, log4j 2.15.0 (mitigated)
[?] Found CVE-2021-44228 (log4j 2.x) vulnerability in /opt/commvault/Updates/linux-x8664_11.0.0B80-SP24_SP24-HotFix-4564/Base64/DbJars/DbArchiveEngine.jar, log4j N/A - potentially vulnerable
[?] Found CVE-2021-44228 (log4j 2.x) vulnerability in /opt/commvault/Base64/DbJars/DbArchiveEngine.jar, log4j N/A - potentially vulnerable

Scanned 5279 directories and 45651 files
Found 0 vulnerable files
Found 2 potentially vulnerable files
Found 3 mitigated files
Completed in 6.89 seconds

The both required Hotfixes 4551 and 4564 are installed.

Is the issue exist furthermore and it is planned to provide a new patch?

Userlevel 1

What about .17 that was released earlier today to address another issue that was found?

As a fed msp, we are getting requests for information and updates on following apache recommendations as systems are being scanned even with the .16 update in place.

The likelihood of being able to obtain waivers for CV is low as everyone has visibility on this (way outside the normal chain of command).

Userlevel 5

+11

Jordan
Vaulter
134 replies
2 years ago
19 December 2021

What about .17 that was released earlier today to address another issue that was found?

As a fed msp, we are getting requests for information and updates on following apache recommendations as systems are being scanned even with the .16 update in place.

The likelihood of being able to obtain waivers for CV is low as everyone has visibility on this (way outside the normal chain of command).

Hi @chrisp

Commvault engineering team has determined that Log4j 2.17’s vulnerability is only when a specific feature of Log4j is used and this feature can be confirmed to NOT be in use across Commvault’s code base.

This means Commvault cannot be vulnerable to 2.16’s vulnerability and it is secure to stay on 2.16 for Commvault products.

Thank you

Userlevel 7

+23

We have installed the latest patch.

But if we scan the Oracle systems, we got always an vulnerability information.

Scanned 5279 directories and 45651 files
Found 0 vulnerable files
Found 2 potentially vulnerable files
Found 3 mitigated files
Completed in 6.89 seconds

The both required Hotfixes 4551 and 4564 are installed.

Hey @Nishika,

Mitigated means the binary is unaffected from the vulnerability (i.e it has been patched) - so no action needed. The reason why your scanner is detecting potentially vulnerable files is because it cannot identify the version, however with the hotfixes installed those files are safe, so no further action needs to be taken. You can double check using the report in the top of this post as well.

Userlevel 3

Shane
Commvault Certified Expert
74 replies
2 years ago
20 December 2021

Hi All, sorry for beating a dead horse, but Log4J v1.

I’ve seen a few entries above stating that v1 is not vulnerable, or a bit cryptically “These versions are not affected by this CVE.”

Since Nessus and Microsoft Threat Intelligence are reporting that ..\Base\vmheartbeatmon\zookeeper\lib\log4j-1.2.16.jar is vulnerable, is there a non-Commvault site that confirms we do not have to patch that? Customers are a bit wary at the moment and quite paranoid.

Many thanks, and keep up the hard work during this difficult time.