Solved

Tenable Nessus Agents on Commvault Servers

  • 4 February 2021
  • 13 replies
  • 136 views

Badge +2

Is anyone using Tenable Nessus agent in their environment? Have you noticed any issues with throughput for your backups? I ask because we have it installed on our Commvault servers and it is affecting the network but I'm being told that Nessus agent does not interfere with day to day operations. I ran a library throughput trend report with Nessus agent installed, uninstalled and reinstalled, there definitely is a issue there. Anyone else having similar issues?

icon

Best answer by dude 5 February 2021, 12:22

@Tenzin I did have similar issues with Tenable Nessus before (about a year ago). Had always backed up different servers, various agents just fine prior the introduction of Nessus. Tenable Nessus was implemented and I started seeing CV ports being "scanned/blocked" interrupting CV Backup connection.

While checking the logs I could see ports 8400/8403 being scanned (sort of taken by Nessus). 

Came across this doc and my actual solution was to exclude the Oracle Servers from Nessus Scan.

https://community.tenable.com/s/article/Applications-on-a-host-being-scanned-crash-while-Nessus-is-scanning-the-host

Had a ticket with CV when I was somewhat already convinced Nessus was breaking things. Inquired about some sort of documentation around that and nothing was found in CV Docs. Talked to Security and they said that Nessus as considering CV operation as a threat. And from there it was a ping pong where CV would ask me to talk to Nessus and vice versa.

I`d say, select maybe 2 / 3 different agents you are having issues and create a total exclusion from Nessus, run a backup. It is likely you will see success. Grab the logs, enable Nessus and run another backup and compare the logs, see if you can spot any of the errors below and analyze with your security team.

An exclusion may very well be needed which is what I ended up for one UNIX only. 

@@@@

Some of the logs I had seen it were like this. 

packet size (50331667)

502115 cd  05/18 15:28:59 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (1195725856)

502115 cd  05/18 15:28:59 ### [CVipcD] slRecvMsgWithTout(): socket 15: Connection reset by peer

502115 13  05/18 15:29:07 ### [CVipcD] ERROR: connCVd: accept error [130]

502115 13  05/18 15:29:07 ### [CVipcD] connCvd: listen socket will not be closed.

 

CVFWD.LOG

502117 0001 05/18 15:26:38 ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x16

502117 0001 05/18 15:26:38 TN:00003 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=14. Drop the socket.

 

CVD.LOG

>>>>>>>>>>>>>>>> 502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (369295616)

502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (369295360)

502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (352518400)

View original

13 replies

Userlevel 7
Badge +15

@Tenzin , do you have the agent on your clients and Media Agents?  Is there any way to configure the settings to filter out the library drives and CV directories?

I don’t know Tenable Nessus specifically, though with many AV like programs, that’s the solution: filter out our content and the issue should resolve.

Userlevel 6
Badge +13

Hi @Tenzin,

I’m not familiar with the agent itself, but I did a quick search over our customer cases and came across a similar issue just 7 days ago where the Nessus Agent was slowing down the backup of SQL. It looks like the other Nessus issues actually cause errors rather than slow down performance.

So there could be a correlation there. Is there a way to add monitoring exclusions? That could be a good place to start?

Badge +2

Thanks for the answers guys @Damian Andre and @Mike Struening. I’ve chatted with the folks managing Nessus agent and they are adamant that there isn’t a need to filter out Commvault files, the Index and DDB drives and the backup targets. I’m told all the Nessus agent does is run a scan at specific time and just reports the findings and sits idle the rest of the time. We have a meeting setup with the security admins to discuss the findings workout a solution.

@Damian Andre - We did have issue with a SQL server job where when the Nessus agent was removed the backups completed within minutes versus hours. I certain there is a correlation there but need to go through the channels to get a resolution. 

I am working our SAM at Commvault to see if they have come across this issue as well.

Userlevel 7
Badge +15

Keep us posted.  I’ve definitely heard “this service shouldn’t affect backups” more times than I can count when in fact, it did.

Userlevel 6
Badge +13

 

@Damian Andre - We did have issue with a SQL server job where when the Nessus agent was removed the backups completed within minutes versus hours. I certain there is a correlation there but need to go through the channels to get a resolution. 

I am working our SAM at Commvault to see if they have come across this issue as well.

Our log files do provide some pretty comprehensive stats when it comes to performance to try to narrow down the bottleneck, which could get you further in terms of identifying which component is specifically slowing down (maybe its the file opens, or a slowdown in the loopback networking, or network speed across the wire etc.). There is some automated performance analyses that could help.

Badge +2

@Damian Andre  Thanks, I did see that video tutorial previously and I’ve used before. I’ll go through one of the jobs affected and analyze the perf log. 

@Tenzin I did have similar issues with Tenable Nessus before (about a year ago). Had always backed up different servers, various agents just fine prior the introduction of Nessus. Tenable Nessus was implemented and I started seeing CV ports being "scanned/blocked" interrupting CV Backup connection.

While checking the logs I could see ports 8400/8403 being scanned (sort of taken by Nessus). 

Came across this doc and my actual solution was to exclude the Oracle Servers from Nessus Scan.

https://community.tenable.com/s/article/Applications-on-a-host-being-scanned-crash-while-Nessus-is-scanning-the-host

Had a ticket with CV when I was somewhat already convinced Nessus was breaking things. Inquired about some sort of documentation around that and nothing was found in CV Docs. Talked to Security and they said that Nessus as considering CV operation as a threat. And from there it was a ping pong where CV would ask me to talk to Nessus and vice versa.

I`d say, select maybe 2 / 3 different agents you are having issues and create a total exclusion from Nessus, run a backup. It is likely you will see success. Grab the logs, enable Nessus and run another backup and compare the logs, see if you can spot any of the errors below and analyze with your security team.

An exclusion may very well be needed which is what I ended up for one UNIX only. 

@@@@

Some of the logs I had seen it were like this. 

packet size (50331667)

502115 cd  05/18 15:28:59 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (1195725856)

502115 cd  05/18 15:28:59 ### [CVipcD] slRecvMsgWithTout(): socket 15: Connection reset by peer

502115 13  05/18 15:29:07 ### [CVipcD] ERROR: connCVd: accept error [130]

502115 13  05/18 15:29:07 ### [CVipcD] connCvd: listen socket will not be closed.

 

CVFWD.LOG

502117 0001 05/18 15:26:38 ######## ######## ERROR: cvfwd_validate_command(): Invalid opening message signature 0x16

502117 0001 05/18 15:26:38 TN:00003 ######## ERROR: cvfwd_dispatch_one_tunnel(): Received invalid command from fd=14. Drop the socket.

 

CVD.LOG

>>>>>>>>>>>>>>>> 502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (369295616)

502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (369295360)

502115 cd  05/18 15:26:38 ### [CVipcD] ERROR: slRecvMsgWithTout(): allocated buffer size (65536) is less than incoming packet size (352518400)

Userlevel 7
Badge +15

Amazing answer, @dude!  

Badge +2

Hi @dude - Great answer, thanks.

The test I’ve done to determine that the Nessus agent is interfering with backups is to run the library throughput trend report, with Nessus agent installed, uninstalled and then installed again, one report spanning a little over 35 days. There definitely is a upward shift when it was uninstalled and then shift downward when it was reinstalled. We’ve noticed other applications are having issues with Nessus as well. 

The link you provided from the Tenable community is very helpful and may help get the discussion started with our security team. Am I right in assuming we only need to exclude ports 8400-8403 (CV Listening ports)?

 

Thanks again!

@Tenzin I would review the doc below and make sure that Nessus never touches any of the Commvault communication ports and also add the dynamic ports.

https://documentation.commvault.com/commvault/v11/article?p=8572.htm

Badge +2

Thanks @dude 

@Damian Andre @Mike Struening  I know this is an old discussion but Damian had mentioned seeing some internal tickets around Nessus Tenable and breaking backups. I have searched the docs online and MA but could not find anything related to Commvault and Tenable. Do you guys have any internal documentation or best practices when it comes to Nessus and Commvault?

Userlevel 6
Badge +13

@Damian Andre @Mike Struening  I know this is an old discussion but Damian had mentioned seeing some internal tickets around Nessus Tenable and breaking backups. I have searched the docs online and MA but could not find anything related to Commvault and Tenable. Do you guys have any internal documentation or best practices when it comes to Nessus and Commvault?

Hey @dude - most of the cases relate to flagged security vulnerabilities (they are running outdated versions, require third-party updates like java or python, or producing false positives for components not in use etc.). The other issues are very outlier from what I can see - some strange issues with install, network, and performance - but only a handful. They don't seem to have a common thread as to the solution. I am not aware of any internal best practices for it - its not something it seems that comes up too often.

Reply