commvault.com commvault.com
Solved

Can it be done : NDMP (Isilon) via secondary IP on MediaAgent using DIP

B
Rank
Userlevel 1
Badge +4

Scenario - Three way NDMP backup from Isilon. The Isilon is using an IP pool of x.x.x.x (eg 192.168.1.1 - .4) for NDMP operations The media agent has a primary IP of y.y.y.y (eg 192.168.2.1). I have put a second IP on the media agent that is in the same subnet as the Isilon, eg 192.168.1.250. This second IP is on a different nic from the primary IP. When I do a backup of the Isilon the job fails due to “network communication error”. When I look at the logs the problem seems to be that the nasbackup command is still using the media agents original primary IP and not the new secondary IP.

 

I have tried setting up DIP/Backup Network pair but the job still fails and the logs show that the media agent primary IP is still being used. Is it possible for the NDMP backup to use a specified ip address for the MediaAgent that is not the primary IP? Based on a bit of digging around in the forum it seems that DIPs are not used for NDMP traffic management since the NDMP server (Isilon) is not really a CommVault client. So if DIP is not the right knob what is? Is it even possilble?

 

Cheers,

Ron

icon

Best answer by Mike Struening RETIRED 15 July 2022, 21:14

View original

15 replies

T
Rank
Userlevel 2
Badge +5

Hi @bRonDoh can you go to the array management, I would not be surprised if your Isilon was registered there using the “old” IP. Can you confirm.

C
Rank
Userlevel 5
Badge +14

Hello @bRonDoh 

You can put a DIP between the MediaAgent and the Filer but note we can only control what IP address is used on the MediaAgent and what IP address it talks to for the filer. Outgoing communication from the Filer is based on the interface roles. 

B
Rank
Userlevel 1
Badge +4

Hi @tph , Yes the Isilon is defined as an array. However Isilon is registered with an IP/Hostname that is on a different ip/subnet from both the NDMP server pool of IP’s and the MediaAgent IP. This stuff can get very complicated… So let me flesh it out a bit more and hopefully a bit more clearly.

The CommServe (aka the DMA) : a vm with IP 192.168.2.2

The MediaAgent : a physical box with primary IP of 192.168.2.1 and a new secondary IP of 192.168.1.250

The Isilon management IP : 192.168.3.1

The Isilon NDMP IP Pool : 192.168.1.1 - .4

(Note all the above IPs are not the act

ual IPs, but they convey the topology wrt subnets used.)

IF I understand the traffic flow - the CommServe sends NDMP instructions / options to the Isilon on the Isilon Management IP. The Isilon initiates and NDMP backup per the instructions from the CommServe. One of those instructions from the CommServe is the hostname/ip addres of the MediaAgent that the Isilon NDMP server is supposed to talk to. The Isilon initiates the tcp/ip  session from an IP adress from its defined pool of NDMP preferred IPs.

The CommServe only ever tells the Isilon to use the primary IP of the MediaAgent. This causes the comms to fail if the MediaAgent has an IP on the same subnet as the Isilon NDMP preferred IP subnet - this is due to asymmetric routing.

Looking at the Array Management configuration I do not see any knob that would let me define the use of a secondary IP on the MediaAgent for NDMP comms. Maybe it is embedded/implicit in the configuration of the array since it was done when the MediaAgent had only one IP. Perhaps if I recreated the array now that the MediaAgent has a secondary IP it would allow me the option of picking the IP to use for NDMP backups - ie either the MediaAgents primary or secondary IP…

Thanks,

Ron

B
Rank
Userlevel 1
Badge +4

Hi @Collin Harper - Yes I have tried setting up DIP/Backup Network between the NDMP server (ie the NDMP client from the perspective of the CommServe) and the MediaAgent, but it does not work. The IP address that the DMA tells the Isilon NDMP server to use is always the MediaAgents primary IP.

I suspect that NDMP comms are not affected by DIPs.

Cheers,

Ron

C
Rank
Userlevel 5
Badge +14

@bRonDoh 

When no DIPs are being used, we simply use whatever is the first IP address as per the OS’s binding order. We also do not tell the Isilon what IP address to talk to for us. This is why DIPs are sometimes needed.

B
Rank
Userlevel 1
Badge +4

@Collin Harper  Sorry but I do not understand. Without a DIP the Isilon talks to the MediaAgent just fine - it initiates the NDMP connection from it’s pool of ndmp preferred IPs to the MediaAgent IP. The DMA (CommServe) tells the Isilon the IP of the MediaAgent it should talk to. So we do “tell the Isilon what IP address to talk to for us”. Right? How else would the NDMP server (the Isilon) know where to send the NDMP data if the DMA (CommServe) did not tell it the IP of the MediaAgent?

So the problem is that even though I have defined a DIP between the Isilon NDMP preferred IP pool (actually the entire subnet) and the MediaAgent’s new secondary IP (which again is in the same subent as the Isilon ndmp preferred IP pool) the network connection fails.

It fails because, as the logs show, the NDMP server is NOT given the MediaAgents secondary IP, rather it is given the original primary IP. This causes an asymmetric routing issue and thus the comms fail. This also happens when a DIP is NOT defined and the MediaAgent’s secondary IP is up - asymetric routing due to the DMA telling the Isilon to use the MediaAgent’s primary IP when the MediaAgent has a secondary IP in the same subnet as the Isilon NDMP preferred-ip pool. If the DMA told the Isilon to use the secondary IP of the MediaAgent then there would be no problem. How do we get the DMA to tell the Isilon to talk to the secondary IP of the MediaAgent?

Cheers,

Ron

C
Rank
Userlevel 5
Badge +14

@bRonDoh

Correct. A DIP is not a pre-requisite for NDMP backups, but I may have mis-understood your previous statements, thinking you are referring to general connectivity without the use of a DIP.

The CommServe doesn’t tell the Isilon what IP address to talk to for the MediaAgent unless there is a DIP in place. Without a DIP this would occur via DNS and whatever IP address the MediaAgent resolves as.

If you are having issues with the backup I would suggest getting a Support case opened to verify the DIP configuration is set correctly and to investigate the cause of the issue.

B
Rank
Userlevel 1
Badge +4

@Collin Harper Yeah, I am trying to get the Isilon to send the ndmp data to the MediaAgent on a new secondary IP address on the MediaAgent. The DMA provides a hostname/ipaddress the Isilon as to where the ndmp session is to be built. Whether I use a DIP or do not the DMA always tells the Isilon where to send the ndmp traffic - it has to because the Isilon has no idea what the hostname/ipaddress of the MediaAgent is.

The logs show that with and without a dip the DMA is telling the Isilon to use the original/primary hostname/ipaddress of the MediaAgent. In other words - the DIP is ignored by the DMA and the Isilon NDMP server is given the ‘wrong’ hostname/ipaddress.

Cheers,

Ron

B
Rank
Userlevel 1
Badge +4

Just adding some of the log info showing the NDMP control options being set by the DMA - specifically the hostname/ipaddress of the MediaAgent for the Isilon to talk to.

 

4108  d74   06/16 14:42:23 47686 Scheduler  Phase [4-Backup] (0,0) started on [s01.ok.ubc.ca] in [0] second(s) - NasBackup.exe -j 47686 -a 2:69 -t 1 -d s01.ok.ubc.ca*s01*8400*8402 -c clokadmis01.ok.ubc.ca  -jt 47686:4:1:0:0:51936  -idxma s01.ok.ubc.ca*s01*8400*8402  -lastMA :  -numstreams 2
4108  d9c   06/16 14:42:23 47686 Servant    Reg [Control] received. Client [s01] plattype = 4. Token [47686:4:1:0:0:51936]
4108  d9c   06/16 14:42:26 47686 Servant    Reg [NotControl] received. Client [s01] plattype = 4. Token [47686:4:1:0:0:51936]
4108  213c  06/16 14:42:26 47686 Servant    Reg [NotControl] received. Client [s01] plattype = 4. Token [47686:4:1:0:0:51936]
4108  d9c   06/16 14:43:44 47686 Scheduler  Set pending cause [Client [clokadmis01.ok.ubc.ca] was unable to connect to the tape server [s01] on port [50689]. Please check network connectivity.]::Client [s01] Application [NasBackup] Message Id [654311848] Error code [39:424] RCID [0] ReservationId [0].  Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
4108  213c  06/16 14:43:44 47686 Scheduler  Ignored pending cause [Client [clokadmis01.ok.ubc.ca] was unable to connect to the tape server [s01] on port [58665]. Please check network connectivity.]::Client [s01] Application [NasBackup] Message Id [654311848] RCID [0] Reservation Id [0].  Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
4108  d9c   06/16 14:44:50 47686 Scheduler  Got stream complete message for RCID [104331], status [2]. [s01/cv02.ok.ubc.ca/SockIP(192.168.2.1)/192.168.2.1:0/0 NasBackup(15180:d11df740)]
4108  213c  06/16 14:44:50 47686 Scheduler  Got stream complete message for RCID [104332], status [2]. [s01/cv02.ok.ubc.ca/SockIP(192.168.21)/192.168.2.1:0/0 NasBackup(15180:d11df740)]

Now, I am guessing here that the hostname/ip address I’ve highlighted in the above log (JobManager.log)  is the bit that the DMA sends to the Isilon NDMP server. The NDMP server on the Isilon has no idea where to send the backup data - there is no configuration of that on the Isilon. You can configure a pool of IPs on the Isilon that are used by the Isilon as the source IP for NDMP backup, but you do not configure a destination IP. The destination IP has to be provided to the Isilon by the DMA (Data Managemnt Agent, ie the CommServe). From the log it appears that the CommServ provides the destination hostname/ip in the NasBackup.exe command in the paramter “-d s01.ok.ubc.ca*s01*8400*8402” or maybe not. I could not find any documentation of this command - not in CommVault Docs, not in man pages, not in the built in command help. So this may not be where/how the NDMP destination IP is set. However I am pointing out that no matter whether a DIP is defined or not there is no change in the hostname/ipaddress that is recorded in the log. Maybe I am looking in the wrong log?

Cheers,

Ron

 

C
Rank
Userlevel 5
Badge +14

@bRonDoh 

I want to correct my previous statement. We do not tell the Isilon what IP address to talk to. Since there is no Commvault Agent on it, we cannot control it. We simply bind out processes to the desired IP address.

Again, I would recommend opening a Support case to make sure everything is configured correctly.

Thank you.

B
Rank
Userlevel 1
Badge +4

@Collin Harper I have opened a ticket with support. Thanks to all for taking the time to try and assist.

 

Cheers,

 

Ron

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

@bRonDoh , can you share the case number?

https://www.linkedin.com/in/michael-struening
B
Rank
Userlevel 1
Badge +4

Sure - it is 220617-515. I have a follow up zoom session today. Will post the outcome when there is one.

 

Cheers,

Ron

Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

You’re awesome, thanks!!

https://www.linkedin.com/in/michael-struening
Mike Struening RETIRED
Rank
Userlevel 7
Badge +23

Sharing case solution:

DIPS were not getting honored between Mediaagent and NDMP Client
Case was escalated to development
Ran operation to allow firewall config with NDMP client
Found an old name in the MA's registry which was affecting the push of the DIP
Once this was corrected, NDMP jobs were able to be successfully run and used the correct interface

https://www.linkedin.com/in/michael-struening

Reply


Terms & ConditionsCookie settings