commvault.com commvault.com
Solved

Backup Cloudian S3 Bucket (source) to other library (dest)

  • 30 September 2022
  • 10 replies
  • 207 views
C
Rank
Badge +5

We need to backup S3 buckets residing on Cloudian as platform.

According to the ‘Essentials’ documentation the repository can be added in the AdminConsole under Protect > Object storage.

But in our Console of v11.28.14 there is no “Object Storage” - are we missing a license?

So I switched to the ‘Expert’ documentation, which outlines everything for the Java CommCell Console.

I was able to create the pseudo-client for the Cloud Apps and create an instance, pointing to our Cloudian system at https://s3-something

As ‘Authentication Type’ I use ‘Access and secret keys’ and prepared beforehand in the Credential Manager a ‘Cloud Account’ with vendor type ‘Cloudian HyperStore’ and filled in Access- and Secret Key.
But to my surprise, the Credential drop-down only lists credential of the type “Amazon Web Services(Access&Secret Keys)” - so of what use is the Cloudian-Type?

Long story short: despite it seems to be able to read the content of the bucket (get/dir, /fileList) it copies nothing and fails backing up.

Anyone with a similar use case with an idea what I missed?

Thank you

 

 

icon

Best answer by chelydridae 30 September 2022, 15:50

View original

10 replies

Onno van den Berg
Rank
Userlevel 7
Badge +19

Hi @chelydridae,

So to fix your first problem in where Command Center doesn't show Object Storage. Just open Command Center and go to Manage->Customization and select navigation and expand the Protect section. Make sure Object Storage is checked as I think it is still unchecked.

The Credential Manager currently only supports the management of the credentials of cloud storage devices and doesn't cover the object storage protection agent. So it perfectly makes sense.

Hope this explains/helps!

Onno
 

C
Rank
Badge +5

Hi Onno

Your explanation matches perfectly!

My ‘Object Backup’ is now visible in the WebGUI / Command Center.

I am also able to browse the content of the bucket - therefor the credentials also seem to work.

But why the backup is failing is still unclear, for I don’t see a striking and clear error message in the job-logs … opening a support case might be the better / faster track.

Onno van den Berg
Rank
Userlevel 7
Badge +19

Hi @chelydridae. Good to hear! So one the issues you might face now, assuming the access node is not also the MA, is that you are running into a connection issue that the access node cannot communicate the MediaAgent. The access node is the one who does the browsing and who pulls the data from the configured S3 endpoint, performs efficiency and send the unique blocks to the MA. I think you might be running into an issue here. 

C
Rank
Badge +5

shieet - I accidentally tapped my mouse and now my own answer is marked as ‘best’ … any way to revert?

C
Rank
Badge +5

Hi @chelydridae. So one the issues you might face now, assuming the access node is not also the MA, is that you are running into a connection issue that the access node cannot communicate the MediaAgent. ... I think you might be running into an issue here. 

Unlikely, I would say, for the access node serves also as a network gateway for a bunch of other clients (File Servers)  - and none of those shows any issue. Therefor I assume the chain ‘client - access node - MA / CommServe’ is working.

Onno van den Berg
Rank
Userlevel 7
Badge +19

What do you see in the logs ?

C
Rank
Badge +5

uuhmm… the job gets initiated, thousands of lines ‘calling GET/Files ...’

‘CCloudStorageScan::HierarchicalScan(704) - Scan completed successfully’ at the end on the access node, later ‘ SdtNetLink::authenticate() - Connection authenticated successfully’,

then some MA-Agent communication, some errors regarding  ‘CFileReaderContext::InitDirectIO(210) - Failed to read registry key 'nUseDirectIO' from 'FileSystemAgent' section of registry
…. - This backup will not use DIRECT IO

the only ‘real’ error IMHO after this: ConnectGroupInternal: setsockopt failed. Error [0xECCC005F:{CQiSocket::SetSockOpt(564)/ErrNo.95.(Operation not supported)}]

and few lines later: OBJECTOPSAPI::WorkerThread::workerLoop(1037) - backup complete flag is set and item queue is empty.

OBJECTOPSAPI::CAsyncBackup::endBackup(549) - ====== ALL WORKERS ARE STOPPED ======

then clean up and signaling: CVCloudApp::cleanup(607) - Failed to backup any files, marking job as Failed

… hard to tell what to look for …

BTW: the “Reason for job failure:” in the Job Details states:

Error Code: [72:247]
Description: Backup failed for all files, job will not be marked as completed successfully
Source: bgs-rz-prod-cvltpxy-03, Process: CvCloudBkp

but for 72:247 I found neither any information.

 

 

 

Onno van den Berg
Rank
Userlevel 7
Badge +19

Can you confirm that you are actually able to download objects using the configured credentials? If you can download object than I would definitely open a ticket. 

C
Rank
Badge +5

Hi Onno

Before I left office last Friday I did a reboot of the access node / network gateway - and over the weekend the jobs completed successfully.

A new experience for me, that Commvault needs reboots - normally it is rock-solid and self-healing without such ...

Onno van den Berg
Rank
Userlevel 7
Badge +19

Well at least it works now, but never encountered something similar. So keep an eye on it ;-) 

Reply


Terms & ConditionsCookie settings