Looking for policy advice

Hi Cheers,
Welcome to the Commvault community dude 
 

Reading your post really took me back to when I started my career – this is a pretty common challenge. I’ve worked with Commvault and a few other backup products (but won’t name names 😉). Commvault is great because it gives you a single interface to manage everything, and I highly recommend using the Command Center for this.

don’t worry about knowing all the details of every single app. What you need is an overview of the environment from the info security team or IT managers. They should have that information or can help you gather it. If you don't have that info yet, I use a simple approach with new customers: an Excel sheet with 5 columns. The first column is "Servers" where you list all servers, and the other 4 are for:
Full
Incremental
Differential
Priority (Critical, Medium, Low)
 

Then, you can ask the app owner or manager, “Is this application critical, medium, or low priority?” Trust me, they’ll appreciate it, and it’ll help you prioritize better. For example
Critical apps: Daily full backups with incremental backups every 1–2 hours
Medium apps: Full backups every 3 days with incremental backups every 5 hours
Low apps: Weekly full backups and daily incremental backups
 

Of course, you can adjust the RPOs as needed depending on what works best for your environment.
 

I hope that helps! If you need any more info or have questions, feel free to ask here in the community. There are plenty of experts around who are happy to help.

i know sometimes it's hard to understand everything at begin i hope you all the best my friend

Best Regards,
Mohammed Ramadan
Data Protection Engineer

Hi there,

I assume that by consistency group he meant ensuring everything is protected simultaneously to avoid configuration mismatches. In other words, just run the backups at the same time. As long as these are VMs, that’s easy to achieve.

Many applications follow different vendors’ backup requirements, but in most cases a full OS or VM backup is sufficient. The main exception is when databases such as MSSQL, Oracle, or MySQL are involved. In those cases a dedicated backup agent should be used. If that’s not supported, at least set up a script-based database dump. Vendors usually specify what’s required.

Databases often keep recent transactions in logs before they’re fully committed to the data files. So if your VM backup runs at 23:00, but the database’s last log flush was at 18:00, you could end up with a backup that reflects the state from 18:00 and lose a few hours of data. That’s why database-level protection is crucial for consistency. However, if the database supports Windows VSS mechanisms, such as MSSQL, then an application-consistent backup will do the job. Still, the best practice is to provide database backups with dedicated agents, tasks, or dumps whenever possible.

It seems like you’re a bit overwhelmed with the tasks at hand, but luckily, you asked in the right place. In general, application backups are a fairly straightforward topic once you get the hang of it.

Feel free to ask more specific questions here or via PM and good luck on your backup journey!

You will need your manager’s support here.  Backup Admins don’t get the credit they deserve.  We need to be jacks of all trades in order to protect everything thrown at us.  This can be daunting if you are in a very diverse environment.

You and your manager need to set expectations with other groups that data protection is going to be “best effort” unless they participate in the process.  The app owners, and especially their managers, should have a vested interested in making sure their apps are getting protected correctly AND able to test a recovery.

If you are unable to get them to help, document your attempts to get their assistance, do your best, and when they have issues in the future you can prove you asked...

Thanks,
Scott
 

I assume that by consistency group he meant ensuring everything is protected simultaneously to avoid configuration mismatches.

Apologies, consistency groups are more of a storage thing (been a storage guy for 25 years).  I think the Commvault equivalent are “protection groups”.

My understanding is, for VM environments servers in protection groups will be snapped at the same moment, ensuring that if/ when a restoration happens, they come back at the same moment in time.  If I quiesce the host/s, that much better.

The problem that I’m facing is that we have a mission critical app, like near the top and I don’t believe the application owner understands the constraints we face when trying to restore.  I spent the better part of several days uncovering details about how to backup this particular application (I won’t say which out of an abundance of caution for operational security), but I know that I cannot spend this kind of time on every application we have to ensure that we can meet RTO/ RPO requirements.

If I do it his way, there’s a chance that, at best a restoration could take days rather than hours and at worst we could loose days/ weeks of information or worse.

So while it wouldn’t be my fault, the fact is, it would be my problem.  I won’t spend this level of effort for every application if the owner isn’t interested in doing their homework, but in this case I felt I had to.  Which is, of course, what lead me to ask how others deal with issue, not as a technical question, but an operational question.

However, if the database supports Windows VSS mechanisms, such as MSSQL, then an application-consistent backup will do the job. Still, the best practice is to provide database backups with dedicated agents, tasks, or dumps whenever possible.

I’m not sure if this is what you’re saying, but, if I do a VM backup of a DB server using VSS, will that allow me to recover it to that point in time?  Or are you saying the DB agents go an extra step of keeping the Logs up to date (where the VM backup won’t force the DB to “commit logs” (my language) but the DB agents for SQL and MySQL will?

That would make sense as plans have log backups enabled by default.  I don’t think you can do DB log backups, at least making sure that logs and data are consistent in time like you mention, correct?

I really took over backups about 6 months ago, but I’ve been involved at some level for just over a year.

My environment is a mess after years of neglect.  I’m trying to balance environment fixes and optimizations and ongoing/ future planning.  I very much appreciate the comments and suggestions.

Thank you

don’t worry about knowing all the details of every single app. What you need is an overview of the environment from the info security team or IT managers.

There are extreme challenges at my site.  I wish it was that easy.  I’ve ever worked anywhere, where I had to do so much work, to get to do my work…..

  Throughout my 30 year IT career I’ve found the best way to get management/ peers to take action, when it’s become obvious there is little interest or incentive to do so, is not to ask what is wanted, but to take control of the situation and tell them what you (me) are going to do and expect their participation.  Then others tend to give you their thoughts.

That’s why I’m here, because I want to ensure I’m not asking anything out of the ordinary.

I’m happy to shoulder some or even most of the effort, but there are applications that have specific requirements and at the end of the day, I want management to understand that the buck stops with the application owners.  Most applications are pretty straight forward and I’ll take the lead, document the process, share it with the team, and it will be up to the application folks to make sure my procedures are consistent with their app.

For multi-server, multi-site, HA, clusters things can get more challenging.   I’ve found in some cases we were backing up way more than we had to because we don’t understand the way an app works or what servers work together or how they work.

I don’t see how we can meet requirements if the app owners don’t take the time to ensure their apps are being backed up right.

And if you are thinking; Testing!…..I’ll just “say” 😬😫😒  I’m working on it.

Do you have a formalised Backup and Recovery Policy?

We do not. I walked into a bit of a mess at this site, and I’m trying to clean it up.

Slightly more context: As I said already in another response, I am an IT engineer with decades of Storage and SAN experience, but I only started learning Commvault (and backups) about 3 years ago, and only half of that at the engineering level. I am quickly learning that policy is as hard or harder than engineering because I need buy-in from the rest of the team, which is nearly impossible. This means I have to explain to management why they should care and encourage policy changes, but that risks uncovering serious issues and while some of you are thinking that what I should be doing, I assure you it is not that simple, there is an entrenched bureaucracy here and if you rock the boat too hard you risk falling out of you catch my drift.

My predecessors did not rock the boat, and as a result, the boat is falling apart.  That said, I want to fix it, but I have to do so carefully.

To make things even more fun, my company manages the customers resources (managed hosting). Thus, the resources belong to the customer, and as such, the customer should ideally set policy or at the very least, should be involved in and approve all requests for policy and changes (either administrative or engineering) before they happen (I’m not speaking of routine maintenance changes, which are approved as part of a change control process). However, the bureaucracy here is the worst I’ve ever seen in my 30+ year career. The only policy document I have been given was written in 2012, and it’s so technologically out of date, it’s useless as a guide.

I spend 60-70% of my day just putting out the proverbial fires, then I try to use the rest to make changes in Commvault settings and policy that will prevent those fires in the first place.

The OP is my reaching out to folks who have the experience dealing with these kinds of issues.

The good news is that there are a series of tech refreshes planned, of which one has already taken place. As part of that, I was able to start fresh as previous backups are going to age out in place. This won’t fix the policy issues, but the refresh that has taken place has reduced the daily maintenance workload on that Commcell by 80-90% I have 4 other Commcell's, all of which are in rough shape.

Given the lack of internal guidance, I’ve tried to keep things as simple as I can, but there are features and performance that are almost certainly being left on the table. My goal is to get past the tech refreshes and to take advantage of the reduced time spent on daily maintenance to circle back and address efficiency, performance, security posture and improvements and push for documented and approved requirements. With that said, all advice is welcome.

-Cheers

I wish I could tell you that it gets better, but in my experience, it often takes a significant event before Backup and Recovery becomes anything other than an ugly afterthought. So, I don't recommend waiting for buy-in. 

I think a formalised Backup and Recovery Policy is going to be a fairly low effort, but valuable starting point. It may not assist in putting out the existing fires, but it may assist in reducing the number of new fires being lit. 

There are plenty of Backup and Recovery Policies published online - which should provide a reasonable template. I also highly recommend reviewing and incorporating guidance from trusted and reputable sources, such as ACSC / CISA / NCSC / your country's equivalent Cyber Security Agency, etc.

A foundational level document that clearly defines responsibilities, provides bare minimum requirements, and incorporates industry best practices and guidance from reputable sources is going to be hard to repudiate.

Thank you for sharing.  And I’ve have been very interested in trying to find backup and recovery policies from other organizations and trying to build a framework on top of a proven and tested policy.  Any links to resources would be appreciated.