VADP or Agent Based Backups

In light of ongoing bugs with VMware’s API for Data Protection (VADP), I figured it worth re-visiting the topic of VADP or Agent Based backups.

VADP gives backup products the ability to kick off snapshots and use Changed Block Tracking (CBT) to allow incremental style backups which improve the efficiency of backup solutions by reducing the impact (performance, think storage, network and compute overheads) and duration (backup window).

But the problem is, there has now been several instances of VADP bugs in recent years which has meant incremental backups have lacked integrity due to the changed blocks not being correctly reported.

Here is a list of some of the VADP related issues/bugs:

  1. Backups with Changed Block Tracking can return incorrect changed sectors in ESXi 6.0 (2136854)
  2. Backing up a virtual machine with Changed Block Tracking (CBT) enabled fails after upgrading to or installing VMware ESXi 6.0 (2114076)
  3. Changed Block Tracking (CBT) on virtual machines (1020128)
  4. Enabling or disabling Changed Block Tracking (CBT) on virtual machines(1031873)
  5. Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201)
  6. When Changed Block Tracking is enabled in VMware vSphere 5.x, vMotion migration fails with error: The source detected that the destination failed to resume (2086670)
  7. QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled (2090639)

From the above (albeit a limited list of VADP related issues) we can see that there are issues related to integrity of VADP CBT as well as operational considerations (limitations) when using CBT, such as not being able to Storage vMotion and having vMotion operations fail.

So while VADP in theory has its advantages, should it be used in production environments?

At this stage I am highlighting the risks associated with using VADP with customers and where required/possible mitigating the issue.

But what about good ol’ agent based backups?

Agent based backups have a bad rap in my opinion mainly because of 3-Tier solutions and the fact backup windows take a long time due to the contention in the storage network, controllers and back end disk.

Now people ask me all the time, how can we do backups on Nutanix? The answer is, you have numerous (very good) options without using VADP (or for non vSphere customers).

Using a product like Commvault, In-Guest Agent’s can be deployed and managed centrally, removing much of the administrative overhead (downside) of agent based backups.

Then by configuring incremental forever backups, Commvault manages the change block tracking (regardless of hypervisor) and can even do source side deduplication and compression before sending the delta’s over the network to the Commvault Media Agent (ie.: The backup server).

Now since all new write I/O is written to Nutanix SSD tier, it is very likely that all changes will still be in the SSD tier when a daily incremental backup is started meaning the delta’s will be quickly read and send over the network. Why is this solving the problems of 3-Tier i discussed earlier, well its thanks to data locality and the fact Nutanix XCP is a highly distributed platform.

Because each Nutanix node has a local storage controller with local SSD, AND critically, Data Locality writes new data to the node where the VM is running, most data (under normal situations) will be read locally (without traversing a NIC/HBA or the storage network). This means there is no impact on other nodes from the backup of VMs on each node.

Due to these factors, the only traffic traversing the IP network to the backup server (Commvault Media Agent in this example), are the delta changes in a compressed and deduplicated format.

So a Commvault Agent Based backup solution on Nutanix XCP, on any hypervisor, avoids the dependancy on hypervisor APIs (which have proven in several cases not to be reliable) and ensures backup windows and the impact of backup jobs is minimal due to intelligent incremental forever style backups running on an intelligent distributed storage fabric.

In-Guest agent based backups may just be making a comeback!

Note: In y experience, Agent based backups typically provide more granularity/flexibility compared to VADP backups, for specifics speak with your preferred backup vendor.

Oh BTW, did I mention Nutanix XCP supports Commvault Intellisnap for storage level snapshots on the Distributed Storage Fabric… again just another option for Nutanix customers wanting to avoid further pain with VADP.

Competition Example Architectural Decision Entry 1 – TSM backup configuration for PureFlex environment?

Name: Ash Simpson
Title: Virtualization Architect
Company: IBM
Twitter: @Yipikaye1
Profile: VCP4

Problem Statement

Which is the ideal method for TSM backup for PureFlex environment? LAN free backup or LAN based backup or both?

Assumptions

1. IBM PureFlex hardware is used

2. Physical TSM server exists within PureFlex.

3. External (Virtual) Tape Library available on PureFlex SAN Fabric.

Constraints

1. Customer has selected PureFlex Infrastructure as hardware platform
2. IBM storage must be used – Storwize V7000 and IBM DS8000
3. ProtecTier VTL available and should be used

Motivation

1. Flexibility of Choice based on specific application requirements requirements.
2. The configuration to be deployed has the capability to support both.
3. LAN free backup is getting popular option in the industry.
4. LAN free backup negates the need for large backup windows.
5. PureFlex V7000 allows for FlashCopy Manager (FCM)
6. FCM is application aware for many critical Intel workloads such as SQL and Exchange.
7. All Backup I/O is retained within a single PureFlex Chassis

Architectural Decision

Deploy LAN free backup and LAN based backup infrastructure in PureFlex environments with LAN free backup via TSM for VE and FlashCopy Manager as the default. Should a particular application have the requirement for LAN based backup, the infrastructure can support it.

Host the Physical TSM server and an ESXi Host with the TSM for VE server (via affinity rule) in the same Chassis.

For the few servers requiring LAN based backup agents use affinity rules to prefer ESXi hosts in the same PureFlex chassis as the TSM server.

Alternatives

1. Provide LAN based backup only

2. Provide LAN free backup only.

Justification

1.Better utilization of network bandwidth in LAN free backup.
2.Improved performance for backup and restore operations is possible in LAN free backup.
3. LAN based backup is still required by certain applications, hence it is recommended to retain this feature.
4. Hosting TSM server in same chassis as proxy/agents prevents North/South network I/O.
5. FlashCopy Manager will reduce backup times by creating application aware snapshots on the storage array.

Implications

1. The hardware infrastructure will have to be configured for both LAN free and LAN based backup. For LAN free backup the SAN fabric in PureFlex system will be used for backup environment. The backup server transfers data from its storage directly to the tape device via FC.

2. Fibre Channel ports needs to be dedicated for backup traffic

3. Separate Zones needs to be configured in the Fibre Channel Switch module environment for backup traffic.

Back to Competition Main Page or Competition Submissions