What Is VMware Changed Block Tracking (CBT) and How Does It Work?
VMware provides APIs to simplify data protection in enterprise virtual environments with features that enable agentless backups and improve the efficiency of incremental VM backups. Incremental backups help save a significant amount of storage space compared with full backups
When it comes to modern VMware backup solutions, most rely on VMware’s APIs and use the Changed Block Tracking (CBT) technology to facilitate and speed up creating incremental backups of virtual machines. In this blog post we explain Changed Block Tracking, including the working principle and practical applications.
What Is Changed Block Tracking?
Changed Block Tracking (CBT) is a native VMware feature that logs changed blocks in a tracking file. CBT is part of VMware vSphere Storage APIs – Data Protection framework first released in version 4.0, which allows third-party data protection applications to leverage CBT to perform more efficient backup and replication.
CBT works on the ESXi Storage Stack level and allows third-party backup solutions to back up only the blocks of data changed since the last backup, that is, create incremental backups, or the blocks in use instead of the full VM. This allows you to dramatically reduce the amount of backup data, especially in large environments with hundreds of VMs, and speed up the backup process.
VMware CBT significantly reduces the amount of time the backup software takes to perform incremental backups because it tracks changes in a special log file. Third-party backup solutions cause more CPU load on an ESXi server if they track changes without CBT by using other methods. Thus, CBT also optimizes the performance of an ESXi server when backing up data.
CBT is also used by VMware’s Storage vMotion process that allows virtual machine disk files to be moved from one datastore to another while it is running.
How VMware Changed Block Tracking Works
The CBT feature works by comparing the last and current changeset IDs and finding the disk sectors that were altered between these changeset IDs. VMware CBT also identifies all Virtual Machine File System (VMFS) blocks across a virtual disk that are in use. The tracking is performed by the hypervisor itself and takes place outside the VM.
CBT Requirements
Requirements for Changed Block Tracking include:
- VMware vSphere (ESXi 4.0 or higher)
- Version 7 virtual hardware or higher
- Local VMFS datastores, NFS or iSCSI datastores
- Thick or thin provisioning
- No RAW disk mappings. Raw Device Mapping (RDM) in Physical Compatibility Mode is not supported.
- A VM must not have snapshots to start using VMware CBT
How to Enable Changed Block Tracking
By default, CBT is disabled as it consumes a small amount of host CPU time. But for a third-party backup solution like NAKIVO Backup & Replication, CBT must be enabled to let the product perform fast and reliable incremental backup.
CBT is enabled in a virtual machine’s advanced configuration settings:
- Open VM settings. You can do it in VMware vSphere Client by right-clicking a VM and selecting Edit Settings.
- In the Edit settings window, click the VM Options tab.
- Click the Advanced option category to expand the settings and click Edit Configuration in the Configuration Parameters line. The Configuration Parameters window for the VM opens, and it contains multiple lines, and each contains a parameter name and value.
- CBT activation is determined by the following line in the .vmx configuration:
ctkEnabled = "TRUE"
Also, for each disk that CBT is enabled, an entry will be included:
scsix:x.ctkEnabled = "TRUE"
These parameters must be set to enable Changed Block Tracking.
Note: To disable CBT, change TRUE to FALSE in the ctkEnabled and scsi0:0.ctkEnabled lines (use the appropriate number of the SCSI device).
When CBT is enabled on a VM, the VMkernel creates an additional file with -ctk.vmdk in the VM directory that is used to store the mapping of virtual disk blocks. This file is related to the respective .vmdk disk and contains a list of blocks changed in this virtual disk between the certain time identified in the last changeset ID and the actual state. The size of the CTK file that gets created stays the same and does not grow beyond the original size unless the size of the virtual disk is increased. The size of the CTK file is approximately 5 MB for every 10 GB of VM disk size.
Most backup solutions will use the VMware SDK method to programmatically enable CBT for a VM and use a stun/unstun cycle, which includes either power on or off, suspend/resume, or create/delete a snapshot. During a stun/unstun cycle, a special change tracking filter is inserted into the VMware storage stack for the particular VM, which allows CBT to instantiate on the VM.
How to Check whether CBT Is Enabled
In cases where you didn’t enable CBT manually, you can check whether backup software enabled CBT to perform VM backup. The steps to check CBT settings are similar to the steps for enabling this feature. But here we’ll cover a slightly different method using a .vmx configuration file.
To check whether CBT is enabled for the VM, do the following:
- Open VMware vSphere Client.
- Find the needed VM and make sure it is powered off.
- Go to the VM folder on a datastore.
- Find a VM configuration file (.vmx).
ctkEnabled entry. The entry’s value has to be “TRUE”. If it is not, then change the value to “TRUE” to enable CBT for the VM.
With the enabled CBT, each virtual disk is accompanied by a xxx-ctk.vmdk file, where “xxx” is a virtual disk name. If you see these CTK files, it indicates the working CBT feature.
In some cases, such as inappropriate host shutdown or power failure, CBT can lose track of changes while the VM is powered on. This may cause CBT to reset, which means that all previously collected CBT data will be flushed and the new data will start being collected from the moment of reset. If you run a VM backup after the CBT reset, a full backup is made. After making a full backup and tracking new changes, you can then perform incremental backups again.
Using VMware CBT in NAKIVO Backup & Replication
NAKIVO Backup & Replication is a universal data protection solution that can back up, recover and replicate virtual machines in VMware vSphere. To speed up incremental backup performed with the NAKIVO solution, CBT is enabled by default in any new backup job.
There are several cases when NAKIVO Backup & Replication, like any other vSphere backup solution, cannot use VMware CBT:
- VMware CBT is not supported by a VM: Possible cause – ESXi host is lower than v4.1, or Virtual Hardware version is lower than v7.
- VMware CBT data was not provided: Possible cause – ESXi host power outage and CBT lost track of VM incremental changes.
In the additional Change tracking settings, you can find the actions the solution should take when it encounters an error while using CBT.
Let’s take a closer look at these On error actions:
- Switch to proprietary method: NAKIVO Backup & Replication proprietary change tracking method will be used, which is slower than VMware CBT and consumes more of the ESXi host’s resources.
- Reset CBT: CBT will be reset for the CBT in case of CBT failure.
- Fail VM processing: VM processing will be stopped.
Also, you can specify when to apply the selected On error action:
- Immediately: On error action is applied immediately (except for the Reset CBT action).
- On the next retry: On error action is applied on the next retry.
- On the last retry: On error action is applied on the last retry.
You can also select the Double-check changed blocks provided by CBT checkbox. In this case, data provided by VMware CBT will be additionally checked against NAKIVO Backup & Replication proprietary change tracking. This approach is more reliable, but may be slower than the regular VMware CBT.
Conclusion
VMware Changed Block Tracking is a powerful feature that makes the backup process faster and reliable in every respect. The technology is well tested and heavily utilized in native VMware technology such as Storage vMotion.
VMware CBT is an integral part of modern backup solutions such as NAKIVO Backup & Replication that provide efficient backups of VMware environments without additional load on hosts.
