Openstack incremental backup
-
Hello
I tried to test incremental backup on openstack with libvirt strategy . However I receive following error[b7277ddb-a4d1-4284-8783-30ba407d38bd] Unable to create incremental backup for a non-excluded QCOW2 driver [UUID: 4afff436-71cc-4a30-8ca5-a9406e6350aa]
This is the log for first full backup
Task Assignment Utility: EXPORT task 162d6b39-42fa-418b-af16-41777176a8c0 has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5) [305/1825] ==> node1/vprotect_daemon.log <== [2024-03-22 15:07:03.747] INFO [pool-5-thread-56] ITask.lambda$startTask$1:39 [] Starting Task 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01:00 [2024-03-22 15:07:03.781] INFO [pool-5-thread-56] ExportTask.run:115 [162d6b39-42fa-418b-af16-41777176a8c0] Starting export of: VM cirros15 (551df74b-dd5a-4b64-b5e2-0fe523d47da3) from 10.150.6.73 (OpenStack)... [2024-03-22 15:07:03.781] INFO [pool-5-thread-56] BackupProviderUtil.testConnection:238 [162d6b39-42fa-418b-af16-41777176a8c0] Testing connection with backup provider: backupdest1 [2024-03-22 15:07:03.782] INFO [pool-5-thread-56] FileUtil.removeLocalTestFiles:645 [162d6b39-42fa-418b-af16-41777176a8c0] Cleaning up after Connectivity Test [2024-03-22 15:07:03.800] INFO [pool-5-thread-56] ExportImportUtils.getVmExportImportModeByHvManager:43 [162d6b39-42fa-418b-af16-41777176a8c0] OpenStack export import strategy by hypervisor manager: SSH_TRANSFER [2024-03-22 15:07:03.812] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140 [162d6b39-42fa-418b-af16-41777176a8c0] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81' [2024-03-22 15:07:04.278] INFO [pool-5-thread-56] SshFingerprintCache.getOrCreateTrustedCertificateWithDecodedChain:55 [162d6b39-42fa-418b-af16-41777176a8c0] Getting ssh fingerprint for TrustedCertificateChainId(sourceId=4d54f698-c559-4261-8e5c-46b23ea99bf7, sourceType=HYPERVISOR, chainType=SSH_HOST_FINGERPRINT) and task: 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15: 06+01:00 to 2024-03-22T20:06+01:00 [2024-03-22 15:07:04.283] INFO [pool-5-thread-56] SshUtil.getConfig:17 [162d6b39-42fa-418b-af16-41777176a8c0] StrictHostKeyChecking = yes [2024-03-22 15:07:04.283] INFO [pool-5-thread-56] SshUtil.getConfig:17 [162d6b39-42fa-418b-af16-41777176a8c0] StrictHostKeyChecking = yes [2024-03-22 15:07:05.051] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140 [162d6b39-42fa-418b-af16-41777176a8c0] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'OpenStackHypervisorManager.java#listVmDisks:204' [2024-03-22 15:07:05.586] INFO [pool-5-thread-56] ExportFull.export:30 [162d6b39-42fa-418b-af16-41777176a8c0] Performing full backup [2024-03-22 15:07:06.280] WARN [pool-5-thread-56] SshConnector.chownRemote:424 [162d6b39-42fa-418b-af16-41777176a8c0] Could not chown file /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa [2024-03-22 15:07:07.372] INFO [pool-5-thread-56] SshConnector.chownSnapshots:398 [162d6b39-42fa-418b-af16-41777176a8c0] Trying chown snapshots [2024-03-22 15:07:07.661] WARN [pool-5-thread-56] SshConnector.chownRemote:424 [162d6b39-42fa-418b-af16-41777176a8c0] Could not chown file /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa-vProtect-c08f07b0-2373-48c7-b212-8d57a755b15d [2024-03-22 15:07:08.240] INFO [pool-5-thread-56] OpenStackHypervisorManager.exportImage:291 [162d6b39-42fa-418b-af16-41777176a8c0] Start exporting image [2024-03-22 15:07:08.240] INFO [pool-5-thread-56] OpenStackImageApi.fetchAsVmDisks:45 [162d6b39-42fa-418b-af16-41777176a8c0] Fetching images as vm disk in server id: 551df74b-dd5a-4b64-b5e2-0fe523d47da3 [2024-03-22 15:07:08.248] INFO [pool-5-thread-56] ImageExporter.export:32 [162d6b39-42fa-418b-af16-41777176a8c0] isDownloadImageFromGlance : false [2024-03-22 15:07:08.249] INFO [pool-5-thread-56] OpenStackExcludedImageExporter.export:23 [162d6b39-42fa-418b-af16-41777176a8c0] Image excluded from backup - skipping export [2024-03-22 15:07:08.632] INFO [pool-5-thread-56] LibvirtHypervisor.exportDisks:576 [162d6b39-42fa-418b-af16-41777176a8c0] Disks to export: [file/disk: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa] [2024-03-22 15:07:09.364] INFO [pool-5-thread-56] LibvirtHypervisor.prepareSnapDownload:1256 [162d6b39-42fa-418b-af16-41777176a8c0] Adding snapshot file to the download list: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa [2024-03-22 15:07:09.711] INFO [pool-5-thread-56] LibvirtHypervisor.downloadFromTransferList:666 [162d6b39-42fa-418b-af16-41777176a8c0] Downloading VM data: cirros15 [2024-03-22 15:07:10.017] INFO [pool-5-thread-56] SshConnector.transfer:429 [162d6b39-42fa-418b-af16-41777176a8c0] Starting transfer [2024-03-22 15:07:10.295] INFO [pool-5-thread-56] SshConnector.transfer:453 [162d6b39-42fa-418b-af16-41777176a8c0] Downloading: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa -> /vprotect_data/cirros15__551df74b/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa [2024-03-22 15:07:17.472] INFO [pool-5-thread-56] ProtectedEntityUtil.collectFileSystemInfo:198 [162d6b39-42fa-418b-af16-41777176a8c0] Collecting file system information... [2024-03-22 15:07:35.538] INFO [pool-5-thread-56] ITask.lambda$finishTask$2:71 [162d6b39-42fa-418b-af16-41777176a8c0] Finishing 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01:00 Task with message "Backup of Virtual Machine: cirros15 (size: 1 GiB) exported in: 31s." ==> api/api.log <== [2024-03-22 15:07:40] INFO [__ejb-thread-pool14] {} TaskLoadBalanceTimer.assignTasks:263 Task Assignment Utility: STORE task fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5) ==> node1/vprotect_daemon.log <== [2024-03-22 15:07:43.784] INFO [pool-5-thread-56] ITask.lambda$startTask$0:38 [162d6b39-42fa-418b-af16-41777176a8c0] Starting Task fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c: [Store] from 2024-03-22T15:07:35+01:00 to 2024-03-22T20:07:35+01:00 as a result of 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01 :00 [2024-03-22 15:07:43.834] INFO [pool-5-thread-56] StoreTask.run:80 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Files to store: [3a1b0f66-c630-4970-8cc7-a80a3c8f6282: /vprotect_data/cirros15__551df74b/18ea56d6-f5d0-4132-ad6b-e35aeed97890.img - null (modified: null | format: EXCLUDED_DISK | vmDiskGuid: 5175b419-687f-455e-b462-27c17c5ccfa3), 02 fe3a0a-e94f-4aa7-8be9-d31a52fd0a97: /vprotect_data/cirros15__551df74b/4afff436-71cc-4a30-8ca5-a9406e6350aa.json - null (modified: null | format: DISK_META_JSON | vmDiskGuid: 694e11e6-6044-43ce-a5bf-a85d3b55af12), a4d14c95-53ab-4b09-b836-eabc16331882: /vprotect_data/cirro s15__551df74b/551df74b-dd5a-4b64-b5e2-0fe523d47da3.json - null (modified: null | format: VM_META_OPENSTACK | vmDiskGuid: null), 849e3f39-57c9-4d04-bb62-a9041915c583: /vprotect_data/cirros15__551df74b/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa - null (modified: null | fo rmat: RAW | vmDiskGuid: 694e11e6-6044-43ce-a5bf-a85d3b55af12)] [2024-03-22 15:07:43.834] INFO [pool-5-thread-56] StoreTask.run:84 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Storing backup files (Virtual Machine: cirros15) in File System... [2024-03-22 15:07:43.927] INFO [pool-5-thread-56] BackupProvider.checkIfFileCanBeCopiedOrMoved:271 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Files will be copied or moved - they will be stored on the same file system [2024-03-22 15:07:44.177] INFO [pool-5-thread-56] ExportImportUtils.getVmExportImportMode:29 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Export import strategy by backup: SSH_TRANSFER [2024-03-22 15:07:44.190] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81' [2024-03-22 15:07:44.736] INFO [pool-5-thread-56] StoreTask.generateTimeStatsForStoreTask:228 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Backup of Virtual Machine: cirros15 (size: 1 GiB) finished. Stored in: 0s, total time: 41s. [2024-03-22 15:07:44.778] INFO [pool-5-thread-56] ITask.lambda$finishTask$2:71 [fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Finishing fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c: [Store] from 2024-03-22T15:07:35+01:00 to 2024-03-22T20:07:35+01:00 Task with message "Stored in backupdest1 (total time: 41s)"
This is log for incremental backup
==> api/api.log <== [2024-03-22 15:08:02] INFO [__ejb-thread-pool12] {rg=1, rt=USER} AuditLogService.log:87 User has submitted 1 tasks of types EXPORT [2024-03-22 15:08:10] INFO [__ejb-thread-pool5] {} TaskLoadBalanceTimer.assignTasks:263 Task Assignment Utility: EXPORT task b7277ddb-a4d1-4284-8783-30ba407d38bd has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5) ==> node1/vprotect_daemon.log <== [2024-03-22 15:08:13.752] INFO [pool-5-thread-57] ITask.lambda$startTask$1:39 [699ccbec-a419-4745-a5d8-0314f69c8576] Starting Task b7277ddb-a4d1-4284-8783-30ba407d38bd: [Export] from 2024-03-22T15:07+01:00 to 2024-03-22T20:07+01:00 [2024-03-22 15:08:13.789] INFO [pool-5-thread-57] ExportTask.run:115 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Starting export of: VM cirros15 (551df74b-dd5a-4b64-b5e2-0fe523d47da3) from 10.150.6.73 (OpenStack)... [2024-03-22 15:08:13.790] INFO [pool-5-thread-57] BackupProviderUtil.testConnection:238 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Testing connection with backup provider: backupdest1 [2024-03-22 15:08:13.791] INFO [pool-5-thread-57] FileUtil.removeLocalTestFiles:645 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Cleaning up after Connectivity Test [2024-03-22 15:08:13.809] INFO [pool-5-thread-57] ExportImportUtils.getVmExportImportModeByHvManager:43 [b7277ddb-a4d1-4284-8783-30ba407d38bd] OpenStack export import strategy by hypervisor manager: SSH_TRANSFER [2024-03-22 15:08:13.820] INFO [pool-5-thread-57] OpenStackClientFactory.authenticateScopedToProjectName:140 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81' [2024-03-22 15:08:14.324] INFO [pool-5-thread-57] SshFingerprintCache.getOrCreateTrustedCertificateWithDecodedChain:55 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Getting ssh fingerprint for TrustedCertificateChainId(sourceId=4d54f698-c559-4261-8e5c-46b23ea99bf7, sourceType=HYPERVISOR, chainType=SSH_HOST_FINGERPRINT) and task: b7277ddb-a4d1-4284-8783-30ba407d38bd: [Export] from 2024-03-22T15: 07+01:00 to 2024-03-22T20:07+01:00 [2024-03-22 15:08:14.330] INFO [pool-5-thread-57] SshUtil.getConfig:17 [b7277ddb-a4d1-4284-8783-30ba407d38bd] StrictHostKeyChecking = yes [2024-03-22 15:08:14.330] INFO [pool-5-thread-57] SshUtil.getConfig:17 [b7277ddb-a4d1-4284-8783-30ba407d38bd] StrictHostKeyChecking = yes [2024-03-22 15:08:15.827] INFO [pool-5-thread-57] OpenStackClientFactory.authenticateScopedToProjectName:140 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'OpenStackHypervisorManager.java#listVmDisks:204' [2024-03-22 15:08:17.468] WARN [pool-5-thread-57] LibvirtHypervisor.lambda$logUnsupportedQcow2Warning$45:1384 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Unable to create incremental backup for a non-excluded QCOW2 driver [UUID: 4afff436-71cc-4a30-8ca5-a9406e6350aa] ==> api/api.log <== [2024-03-22 15:08:17] INFO [http-thread-pool::http-listener(9)] {r=NODE, s=c20ddf27a4761e7799394d78ab62} SystemNotificationService.createNotification:119 System Notification of code: NODE_BACKUP_PE_NOT_READY_FOR_INCREMENTAL_BACKUP ==> node1/vprotect_daemon.log <== [2024-03-22 15:08:17.546] INFO [pool-5-thread-57] LibvirtHypervisor.snapshotExists:1489 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Checking if snapshot VM cirros15 (bd5d0690-1098-41f1-aab5-473d04351a35) - snapshot: c08f07b0-2373-48c7-b212-8d57a755b15d (8eac6082-aa6e-4b1f-8a35-618b946b73ce) @ 2024-03-22T15:07:07+01:00 exists [2024-03-22 15:08:19.030] WARN [pool-5-thread-57] ExportIncremental.isIncrementalExportPossible:172 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Reasons why incremental is not possible of the VM (551df74b-dd5a-4b64-b5e2-0fe523d47da3 - bd5d0690-1098-41f1-aab5-473d04351a35): Warning: Virtual Machine is not ready for incremental [2024-03-22 15:08:19.031] WARN [pool-5-thread-57] ExportTask.pickApplicableBackupType:248 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Incremental backup for OpenStack is not available at this time. [2024-03-22 15:08:19.064] INFO [pool-5-thread-57] ExportFull.export:30 [b7277ddb-a4d1-4284-8783-30ba407d38bd] Performing full backup
When a new machine is created it has following virsh configuration
<devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1' index='1'/> <backingStore/> <target dev='vda' bus='virtio'/> <serial>68e5a527-05e4-4451-aa96-983f404fa3f1</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
[root@openstackdev rocky]# qemu-img info /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1 image: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1 file format: raw virtual size: 1 GiB (1073741824 bytes) disk size: 75 MiB Child node '/file': filename: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1 protocol type: file file length: 1 GiB (1073741824 bytes) disk size: 75 MiB
After first full snapshot:
<devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1-vProtect-58e8e8b3-9af9-434e-9cbd-7a991ede8328' index='2'/> <backingStore type='file' index='1'> <format type='raw'/> <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <serial>68e5a527-05e4-4451-aa96-983f404fa3f1</serial> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
I tried also to start instance with a nova boot device (no cinder), initial disk is a qcow2, but I receive similar error when try to do incremental backup
-
Hello @victors,
This new limitation was introduced in the 6.2 version of Storware Backup & Recovery. In this strategy, you can make only full backups of OpenStack instances because every time a snapshot is kept on the VM, the disk type changes to QCOW2.
Incremental backups are only available for CEPH volumes in SSH/libvirt strategy,