Openstack incremental backup

victors

Hello
I tried to test incremental backup on openstack with libvirt strategy . However I receive following error

[b7277ddb-a4d1-4284-8783-30ba407d38bd] Unable to create incremental backup for a non-excluded QCOW2 driver [UUID: 4afff436-71cc-4a30-8ca5-a9406e6350aa]

This is the log for first full backup

Task Assignment Utility: EXPORT task 162d6b39-42fa-418b-af16-41777176a8c0 has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5)                                                                                                                 [305/1825]


==> node1/vprotect_daemon.log <==
[2024-03-22 15:07:03.747] INFO [pool-5-thread-56] ITask.lambda$startTask$1:39
[] Starting Task 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01:00

[2024-03-22 15:07:03.781] INFO [pool-5-thread-56] ExportTask.run:115
[162d6b39-42fa-418b-af16-41777176a8c0] Starting export of: VM cirros15 (551df74b-dd5a-4b64-b5e2-0fe523d47da3) from 10.150.6.73 (OpenStack)...

[2024-03-22 15:07:03.781] INFO [pool-5-thread-56] BackupProviderUtil.testConnection:238
[162d6b39-42fa-418b-af16-41777176a8c0] Testing connection with backup provider: backupdest1

[2024-03-22 15:07:03.782] INFO [pool-5-thread-56] FileUtil.removeLocalTestFiles:645
[162d6b39-42fa-418b-af16-41777176a8c0] Cleaning up after Connectivity Test

[2024-03-22 15:07:03.800] INFO [pool-5-thread-56] ExportImportUtils.getVmExportImportModeByHvManager:43
[162d6b39-42fa-418b-af16-41777176a8c0] OpenStack export import strategy by hypervisor manager: SSH_TRANSFER

[2024-03-22 15:07:03.812] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140
[162d6b39-42fa-418b-af16-41777176a8c0] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81'

[2024-03-22 15:07:04.278] INFO [pool-5-thread-56] SshFingerprintCache.getOrCreateTrustedCertificateWithDecodedChain:55
[162d6b39-42fa-418b-af16-41777176a8c0] Getting ssh fingerprint for TrustedCertificateChainId(sourceId=4d54f698-c559-4261-8e5c-46b23ea99bf7, sourceType=HYPERVISOR, chainType=SSH_HOST_FINGERPRINT) and task: 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:
06+01:00 to 2024-03-22T20:06+01:00

[2024-03-22 15:07:04.283] INFO [pool-5-thread-56] SshUtil.getConfig:17
[162d6b39-42fa-418b-af16-41777176a8c0] StrictHostKeyChecking = yes

[2024-03-22 15:07:04.283] INFO [pool-5-thread-56] SshUtil.getConfig:17
[162d6b39-42fa-418b-af16-41777176a8c0] StrictHostKeyChecking = yes

[2024-03-22 15:07:05.051] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140
[162d6b39-42fa-418b-af16-41777176a8c0] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'OpenStackHypervisorManager.java#listVmDisks:204'

[2024-03-22 15:07:05.586] INFO [pool-5-thread-56] ExportFull.export:30
[162d6b39-42fa-418b-af16-41777176a8c0] Performing full backup

[2024-03-22 15:07:06.280] WARN [pool-5-thread-56] SshConnector.chownRemote:424
[162d6b39-42fa-418b-af16-41777176a8c0] Could not chown file /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa

[2024-03-22 15:07:07.372] INFO [pool-5-thread-56] SshConnector.chownSnapshots:398
[162d6b39-42fa-418b-af16-41777176a8c0] Trying chown snapshots

[2024-03-22 15:07:07.661] WARN [pool-5-thread-56] SshConnector.chownRemote:424
[162d6b39-42fa-418b-af16-41777176a8c0] Could not chown file /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa-vProtect-c08f07b0-2373-48c7-b212-8d57a755b15d

[2024-03-22 15:07:08.240] INFO [pool-5-thread-56] OpenStackHypervisorManager.exportImage:291
[162d6b39-42fa-418b-af16-41777176a8c0] Start exporting image

[2024-03-22 15:07:08.240] INFO [pool-5-thread-56] OpenStackImageApi.fetchAsVmDisks:45
[162d6b39-42fa-418b-af16-41777176a8c0] Fetching images as vm disk in server id: 551df74b-dd5a-4b64-b5e2-0fe523d47da3

[2024-03-22 15:07:08.248] INFO [pool-5-thread-56] ImageExporter.export:32
[162d6b39-42fa-418b-af16-41777176a8c0] isDownloadImageFromGlance : false

[2024-03-22 15:07:08.249] INFO [pool-5-thread-56] OpenStackExcludedImageExporter.export:23
[162d6b39-42fa-418b-af16-41777176a8c0] Image excluded from backup - skipping export

[2024-03-22 15:07:08.632] INFO [pool-5-thread-56] LibvirtHypervisor.exportDisks:576
[162d6b39-42fa-418b-af16-41777176a8c0] Disks to export: [file/disk: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa]

[2024-03-22 15:07:09.364] INFO [pool-5-thread-56] LibvirtHypervisor.prepareSnapDownload:1256
[162d6b39-42fa-418b-af16-41777176a8c0] Adding snapshot file to the download list: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa

[2024-03-22 15:07:09.711] INFO [pool-5-thread-56] LibvirtHypervisor.downloadFromTransferList:666
[162d6b39-42fa-418b-af16-41777176a8c0] Downloading VM data: cirros15

[2024-03-22 15:07:10.017] INFO [pool-5-thread-56] SshConnector.transfer:429
[162d6b39-42fa-418b-af16-41777176a8c0] Starting transfer

[2024-03-22 15:07:10.295] INFO [pool-5-thread-56] SshConnector.transfer:453
[162d6b39-42fa-418b-af16-41777176a8c0] Downloading: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa -> /vprotect_data/cirros15__551df74b/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa
[2024-03-22 15:07:17.472] INFO [pool-5-thread-56] ProtectedEntityUtil.collectFileSystemInfo:198
[162d6b39-42fa-418b-af16-41777176a8c0] Collecting file system information...
[2024-03-22 15:07:35.538] INFO [pool-5-thread-56] ITask.lambda$finishTask$2:71
[162d6b39-42fa-418b-af16-41777176a8c0] Finishing 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01:00 Task with message "Backup of Virtual Machine: cirros15 (size: 1 GiB) exported in: 31s."


==> api/api.log <==
[2024-03-22 15:07:40] INFO  [__ejb-thread-pool14] {} TaskLoadBalanceTimer.assignTasks:263
Task Assignment Utility: STORE task fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5)


==> node1/vprotect_daemon.log <==
[2024-03-22 15:07:43.784] INFO [pool-5-thread-56] ITask.lambda$startTask$0:38
[162d6b39-42fa-418b-af16-41777176a8c0] Starting Task fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c: [Store] from 2024-03-22T15:07:35+01:00 to 2024-03-22T20:07:35+01:00 as a result of 162d6b39-42fa-418b-af16-41777176a8c0: [Export] from 2024-03-22T15:06+01:00 to 2024-03-22T20:06+01
:00

[2024-03-22 15:07:43.834] INFO [pool-5-thread-56] StoreTask.run:80
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Files to store: [3a1b0f66-c630-4970-8cc7-a80a3c8f6282: /vprotect_data/cirros15__551df74b/18ea56d6-f5d0-4132-ad6b-e35aeed97890.img - null (modified: null | format: EXCLUDED_DISK | vmDiskGuid: 5175b419-687f-455e-b462-27c17c5ccfa3), 02
fe3a0a-e94f-4aa7-8be9-d31a52fd0a97: /vprotect_data/cirros15__551df74b/4afff436-71cc-4a30-8ca5-a9406e6350aa.json - null (modified: null | format: DISK_META_JSON | vmDiskGuid: 694e11e6-6044-43ce-a5bf-a85d3b55af12), a4d14c95-53ab-4b09-b836-eabc16331882: /vprotect_data/cirro
s15__551df74b/551df74b-dd5a-4b64-b5e2-0fe523d47da3.json - null (modified: null | format: VM_META_OPENSTACK | vmDiskGuid: null), 849e3f39-57c9-4d04-bb62-a9041915c583: /vprotect_data/cirros15__551df74b/volume-4afff436-71cc-4a30-8ca5-a9406e6350aa - null (modified: null | fo
rmat: RAW | vmDiskGuid: 694e11e6-6044-43ce-a5bf-a85d3b55af12)]

[2024-03-22 15:07:43.834] INFO [pool-5-thread-56] StoreTask.run:84
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Storing backup files (Virtual Machine: cirros15) in File System...

[2024-03-22 15:07:43.927] INFO [pool-5-thread-56] BackupProvider.checkIfFileCanBeCopiedOrMoved:271
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Files will be copied or moved - they will be stored on the same file system

[2024-03-22 15:07:44.177] INFO [pool-5-thread-56] ExportImportUtils.getVmExportImportMode:29
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Export import strategy by backup: SSH_TRANSFER

[2024-03-22 15:07:44.190] INFO [pool-5-thread-56] OpenStackClientFactory.authenticateScopedToProjectName:140
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81'

[2024-03-22 15:07:44.736] INFO [pool-5-thread-56] StoreTask.generateTimeStatsForStoreTask:228
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Backup of Virtual Machine: cirros15 (size: 1 GiB) finished. Stored in: 0s, total time: 41s.

[2024-03-22 15:07:44.778] INFO [pool-5-thread-56] ITask.lambda$finishTask$2:71
[fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c] Finishing fa94c0fe-5e62-45bf-b4b3-ecb22cfc7d8c: [Store] from 2024-03-22T15:07:35+01:00 to 2024-03-22T20:07:35+01:00 Task with message "Stored in backupdest1 (total time: 41s)"

This is log for incremental backup

==> api/api.log <==
[2024-03-22 15:08:02] INFO  [__ejb-thread-pool12] {rg=1, rt=USER} AuditLogService.log:87
User has submitted 1 tasks of types EXPORT

[2024-03-22 15:08:10] INFO  [__ejb-thread-pool5] {} TaskLoadBalanceTimer.assignTasks:263
Task Assignment Utility: EXPORT task b7277ddb-a4d1-4284-8783-30ba407d38bd has been assigned to the node node1 (206e4177-b07e-413d-ac71-2287ac5df8f5)






==> node1/vprotect_daemon.log <==
[2024-03-22 15:08:13.752] INFO [pool-5-thread-57] ITask.lambda$startTask$1:39
[699ccbec-a419-4745-a5d8-0314f69c8576] Starting Task b7277ddb-a4d1-4284-8783-30ba407d38bd: [Export] from 2024-03-22T15:07+01:00 to 2024-03-22T20:07+01:00

[2024-03-22 15:08:13.789] INFO [pool-5-thread-57] ExportTask.run:115
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Starting export of: VM cirros15 (551df74b-dd5a-4b64-b5e2-0fe523d47da3) from 10.150.6.73 (OpenStack)...

[2024-03-22 15:08:13.790] INFO [pool-5-thread-57] BackupProviderUtil.testConnection:238
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Testing connection with backup provider: backupdest1

[2024-03-22 15:08:13.791] INFO [pool-5-thread-57] FileUtil.removeLocalTestFiles:645
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Cleaning up after Connectivity Test

[2024-03-22 15:08:13.809] INFO [pool-5-thread-57] ExportImportUtils.getVmExportImportModeByHvManager:43
[b7277ddb-a4d1-4284-8783-30ba407d38bd] OpenStack export import strategy by hypervisor manager: SSH_TRANSFER

[2024-03-22 15:08:13.820] INFO [pool-5-thread-57] OpenStackClientFactory.authenticateScopedToProjectName:140
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'SshTransferOpenStackImportExportStrategy.java#<init>:81'

[2024-03-22 15:08:14.324] INFO [pool-5-thread-57] SshFingerprintCache.getOrCreateTrustedCertificateWithDecodedChain:55
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Getting ssh fingerprint for TrustedCertificateChainId(sourceId=4d54f698-c559-4261-8e5c-46b23ea99bf7, sourceType=HYPERVISOR, chainType=SSH_HOST_FINGERPRINT) and task: b7277ddb-a4d1-4284-8783-30ba407d38bd: [Export] from 2024-03-22T15:
07+01:00 to 2024-03-22T20:07+01:00

[2024-03-22 15:08:14.330] INFO [pool-5-thread-57] SshUtil.getConfig:17
[b7277ddb-a4d1-4284-8783-30ba407d38bd] StrictHostKeyChecking = yes

[2024-03-22 15:08:14.330] INFO [pool-5-thread-57] SshUtil.getConfig:17
[b7277ddb-a4d1-4284-8783-30ba407d38bd] StrictHostKeyChecking = yes

[2024-03-22 15:08:15.827] INFO [pool-5-thread-57] OpenStackClientFactory.authenticateScopedToProjectName:140
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Authentication scoped by project name - url: 'http://10.150.6.73:5000/v3' | region: 'RegionOne' | domain: 'default' | project: 'admin' | user: 'admin' | execution source: 'OpenStackHypervisorManager.java#listVmDisks:204'

[2024-03-22 15:08:17.468] WARN [pool-5-thread-57] LibvirtHypervisor.lambda$logUnsupportedQcow2Warning$45:1384
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Unable to create incremental backup for a non-excluded QCOW2 driver [UUID: 4afff436-71cc-4a30-8ca5-a9406e6350aa]


==> api/api.log <==
[2024-03-22 15:08:17] INFO  [http-thread-pool::http-listener(9)] {r=NODE, s=c20ddf27a4761e7799394d78ab62} SystemNotificationService.createNotification:119
System Notification of code: NODE_BACKUP_PE_NOT_READY_FOR_INCREMENTAL_BACKUP


==> node1/vprotect_daemon.log <==
[2024-03-22 15:08:17.546] INFO [pool-5-thread-57] LibvirtHypervisor.snapshotExists:1489
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Checking if snapshot VM cirros15 (bd5d0690-1098-41f1-aab5-473d04351a35) - snapshot: c08f07b0-2373-48c7-b212-8d57a755b15d (8eac6082-aa6e-4b1f-8a35-618b946b73ce) @ 2024-03-22T15:07:07+01:00 exists

[2024-03-22 15:08:19.030] WARN [pool-5-thread-57] ExportIncremental.isIncrementalExportPossible:172
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Reasons why incremental is not possible of the VM (551df74b-dd5a-4b64-b5e2-0fe523d47da3 - bd5d0690-1098-41f1-aab5-473d04351a35): Warning: Virtual Machine is not ready for incremental

[2024-03-22 15:08:19.031] WARN [pool-5-thread-57] ExportTask.pickApplicableBackupType:248
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Incremental backup for OpenStack is not available at this time.

[2024-03-22 15:08:19.064] INFO [pool-5-thread-57] ExportFull.export:30
[b7277ddb-a4d1-4284-8783-30ba407d38bd] Performing full backup

When a new machine is created it has following virsh configuration

  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>68e5a527-05e4-4451-aa96-983f404fa3f1</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

[root@openstackdev rocky]# qemu-img info /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1
image: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1
file format: raw
virtual size: 1 GiB (1073741824 bytes)
disk size: 75 MiB
Child node '/file':
    filename: /var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1
    protocol type: file
    file length: 1 GiB (1073741824 bytes)
    disk size: 75 MiB

After first full snapshot:

  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1-vProtect-58e8e8b3-9af9-434e-9cbd-7a991ede8328' index='2'/>
      <backingStore type='file' index='1'>
        <format type='raw'/>
        <source file='/var/lib/nova/mnt/404d50d99406bdb14eefa7d751782ec1/volume-68e5a527-05e4-4451-aa96-983f404fa3f1'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>68e5a527-05e4-4451-aa96-983f404fa3f1</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

I tried also to start instance with a nova boot device (no cinder), initial disk is a qcow2, but I receive similar error when try to do incremental backup

a.spinek

Hello @victors,

This new limitation was introduced in the 6.2 version of Storware Backup & Recovery. In this strategy, you can make only full backups of OpenStack instances because every time a snapshot is kept on the VM, the disk type changes to QCOW2.

Incremental backups are only available for CEPH volumes in SSH/libvirt strategy,

https://storware.gitbook.io/backup-and-recovery/changelog

Storware Backup & Recovery

Navigation

Account

Navigation

Account

Openstack incremental backup