Hello @jguilford81 Please go to the Virtual Environments -> Virtualization Providers -> click your vCenter -> Certificates tab. Then select "Trust all certificates" and save. Perform inventory synchronization again and check the backup.

Hello @branes What do your backup retention settings look like? With what parameters is NFS shared? Does the storware node have full access to this NFS resource? You are sure that nothing changed in the DataDomain settings in the meantime?

Hello @kirkscollision Sorry for the late response time, I hope you are still interested in our solution. I'm not sure what installation you're asking about. The error occurs while installing our software or XCP? Please provide more information and we will try to help.

@Argus I knew it would end up being me! Thanks, I have cleared up the formatting in the file and enter at the end. I have finally proceeded to inventory. And thanks for the quick response.

@j-linda @krzysztof-sz attached is the log. [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] InventorySyncTask.initHVSync:306 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Started Inventory Sync Task of Hypervisor: 192.168.250.53 [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] HypervApi.getAgentVersion:557 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting Agent version [2024-09-23 08:50:08.724] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:09.742] WARN [pool-5-thread-89] IVmSource.fetchApiVersion:138 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Cannot fetch API version: Bad Request. Skipping. [2024-09-23 08:50:09.805] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type INIT to Server [2024-09-23 08:50:09.826] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: INIT [2024-09-23 08:50:09.913] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type DATACENTERS to Server [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: DATACENTERS [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervApi.getCluster:303 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting cluster information from host: https://192.168.250.53:50882 [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:10.855] ERROR [pool-5-thread-89] InventorySyncTask.lambda$run$0:118 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Bad Request eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) [?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.857] ERROR [pool-5-thread-89] ITask.finishTask:48 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Synchronization task failed: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.891] INFO [pool-5-thread-89] ITask.lambda$finishTask$2:72 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finishing d80c9435-2fc4-461b-9f90-e119bc4e7e8f: [Inventory synchronization] from 2024-09-23T08:49:55+02:00 to 2024-09-23T09:49:55+02:00 Task with message "Synchronization task failed: ExternalAPIException: Bad Request" [2024-09-23 08:50:10.989] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type FINALIZE to Server [2024-09-23 08:50:11.017] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: FINALIZE

Hello @centrisag I am sorry for late response. I must to ask, do you mean OLVM (Oracle Linux VM, ovirt based hypervisormanager) or OVM (which is older solution from Oracle)? It is important in meaning of choose strategy. At this moment we do not make another HA feature, then in our documentation, but I think, that we could handle your environment in other way: Our software is build in architecture Server+Nodes. You could use many Nodes with own config (for example for Main and External site) Server could be in completly another place than worker nodes (for example in another hypervisor) You can restore VM from Main to external site only choosing where this vm should be restored (if you have Node configs, we choose based on assinged Node config to hypervisor) You can use guide for Disaster Recovery if you want to arise server in another location. If you need additional information, the best option will be contact with our PreSales Team (presales@storware.eu) where my coworkers could make meeting where they could show you how we handle similar cases. Regards Krzysztof Szawara IT Systems Specialist

@lsroga Hello! Thanks for the reply. vProtect version is 6.2.0-41. The backup destination is a boostfs file system. Haven't tried to run the script manually. The schedule job sometimes works, other times don't with the error reports. Internal database works ok. Here follows the requested commands output: [Wed Jul 31 15:53:08] root@cvprotect01:~ $ rpm -qa | grep vprotect vprotect-node-6.2.0-34.el9.x86_64 vprotect-server-6.2.0-41.el9.x86_64 [Wed Jul 31 15:53:18] root@vprotect01:~ $ ls -l / total 29 dr-xr-xr-x. 2 root root 6 Aug 9 2021 afs lrwxrwxrwx. 1 root root 7 Aug 9 2021 bin -> usr/bin dr-xr-xr-x. 6 root root 4096 Jul 11 15:47 boot drwxr-xr-x. 21 root root 3360 Jul 31 15:10 dev drwxr-xr-x. 129 root root 8192 Jul 23 10:05 etc drwxr-xr-x. 2 root root 6 Aug 9 2021 home lrwxrwxrwx. 1 root root 7 Aug 9 2021 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Aug 9 2021 lib64 -> usr/lib64 drwxr-xr-x. 2 root root 6 Aug 9 2021 media drwxr-xr-x. 3 root root 22 May 10 14:33 mnt drwxr-xr-x. 7 root root 84 Jul 23 10:05 opt dr-xr-xr-x. 342 root root 0 Jul 15 11:41 proc dr-xr-x---. 6 root root 4096 Jul 30 09:39 root drwxr-xr-x. 42 root root 1340 Jul 31 15:10 run lrwxrwxrwx. 1 root root 8 Aug 9 2021 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Aug 9 2021 srv dr-xr-xr-x. 13 root root 0 Jul 15 11:41 sys drwxrwxrwt. 19 root root 4096 Jul 31 15:53 tmp drwxr-xr-x. 14 root root 180 Jun 19 18:14 usr drwxr-xr-x. 22 root root 4096 Jul 11 15:47 var drwxrwxrwx. 10 vprotect vprotect 579 Jul 31 13:02 vprotect_data [Wed Jul 31 15:53:28] root@vprotect01:~ $ ls -l /vprotect_data/ total 4 drwxr-xr-x. 28 vprotect vprotect 2176 Jul 31 03:03 backups drwxr-xr-x. 2 vprotect vprotect 101 Jun 20 11:29 export drwxr-xr-x. 2 vprotect vprotect 101 Jul 15 17:52 import drwxr-xr-x. 2 vprotect vprotect 101 Jun 11 19:25 mount [Wed Jul 31 15:53:40] root@vprotect01:~ $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 9.7G 0 9.7G 0% /dev/shm tmpfs 3.9G 161M 3.7G 5% /run /dev/mapper/cs-root 45G 9.1G 36G 21% / /dev/sda1 960M 356M 605M 38% /boot boostfs 499T 154T 345T 31% /vprotect_data tmpfs 2.0G 4.0K 2.0G 1% /run/user/0 Here are new examples of another application backup that fails first, then the relaunch is successful and the next scheduled backup works as well: 2024-07-29__03.00.20_job scheduled failed - https://pastebin.com/k9xdE4F4 2024-07-29__09.45.12_job relaunched OK - https://pastebin.com/0erYHa8b 2024-07-30__03.00.22_job scheduled OK - https://pastebin.com/xzdp4TsA

When a Storware update goes wrong, here's how to handle it: Stay Calm: These issues can be fixed. Check Logs: Look for error messages in the update logs. Rollback if Needed: Revert to the previous version if the update caused major problems. Contact Support: Reach out to Storware support for help. Verify Compatibility: Ensure the update is compatible with your system. Backup Data: Securely back up your data before making further changes. Check Dependencies: Make sure all necessary components are installed and updated. Consider a Clean Install: Uninstall and reinstall the latest version if problems persist. Review Documentation: Check Storware’s documentation for any known issues and solutions.

@krzysztof-sz Yes, I tried that, didn't work. Could do 1 backup, the next one failed. I think I'll just reinstall it all.

@a-spinek Thank you, that works!

@djaw Yes it is a stand alone host

Hi @mario-krizan, Also, is it possible to migrate a virtual machine from OVM to OLVM with Stormware? Unfortunately, we do not support this type of migration. Failed to collect file system information: [Command execution failed: [/usr/bin/virt-filesystems, --filesystems, -l, --csv, -a, /vprotect_data/abc_fmo_pool_01/A22000A0__0004fb00000600007ee414f01ab0c9f5/0004fb00001200002bb2cfe407f43a53.img] Return code: 1 output: [] error: [libguestfs: error: force_kvm supplied but kvm not available. ]] Please provide a log with DEBUG mode enabled. Instructions how to do this: https://storware.gitbook.io/backup-and-recovery/troubleshooting/how-switch-vprotect-to-debug-mode

Hello! Sorry for late response. Please tell me at first, did you tried to update to version 6.2.0 and then restore backup? If problem still occurs we need to look into log f backup and restore. Before you generate it please follow this guide: DEBUG MODE Regards Krzysztof

@carl-ceriola Hi, what backup method do you use?

@vguillard Hi, no Storware is not capable of backuping/restoring user-data yet.

Scratch that, was able to modify some S3 settings and it now shows as a destination. Now my issue is that all backup jobs including internal db show queued and not doing anything.

@krzysztof-sz Finally last week I had some spare time to do some tests, I wasn't able to import the old database because after the import the service storware-server stopped to work, anyway I built from the ground up a new VM with Storware 6.1.0-25, and so far it is working fine (except two minor issues that I'll now discuss here). Hope it will not broke in the next days. Regards, Cristian

Hello @victors, This new limitation was introduced in the 6.2 version of Storware Backup & Recovery. In this strategy, you can make only full backups of OpenStack instances because every time a snapshot is kept on the VM, the disk type changes to QCOW2. Incremental backups are only available for CEPH volumes in SSH/libvirt strategy, https://storware.gitbook.io/backup-and-recovery/changelog

Hi! I am sorry for delay, but could you tell me more about your environment? Where is server and node (inside/outside VMWare)? Regards