The issue of missing VMs in Hyper-V can typically be resolved by checking configuration settings, updating Hyper-V, or using management tools to refresh and register all VMs. https://techdoor.us/

If you're unable to take a backup in an Oracle KVM environment and the status is stuck in "Queued," it could be due to resource contention, network issues, or a problem with the backup software. Try restarting the backup process, checking system resources, ensuring there are no network disruptions, and verifying the backup configuration. If the issue persists, reviewing logs or contacting support for further diagnosis might be necessary.

@sangram Yes, this is expected for MySQL in Storware. Application backups typically require full snapshots for consistency, and synchronization tracks changes, not incremental backups. It's similar to how Spotify syncs data. Check Storware's docs for more on incremental backups.

Hello @jguilford81 Please go to the Virtual Environments -> Virtualization Providers -> click your vCenter -> Certificates tab. Then select "Trust all certificates" and save. Perform inventory synchronization again and check the backup.

Hello @branes What do your backup retention settings look like? With what parameters is NFS shared? Does the storware node have full access to this NFS resource? You are sure that nothing changed in the DataDomain settings in the meantime?

Hello @kirkscollision Sorry for the late response time, I hope you are still interested in our solution. I'm not sure what installation you're asking about. The error occurs while installing our software or XCP? Please provide more information and we will try to help.

@Argus I knew it would end up being me! Thanks, I have cleared up the formatting in the file and enter at the end. I have finally proceeded to inventory. And thanks for the quick response.

@j-linda @krzysztof-sz attached is the log. [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] InventorySyncTask.initHVSync:306 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Started Inventory Sync Task of Hypervisor: 192.168.250.53 [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] HypervApi.getAgentVersion:557 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting Agent version [2024-09-23 08:50:08.724] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:09.742] WARN [pool-5-thread-89] IVmSource.fetchApiVersion:138 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Cannot fetch API version: Bad Request. Skipping. [2024-09-23 08:50:09.805] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type INIT to Server [2024-09-23 08:50:09.826] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: INIT [2024-09-23 08:50:09.913] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type DATACENTERS to Server [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: DATACENTERS [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervApi.getCluster:303 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting cluster information from host: https://192.168.250.53:50882 [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:10.855] ERROR [pool-5-thread-89] InventorySyncTask.lambda$run$0:118 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Bad Request eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) [?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.857] ERROR [pool-5-thread-89] ITask.finishTask:48 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Synchronization task failed: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.891] INFO [pool-5-thread-89] ITask.lambda$finishTask$2:72 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finishing d80c9435-2fc4-461b-9f90-e119bc4e7e8f: [Inventory synchronization] from 2024-09-23T08:49:55+02:00 to 2024-09-23T09:49:55+02:00 Task with message "Synchronization task failed: ExternalAPIException: Bad Request" [2024-09-23 08:50:10.989] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type FINALIZE to Server [2024-09-23 08:50:11.017] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: FINALIZE

Hello @centrisag I am sorry for late response. I must to ask, do you mean OLVM (Oracle Linux VM, ovirt based hypervisormanager) or OVM (which is older solution from Oracle)? It is important in meaning of choose strategy. At this moment we do not make another HA feature, then in our documentation, but I think, that we could handle your environment in other way: Our software is build in architecture Server+Nodes. You could use many Nodes with own config (for example for Main and External site) Server could be in completly another place than worker nodes (for example in another hypervisor) You can restore VM from Main to external site only choosing where this vm should be restored (if you have Node configs, we choose based on assinged Node config to hypervisor) You can use guide for Disaster Recovery if you want to arise server in another location. If you need additional information, the best option will be contact with our PreSales Team (presales@storware.eu) where my coworkers could make meeting where they could show you how we handle similar cases. Regards Krzysztof Szawara IT Systems Specialist

@lsroga Hello! Thanks for the reply. vProtect version is 6.2.0-41. The backup destination is a boostfs file system. Haven't tried to run the script manually. The schedule job sometimes works, other times don't with the error reports. Internal database works ok. Here follows the requested commands output: [Wed Jul 31 15:53:08] root@cvprotect01:~ $ rpm -qa | grep vprotect vprotect-node-6.2.0-34.el9.x86_64 vprotect-server-6.2.0-41.el9.x86_64 [Wed Jul 31 15:53:18] root@vprotect01:~ $ ls -l / total 29 dr-xr-xr-x. 2 root root 6 Aug 9 2021 afs lrwxrwxrwx. 1 root root 7 Aug 9 2021 bin -> usr/bin dr-xr-xr-x. 6 root root 4096 Jul 11 15:47 boot drwxr-xr-x. 21 root root 3360 Jul 31 15:10 dev drwxr-xr-x. 129 root root 8192 Jul 23 10:05 etc drwxr-xr-x. 2 root root 6 Aug 9 2021 home lrwxrwxrwx. 1 root root 7 Aug 9 2021 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Aug 9 2021 lib64 -> usr/lib64 drwxr-xr-x. 2 root root 6 Aug 9 2021 media drwxr-xr-x. 3 root root 22 May 10 14:33 mnt drwxr-xr-x. 7 root root 84 Jul 23 10:05 opt dr-xr-xr-x. 342 root root 0 Jul 15 11:41 proc dr-xr-x---. 6 root root 4096 Jul 30 09:39 root drwxr-xr-x. 42 root root 1340 Jul 31 15:10 run lrwxrwxrwx. 1 root root 8 Aug 9 2021 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Aug 9 2021 srv dr-xr-xr-x. 13 root root 0 Jul 15 11:41 sys drwxrwxrwt. 19 root root 4096 Jul 31 15:53 tmp drwxr-xr-x. 14 root root 180 Jun 19 18:14 usr drwxr-xr-x. 22 root root 4096 Jul 11 15:47 var drwxrwxrwx. 10 vprotect vprotect 579 Jul 31 13:02 vprotect_data [Wed Jul 31 15:53:28] root@vprotect01:~ $ ls -l /vprotect_data/ total 4 drwxr-xr-x. 28 vprotect vprotect 2176 Jul 31 03:03 backups drwxr-xr-x. 2 vprotect vprotect 101 Jun 20 11:29 export drwxr-xr-x. 2 vprotect vprotect 101 Jul 15 17:52 import drwxr-xr-x. 2 vprotect vprotect 101 Jun 11 19:25 mount [Wed Jul 31 15:53:40] root@vprotect01:~ $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 9.7G 0 9.7G 0% /dev/shm tmpfs 3.9G 161M 3.7G 5% /run /dev/mapper/cs-root 45G 9.1G 36G 21% / /dev/sda1 960M 356M 605M 38% /boot boostfs 499T 154T 345T 31% /vprotect_data tmpfs 2.0G 4.0K 2.0G 1% /run/user/0 Here are new examples of another application backup that fails first, then the relaunch is successful and the next scheduled backup works as well: 2024-07-29__03.00.20_job scheduled failed - https://pastebin.com/k9xdE4F4 2024-07-29__09.45.12_job relaunched OK - https://pastebin.com/0erYHa8b 2024-07-30__03.00.22_job scheduled OK - https://pastebin.com/xzdp4TsA

When a Storware update goes wrong, here's how to handle it: Stay Calm: These issues can be fixed. Check Logs: Look for error messages in the update logs. Rollback if Needed: Revert to the previous version if the update caused major problems. Contact Support: Reach out to Storware support for help. Verify Compatibility: Ensure the update is compatible with your system. Backup Data: Securely back up your data before making further changes. Check Dependencies: Make sure all necessary components are installed and updated. Consider a Clean Install: Uninstall and reinstall the latest version if problems persist. Review Documentation: Check Storware’s documentation for any known issues and solutions.