@smmj We can try one more thing, can you please send me database of storware app ? Script to dump it to file is located in /opt/vprotect/scripts/backup_db.sh and result file is saved to /tmp

@Argus Thank you. I just checked my instance and for some reason it is backing up now. The jobs are still running, but they are not failing yet. I do not recall making any changes since the last communication and I will continue to monitor for results.

@andreargamgroup I can confirm that the same behavior occurs when you are using a password with UTF-8 characters. I will let you know when the fix for this problem will be ready.

@tanawat do you have the same error during backup? Can you send the log from Node (/opt/vprotect/logs/node_name/vprotect_daemon.log) and from vmware service (/opt/vprotect/logs/vmware/vprotect_vmware.log) ? We need log just after the backup.

@krzysztof-sz Yes, I tried that, didn't work. Could do 1 backup, the next one failed. I think I'll just reinstall it all.

@j-linda @krzysztof-sz attached is the log. [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] InventorySyncTask.initHVSync:306 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Started Inventory Sync Task of Hypervisor: 192.168.250.53 [2024-09-23 08:50:08.718] INFO [pool-5-thread-89] HypervApi.getAgentVersion:557 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting Agent version [2024-09-23 08:50:08.724] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:09.742] WARN [pool-5-thread-89] IVmSource.fetchApiVersion:138 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Cannot fetch API version: Bad Request. Skipping. [2024-09-23 08:50:09.805] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type INIT to Server [2024-09-23 08:50:09.826] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: INIT [2024-09-23 08:50:09.913] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type DATACENTERS to Server [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: DATACENTERS [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervApi.getCluster:303 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Getting cluster information from host: https://192.168.250.53:50882 [2024-09-23 08:50:09.953] INFO [pool-5-thread-89] HypervRequestSender.refreshAuthTokenIfExpired:70 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Auth token expired, calling refresh(): 192.168.250.53 [2024-09-23 08:50:10.855] ERROR [pool-5-thread-89] InventorySyncTask.lambda$run$0:118 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Bad Request eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) [?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.857] ERROR [pool-5-thread-89] ITask.finishTask:48 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Synchronization task failed: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:134) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorCluster(InventorySyncTask.java:370) ~[engine.jar:?] at java.util.Optional.orElseGet(Optional.java:369) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.getHypervisorClusters(InventorySyncTask.java:363) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:256) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:251) ~[engine.jar:?] at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?] at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:251) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:198) ~[engine.jar:?] at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:105) [engine.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:50) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.storware.vprotect.engine.exception.ExternalAPIException: Bad Request at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:128) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more Caused by: eu.vprotect.hypervclient.ApiException: Bad Request at eu.vprotect.hypervclient.ApiClient.handleResponse(ApiClient.java:926) ~[engine.jar:?] at eu.vprotect.hypervclient.ApiClient.execute(ApiClient.java:842) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.getAuthenticationToken(HypervRequestSender.java:122) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.refreshAuthTokenIfExpired(HypervRequestSender.java:71) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.checkTokenAndCall(HypervRequestSender.java:64) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervRequestSender.send(HypervRequestSender.java:41) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervApi.getCluster(HypervApi.java:304) ~[engine.jar:?] at eu.storware.vprotect.engine.hypervisor.hyperv.HypervHypervisor.getHvCluster(HypervHypervisor.java:124) ~[engine.jar:?] ... 18 more [2024-09-23 08:50:10.891] INFO [pool-5-thread-89] ITask.lambda$finishTask$2:72 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finishing d80c9435-2fc4-461b-9f90-e119bc4e7e8f: [Inventory synchronization] from 2024-09-23T08:49:55+02:00 to 2024-09-23T09:49:55+02:00 Task with message "Synchronization task failed: ExternalAPIException: Bad Request" [2024-09-23 08:50:10.989] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:603 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Sending inventory with type FINALIZE to Server [2024-09-23 08:50:11.017] INFO [pool-5-thread-89] InventorySyncTask.sendInventory:607 [d80c9435-2fc4-461b-9f90-e119bc4e7e8f] Finished step: FINALIZE

@lsroga I disconnected cdrom and ran inventory but still the same issue. I've emailed you the latest logs.

@Cristian-Eglue Yes I can see thank you. So in this configuration, you set 7 days and 14 versions to keep, but your scheduler is configured to make a backup every day. So after a week, you have 7 backups but you also set a time restriction so backups older than 7 days will be marked as expired and removed after the next backup cleanup job so you only get 7 versions, not 14. nevertheless, you should be able to restore from any of those 7 versions.

@krzysztof-sz Finally last week I had some spare time to do some tests, I wasn't able to import the old database because after the import the service storware-server stopped to work, anyway I built from the ground up a new VM with Storware 6.1.0-25, and so far it is working fine (except two minor issues that I'll now discuss here). Hope it will not broke in the next days. Regards, Cristian

@KLM You can delete backups by removing VMs from the list in the Instances tab.

When a Storware update goes wrong, here's how to handle it: Stay Calm: These issues can be fixed. Check Logs: Look for error messages in the update logs. Rollback if Needed: Revert to the previous version if the update caused major problems. Contact Support: Reach out to Storware support for help. Verify Compatibility: Ensure the update is compatible with your system. Backup Data: Securely back up your data before making further changes. Check Dependencies: Make sure all necessary components are installed and updated. Consider a Clean Install: Uninstall and reinstall the latest version if problems persist. Review Documentation: Check Storware’s documentation for any known issues and solutions.

@nsniko777 Can you use HTTP only for the test? What port do you have set for the API in OpenNebula? Can you try connecting via HTTPS to the default API port 2633? For example: https://192.168.1.2:2633

@Argus ohh ok... I don't have it... that's the problem... thanks for message....

@Argus thanks very much!

Hello @user If you still have this problem, please send us the full DEBUG logs of this operation (server and node logs). If you cannot attach logs to this thread, you can put them somewhere, e.g. OneDrive or similar, and send the link. Here is the documentation on how to switch logs to debug: Logs in DEBUG

Hi! Did you try use that KB? There are few additional steps. KB10025 Regards Krzysztof

@tomasdeltell Yes, it is.

Hi @matti Disk Image Transfer requires RHV to be in version 4.3+. For version 4.2 you can only use the disk attachment strategy which gives you the possibility to do only full backups. This strategy also requires having a node installed inside the RHV cluster as a proxy vm. https://storware.gitbook.io/backup-and-recovery/protecting-virtual-machines/virtual-machines/red-hat-virtualization#disk-attachment-with-proxy-vm

Hi! @user NFS4.2 have support to synthetic backup, but it is necessary to use XFS under NFS share. NFS only provide possibility to use synthetic file system under share, we support XFS as Synthetic FS. The same error you have in XCP-ng. Regards

Hi! After deep analysis we found, that issue is caused by deleting of snapshots files without merging it to vm. We do not this kind of operation, so it could be done only by user itself. Regards