Storware Backup & Recovery
    • Storware Backup & Recovery

    • Store task failed: Server has encountered the runtime error (RUNTIME_ERROR)

      Virtual Environments 3 5 1.8k
      U
      U

      Hi.
      I'm testing the deployed OVA storware instance.
      I have 2 ESXi hosts and 1 XCP-ng host. Backup storage is mounted as NFS share. The staging folder is placed on that share too.
      I've successfully backed up windows VM from vmware host. Then I tried to backup ubuntu VM from xcp-ng host. Because of long backup time I left backup task to proceed and came back to it in the morning. I saw the error "Store task failed: Server has encountered the runtime error (RUNTIME_ERROR)". All my hosts are disappeared from the inventory and I couldn't rescan them again. I see the XVA image of the ubuntu VM in the staging folder. After vprotect service reload my hosts returned to the inventory. But there are no any signs of the failed backup in the "Instances". I've tried to backup again and now everything looks fine except I can't mount backuped vm with error "Cannot mount backup made by hypervisor with VM image strategy enabled".

      0
    • A
      A

      Hi we are not supporting file level restore for XCP we cannot read filesystems in its backup files, it is a technical limitation
      You can check what features we support or not for different entities here

      0
    • U
      U

      Hi. Thanks for information.
      Anyway the first mentioned error is not belonged to XCP-ng limitations. OK. I'll watching for it.

      0
    • U
      U

      Today all hosts are gone from the inventory with the same error. Why can't I attach files in thread? This is a part of the log.

      eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
      at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
      at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
      at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) [?:?]
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?]
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
      at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
      at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) [?:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) [engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) [engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      at java.lang.Thread.run(Thread.java:829) [?:?]
      Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
      at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
      at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
      ... 23 more
      [2023-12-03 08:10:43.528] ERROR [pool-5-thread-5] InventorySyncTask.lambda$getRetryPolicy$24:699
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Attempt number 6 failed.

      eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
      at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
      at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
      at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) [?:?]
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?]
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
      at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
      at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) [?:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) [engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) [engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      at java.lang.Thread.run(Thread.java:829) [?:?]
      Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
      at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
      at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
      ... 23 more
      [2023-12-03 08:10:43.529] ERROR [pool-5-thread-5] InventorySyncTask.lambda$getRetryPolicy$27:702
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Failed to send VMS, max retries exceeded

      [2023-12-03 08:10:43.530] ERROR [pool-5-thread-5] InventorySyncTask.lambda$run$0:121
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}

      eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
      at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
      at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
      at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?]
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
      at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
      at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      at java.lang.Thread.run(Thread.java:829) [?:?]
      Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
      at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
      at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
      ... 23 more
      [2023-12-03 08:10:43.531] ERROR [pool-5-thread-5] ITask.finishTask:47
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Synchronization task failed:

      eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
      at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
      at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
      at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
      at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
      at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?]
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
      at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
      at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) ~[engine.jar:?]
      at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      at java.lang.Thread.run(Thread.java:829) [?:?]
      Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
      at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
      at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
      at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
      ... 23 more
      [2023-12-03 08:10:43.555] INFO [pool-5-thread-5] ITask.lambda$finishTask$2:71
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Finishing 3655a122-fbfb-4a7f-ba52-595a0de48f5f: [Inventory synchronization] from 2023-12-03T08:07:46+01:00 to 2023-12-03T09:07:46+01:00 Task with message "Synchronization task failed: Server has encountered the runtime error (RUNTIME_ERROR)"

      [2023-12-03 08:10:43.606] INFO [pool-5-thread-5] InventorySyncTask.sendInventory:620
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Sending inventory with type FINALIZE to Server

      [2023-12-03 08:10:43.624] INFO [pool-5-thread-5] InventorySyncTask.sendInventory:624
      [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Finished step: FINALIZE

      0
    • L
      L

      Hello @user

      If you still have this problem, please send us the full DEBUG logs of this operation (server and node logs). If you cannot attach logs to this thread, you can put them somewhere, e.g. OneDrive or similar, and send the link.

      Here is the documentation on how to switch logs to debug: Logs in DEBUG

      0