Storware Backup & Recovery
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Store task failed: Server has encountered the runtime error (RUNTIME_ERROR)

    Scheduled Pinned Locked Moved
    Virtual Environments
    3
    5
    1.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • U
      user
      last edited by user

      Hi.
      I'm testing the deployed OVA storware instance.
      I have 2 ESXi hosts and 1 XCP-ng host. Backup storage is mounted as NFS share. The staging folder is placed on that share too.
      I've successfully backed up windows VM from vmware host. Then I tried to backup ubuntu VM from xcp-ng host. Because of long backup time I left backup task to proceed and came back to it in the morning. I saw the error "Store task failed: Server has encountered the runtime error (RUNTIME_ERROR)". All my hosts are disappeared from the inventory and I couldn't rescan them again. I see the XVA image of the ubuntu VM in the staging folder. After vprotect service reload my hosts returned to the inventory. But there are no any signs of the failed backup in the "Instances". I've tried to backup again and now everything looks fine except I can't mount backuped vm with error "Cannot mount backup made by hypervisor with VM image strategy enabled".

      1 Reply Last reply Reply Quote 0
      • A
        Argus Storware Professional Services
        last edited by

        Hi we are not supporting file level restore for XCP we cannot read filesystems in its backup files, it is a technical limitation
        You can check what features we support or not for different entities here

        1 Reply Last reply Reply Quote 0
        • U
          user
          last edited by

          Hi. Thanks for information.
          Anyway the first mentioned error is not belonged to XCP-ng limitations. OK. I'll watching for it.

          1 Reply Last reply Reply Quote 0
          • U
            user
            last edited by

            Today all hosts are gone from the inventory with the same error. Why can't I attach files in thread? This is a part of the log.

            eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
            at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
            at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
            at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) [?:?]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?]
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]
            at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
            at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) [?:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) [engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) [engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
            at java.lang.Thread.run(Thread.java:829) [?:?]
            Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
            at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
            at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
            ... 23 more
            [2023-12-03 08:10:43.528] ERROR [pool-5-thread-5] InventorySyncTask.lambda$getRetryPolicy$24:699
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Attempt number 6 failed.

            eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
            at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
            at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
            at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) [?:?]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?]
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?]
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) [?:?]
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) [?:?]
            at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [?:?]
            at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) [?:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) [engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) [engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
            at java.lang.Thread.run(Thread.java:829) [?:?]
            Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
            at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
            at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
            ... 23 more
            [2023-12-03 08:10:43.529] ERROR [pool-5-thread-5] InventorySyncTask.lambda$getRetryPolicy$27:702
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Failed to send VMS, max retries exceeded

            [2023-12-03 08:10:43.530] ERROR [pool-5-thread-5] InventorySyncTask.lambda$run$0:121
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}

            eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
            at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
            at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
            at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
            at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
            at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
            at java.lang.Thread.run(Thread.java:829) [?:?]
            Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
            at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
            at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
            ... 23 more
            [2023-12-03 08:10:43.531] ERROR [pool-5-thread-5] ITask.finishTask:47
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Synchronization task failed:

            eu.storware.vprotect.engine.api.APIErrorException: Server has encountered the runtime error (RUNTIME_ERROR)
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:452) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$sendInventory$21(InventorySyncTask.java:623) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$toCtxSupplier$9(Functions.java:228) ~[engine.jar:?]
            at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[engine.jar:?]
            at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[engine.jar:?]
            at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[engine.jar:?]
            at dev.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:220) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.sendInventory(InventorySyncTask.java:623) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncAndUpdateProgress(InventorySyncTask.java:266) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.lambda$updateProgress$1(InventorySyncTask.java:254) ~[engine.jar:?]
            at java.util.stream.ReduceOps$1ReducingSink.accept(ReduceOps.java:80) ~[?:?]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
            at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
            at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
            at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
            at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:563) ~[?:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.updateProgress(InventorySyncTask.java:254) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.syncHypervisor(InventorySyncTask.java:201) ~[engine.jar:?]
            at eu.storware.vprotect.engine.task.InventorySyncTask.run(InventorySyncTask.java:108) [engine.jar:?]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
            at java.lang.Thread.run(Thread.java:829) [?:?]
            Caused by: io.swagger.client.ApiException: {"errorCode":"RUNTIME_ERROR","errors":[],"message":"Server has encountered the runtime error"}
            at io.swagger.client.ApiClient.invokeAPI(ApiClient.java:706) ~[engine.jar:?]
            at io.swagger.client.api.TaskControllerApi.updateInventory(TaskControllerApi.java:1769) ~[engine.jar:?]
            at eu.storware.vprotect.engine.api.controller.TaskController.sendInventory(TaskController.java:450) ~[engine.jar:?]
            ... 23 more
            [2023-12-03 08:10:43.555] INFO [pool-5-thread-5] ITask.lambda$finishTask$2:71
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Finishing 3655a122-fbfb-4a7f-ba52-595a0de48f5f: [Inventory synchronization] from 2023-12-03T08:07:46+01:00 to 2023-12-03T09:07:46+01:00 Task with message "Synchronization task failed: Server has encountered the runtime error (RUNTIME_ERROR)"

            [2023-12-03 08:10:43.606] INFO [pool-5-thread-5] InventorySyncTask.sendInventory:620
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Sending inventory with type FINALIZE to Server

            [2023-12-03 08:10:43.624] INFO [pool-5-thread-5] InventorySyncTask.sendInventory:624
            [3655a122-fbfb-4a7f-ba52-595a0de48f5f] Finished step: FINALIZE

            L 1 Reply Last reply Reply Quote 0
            • L
              lsroga @user
              last edited by

              Hello @user

              If you still have this problem, please send us the full DEBUG logs of this operation (server and node logs). If you cannot attach logs to this thread, you can put them somewhere, e.g. OneDrive or similar, and send the link.

              Here is the documentation on how to switch logs to debug: Logs in DEBUG

              Łukasz Sroga
              IT Systems Specialist at Storware

              1 Reply Last reply Reply Quote 0
              • First post
                Last post