vMwareStrategyException

benitowarez

I am having trouble getting an initial backup to be successful. When attempting to open the log I get an error stating "server has encountered an unexpected error." Below is the status info.

VmwareStrategyException: VMware-service error while operation: OPEN_DISK_HANDLER. Error codes: 14009

Any insight would be greatly appreciated.

krzysztof.sz

Hi!
Thank you for the post!

Could you tell me more about your infrastructure?

Which version of VMWare (and VMware VDDK) do you have?

Regards

benitowarez

@krzysztof-sz

Absolutely! I have a small homelab consisting of two Dell R420 servers both running ESXi 8.0. This environment uses a standard switch with a VCenter Server Appliance managing it. The storage is running an NFS 3.0 with TrueNAS.

I am not sure what VDDK is honestly. I googled it but do not know that I am using it or if I am I do not know what version.

I can get the database backup to work fine to the NAS and the inventory sync works great. The VMs do perform a snapshot just fine but fails with the error above.

Argus

@benitowarez Hi, we spoke on the mail, can you give me your vprotect version ? It is possible that this error was already fixed

benitowarez

@Argus

I am currently using version 6.0.0-13. I am running the virtual machine version.

benitowarez

@Argus

I have updated to 6.0.0-24, however, the issue persists.

Argus

@benitowarez I see slightly different error from vmware I will check this with our dev tomorrow

benitowarez

@Argus

I now have a new error. I updated my hosts to 8.0U2 and my vCenter Server Appliance to 8.0.2 and now I am unable to synchronize the inventory.

The workflow execution details show an error of:

Synchronization task failed: APIErrorException: There was a problem with parsing error messsage io.swagger.client.ApiException: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="ht...

I am not sure what log this might be caught in to further investigate. Have you all run into this error before?

Argus

@benitowarez Can we get logs from this one also?

benitowarez

@Argus

I was unable to attach the zip of logs in this thread, but I have emailed them to your support email. Below is a snippet of the api log file.

[2023-09-27 14:13:24] DEBUG [http-thread-pool::http-listener(7)] {r=NODE, s=65e6119a806c1a3c01721bd76706} RequestLoggingFilter.filter:30
REQUEST: PUT http://localhost:8080/api/tasks/35cb7796-3150-4856-b0a8-cb95fb67b4fc/state| media type: application/json| content: {"state":{"name":"FAILED"},"statusInfo":"Synchronization task failed: APIErrorException: There was a problem with parsing error messsage\nio.swagger.client.ApiException: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"></style> </head><body><h1>HTTP Status 404 - Not Found</h1><hr/><p><b>type</b> Status report</p><p><b>message</b>Not Found</p><p><b>description</b>The requested resource is not available.</p><hr/><h3>Payara Micro #badassfish</h3></body></html>\nUnexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (String)"<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525"[truncated 546 chars]; line: 1, column: 2]"}

[2023-09-27 14:13:24] DEBUG [http-thread-pool::http-listener(7)] {r=NODE, s=65e6119a806c1a3c01721bd76706} TaskService.updateSingleTaskState:427
GUID: 35cb7796-3150-4856-b0a8-cb95fb67b4fc, task state: FAILED, status info: Synchronization task failed: APIErrorException: There was a problem with parsing error messsage
io.swagger.client.ApiException: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"></style> </head><body><h1>HTTP Status 404 - Not Found</h1><hr/><p><b>type</b> Status report</p><p><b>message</b>Not Found</p><p><b>description</b>The requested resource is not available.</p><hr/><h3>Payara Micro #badassfish</h3></body></html>
Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (String)"<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>Payara Micro #badassfish - Error report</title><style type="text/css"><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525"[truncated 546 chars]; line: 1, column: 2]

[2023-09-27 14:13:24] DEBUG [http-thread-pool::http-listener(7)] {r=NODE, s=65e6119a806c1a3c01721bd76706} ResponseLoggingFilter.filter:34
RESPONSE | PUT http://localhost:8080/api/tasks/35cb7796-3150-4856-b0a8-cb95fb67b4fc/state | status code: 204

Argus

@Argus Did you only update vprotect-server ?

Argus

@benitowarez I checked your logs I see you have a version mismatch update vprotect-node package to the latest version and server to we have a new one

benitowarez

@Argus

All up to date now and the inventory sync works again! Thank you.

benitowarez

@Argus
I was just checking in for any updates there might be for the OPEN HANDLER error.

Argus

@benitowarez oh so after the update you still get this DISK_HANDLER error?

benitowarez

@Argus
Yes I do. Something else to order along is I have found after a failed backup attempt, the virtual machine is no longer able to vMotion. I am able to get this through the process below.

https://kb.vmware.com/s/article/1029926

I have tested virtual machines from both hosts and it does not seem to change the results.

benitowarez

@Argus

Are there any updates on this issue or anything I can look at to discover the root cause?

Argus

@benitowarez We are investigating but it looks like some error on the VMware side, did this error occur for particular VMs or all VMs ?

benitowarez

@Argus

Thank you.

I just checked my instance and for some reason it is backing up now. The jobs are still running, but they are not failing yet. I do not recall making any changes since the last communication and I will continue to monitor for results.