postgres backup failed
-
Hi! Hope you can help me here.
Have some Postgres backups that from one day to another started failing
All I can see in the log is:[f3b671d4-8195-464a-9d3a-e7fc1e2cdd85] Backup task failed: eu.storware.vprotect.engine.exception.ExternalAPIException: File could not be saved: /vprotect_data/backups/cmdbuild-dev.apps.ocp4.domain.com_cmdbuild_30021.gz (Resource temporarily unavailable). /vprotect_data/backups/cmdbuild-dev.apps.ocp4.domain.com_cmdbuild_30021.gz (Resource temporarily unavailable)I can see on the file system that the file is created and increases it's size while processing the job but at the end it is not deleted as usual:
This job was working perfectly until 3 days ago, nothing was changed.
Any idea what it might be?
Full log here: https://pastebin.com/gHZmgXgc
-
Hello again.
Any idea of what's the issue here?
In the meanwhile it seems the problem is not related to Postgres backups only but all those that have Data Export configured:
-
Hello @carvalhoiv
Please tell me where is your backup destination located? Is this some local or remote resource? Can you show permissions for /vprotect_data and mounted resources? What is the exact version of vprotect packages? Please send the result of these commands:
rpm -qa | grep vprotect ls -l / ls -l /vprotect_data df -hAre you able to manually execute this vp_backup_postgresql_remote.sh script? Does the backup of the internal vprotect database work properly?
-
@lsroga Hello!
Thanks for the reply.
vProtect version is 6.2.0-41.The backup destination is a boostfs file system.
Haven't tried to run the script manually. The schedule job sometimes works, other times don't with the error reports.Internal database works ok.
Here follows the requested commands output:
[Wed Jul 31 15:53:08] root@cvprotect01:~ $ rpm -qa | grep vprotect vprotect-node-6.2.0-34.el9.x86_64 vprotect-server-6.2.0-41.el9.x86_64 [Wed Jul 31 15:53:18] root@vprotect01:~ $ ls -l / total 29 dr-xr-xr-x. 2 root root 6 Aug 9 2021 afs lrwxrwxrwx. 1 root root 7 Aug 9 2021 bin -> usr/bin dr-xr-xr-x. 6 root root 4096 Jul 11 15:47 boot drwxr-xr-x. 21 root root 3360 Jul 31 15:10 dev drwxr-xr-x. 129 root root 8192 Jul 23 10:05 etc drwxr-xr-x. 2 root root 6 Aug 9 2021 home lrwxrwxrwx. 1 root root 7 Aug 9 2021 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Aug 9 2021 lib64 -> usr/lib64 drwxr-xr-x. 2 root root 6 Aug 9 2021 media drwxr-xr-x. 3 root root 22 May 10 14:33 mnt drwxr-xr-x. 7 root root 84 Jul 23 10:05 opt dr-xr-xr-x. 342 root root 0 Jul 15 11:41 proc dr-xr-x---. 6 root root 4096 Jul 30 09:39 root drwxr-xr-x. 42 root root 1340 Jul 31 15:10 run lrwxrwxrwx. 1 root root 8 Aug 9 2021 sbin -> usr/sbin drwxr-xr-x. 2 root root 6 Aug 9 2021 srv dr-xr-xr-x. 13 root root 0 Jul 15 11:41 sys drwxrwxrwt. 19 root root 4096 Jul 31 15:53 tmp drwxr-xr-x. 14 root root 180 Jun 19 18:14 usr drwxr-xr-x. 22 root root 4096 Jul 11 15:47 var drwxrwxrwx. 10 vprotect vprotect 579 Jul 31 13:02 vprotect_data [Wed Jul 31 15:53:28] root@vprotect01:~ $ ls -l /vprotect_data/ total 4 drwxr-xr-x. 28 vprotect vprotect 2176 Jul 31 03:03 backups drwxr-xr-x. 2 vprotect vprotect 101 Jun 20 11:29 export drwxr-xr-x. 2 vprotect vprotect 101 Jul 15 17:52 import drwxr-xr-x. 2 vprotect vprotect 101 Jun 11 19:25 mount [Wed Jul 31 15:53:40] root@vprotect01:~ $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 9.7G 0 9.7G 0% /dev/shm tmpfs 3.9G 161M 3.7G 5% /run /dev/mapper/cs-root 45G 9.1G 36G 21% / /dev/sda1 960M 356M 605M 38% /boot boostfs 499T 154T 345T 31% /vprotect_data tmpfs 2.0G 4.0K 2.0G 1% /run/user/0Here are new examples of another application backup that fails first, then the relaunch is successful and the next scheduled backup works as well:
2024-07-29__03.00.20_job scheduled failed - https://pastebin.com/k9xdE4F4
2024-07-29__09.45.12_job relaunched OK - https://pastebin.com/0erYHa8b
2024-07-30__03.00.22_job scheduled OK - https://pastebin.com/xzdp4TsA