Question

1 Touch system state backup failed


Badge +2

I have an Oracle Linux 7.9 system that was configured with 1 touch and ive done a BMR on different hardware (same configuration) with a new IP and hostname and also selected the option to clone.  The one touch restore/clone went well on the other server however, a full 1 touch back up on the original is now throwing an error and Im unsure what I can do the rectify the issue.
The specific message I receive is:

Error Code: [6:966]
Description: System state backup failed : [Failed to create a tar archive containing /tmp/initrd.img /boot/vmlinuz-5.4.17-2102.201.3.el7uek.x86_64 /tmp/ARCH /tmp/binfmt /tmp/ver ]
Source: Hostname, Process: sr

 

when i look at the server, initrd.img doesnt exist in /tmp.  Neither do the subfolders /ver or ARCH
Ive tried to reinstall/repair the client on the original machine, but it has not helped.  

Any ideas what i can do to get a full backup of the original machine as its still in production and has had some chanages that need to be backed up.

TIA for any help!


11 replies

Userlevel 4
Badge +12

Hi @Commvault User 

 

Could you ensure /tmp has enough disk space 

 

https://documentation.commvault.com/2022e/expert/3454_1_touch_for_linux_system_requirements_02.html

Hard Drive

2 GB of free disk space is required for the /tmp directory.

No additional disk space required beyond File System Agent disk space requirements.

 

Additionally, check the sr.log on the linux server for details on the failure.

 

Regards,

Badge +2

Thank you Gowri for your quick response!

Theres 47 GB of free space left on “ / “ where tmp is located.

Ive tried to comb through the sr.log but is there something specific i should be looking for? Its quite large and Im not sure what i should be looking for exactly as it pertains to this error.

Thanks again

 

Badge +2
Noticed these errors in the log - im running as root.

So combing through the logs it appears as tho it may be failing somewhere after these warnings.  Im curious why it would state it doesnt have permission though, ive explicitly set the backup to run as root.
Any ideas?

Userlevel 4
Badge +12

Hi @Commvault User 

The issue could be due to mount options too

  1. try running mount and verify the mount options on /opt and /tmp 
  2. check if SELinux is enabled if yes either disable it or follow the doc to add a policy https://documentation.commvault.com/2022e/expert/24204_selinux_configuration.html

 

For a test try creating a file on /tmp

make it as RWX and try running a test script to ensure it allows execute files from tmp 

Have also see some environments where /tmp is mounted with noexec option which could cause such issues 

Badge +2

SELinux is enabled, but its set to permissive and according to the doc requires no config.

Could you explain what I should verify in the mount options?

Just to clarify, i have already successfully run a full backup on this machine.  Its only since doing a one touch restore on another machine that this one is now failing backups.

Userlevel 4
Badge +12

 

  • provide the output of following
  1. mount
  2. ls -l / | grep -i opt
  3. ls -l / | grep -i tmp
  • Following commands is to test if we are able to create and execute a script file from /tmp
  1. echo “date” » /tmp/testfile 
  2. chmod 755 /tmp/testfile
  3. ./tmp/testfile
Badge +2

Hi Gowri, 
I was able to create and execute the script.  Here is the output you requested.

 

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=263882176k,nr_inodes=65970544,mode=755)

securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)

tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)

cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)

pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)

cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)

cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)

cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)

cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)

cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)

cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)

cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)

cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)

cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)

cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)

cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)

configfs on /sys/kernel/config type configfs (rw,relatime)

/dev/mapper/hostname-root on / type ext4 (rw,relatime,seclabel)

selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)

systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=37955)

mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)

debugfs on /sys/kernel/debug type debugfs (rw,relatime,seclabel)

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)

binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)

/dev/mapper/hostname-home on /home type ext4 (rw,relatime,seclabel)

/dev/mapper/hostname-mounthome on /mounthome type ext4 (rw,relatime,seclabel)

/dev/mapper/mountname1 on /mountname1 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname2 on /mountname2 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname3 on /mountname3 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/oracledb on /oracledb type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname4 on /mountname4 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname5 on /oracledb/mountname5 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname6 on /oracledb/mountname5/mountname6 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname7 on /oracledb/mountname5/mountname7 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname8 on /oracledb/mountname5/mountname8 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname9 on /oracledb/mountname5/mountname9 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname10 on /mountname10 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname11 on /oracledb/mountname5/mountname11 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/kdump on /var/kdump type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

/dev/mapper/mountname12 on /oracledb/mountname5/mountname12 type ext4 (rw,relatime,seclabel,discard,stripe=512,_netdev)

tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=52785916k,mode=700)

tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime,seclabel)

tmpfs on /run/user/54321 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=52785916k,mode=700,uid=54321,gid=54321)


 

[root@hostname /]# ls -l / | grep -i opt

drwxr-xr-x.   6 root   root      4096 May 17 13:18 opt

[root@hostname /]# ls -l / | grep -i tmp

drwxrwxrwt.  10 root   root      4096 May 18 08:29 tmp

[root@hostname /]# echo "date" >> /tmp/testfile

[root@hostname /]# chmod 755 /tmp/testfile

[root@hostname /]# ./tmp/testfile

Thu May 18 08:31:49 EDT 2023

[root@hostname /]#

Userlevel 4
Badge +12

tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)

 

tmpfs is mounted with noexec option, please engage your Linux admin and remount it without noexec option and rerun the backup

 

Regards,

Badge +2

Thank you for your time and effort Gowri,

We’ve remounted tmpfs with the exec option and still receive the same error when we run the backup.
I will note: the cloned server of this one - where we did the BMR - is able to be backed up, and it has the noexec option on tmpfs.

Is there something else going on here?

Userlevel 4
Badge +12

Thank you for the update.

 

Could you please open a support incident and one of us could review it further.

 

Regards,

Gowri Shankar

Badge +2

Ok thank you Gowri,

I will open a support incident and reference this thread.

Reply