RECOVER_DISK_FULL script

Both ROBAR-CL and ROBAR-SV require free disk storage space to run correctly. The file system full state, particularly, may cause serious problems in ROBAR-SV. ROBAR-SV is equipped with automatic disk storage space monitoring to avoid the file system full state.

When the available space in the file system where a ROBAR-SV instance runs reaches the limit, an error logging record is generated and the RECOVER_DISK_FULL script is executed.

RECOVER_DISK_FULL causes the following actions:

• The ROBAR-SV instance is terminated.

• The tar files ROBAR-SV.abba_<instance>.<file>.<timestamp>.tar are created in the /var/opt/fujitsu/robar directory, Here <file> has one of the values cmx, trace.inotrc, trace.nbs2, trace.vbs2, trace.netgate_in, trace.netgate_out, trace.robar.

• The contents of the directories cmx and trace of the ROBAR-SV instance is copied into the respective tar file . Next, the tar files are compressed. The contents of the directories cmx and trace is deleted.

• If a specific directory was defined in the ROBAR_BACKUP_DIR configuration parameter (see section "ROBAR_BACKUP_DIR configuration parameter" in chapter "Global configuration file for ROBAR-SV"), the compressed tar files are moved into it.

• The ROBAR-SV instance is then started again.

Disk space monitoring is controlled using the configuration parameter min_disk_space= <limit_value> in the global ROBAR-SV configuration file (see "Global configuration file for ROBAR-SV").

The value <limit_value> means:

• absolute free disk capacity in bytes (permissible value range: 1000000 ... 2147483647)

• relative free disk capacity in percent (1% ... 99%)

The value 0 or 0% means that the free disk space is not monitored. If min_disk_space is not specified, the absolute value 1000000 is used as a default. If min_disk_space is not specified, the absolute value 1000000 is used as a default.

As a ROBAR operator you can adjust the script RECOVER_DISK_FULL to the requirements of your data center at your own responsibility.

If no problems are to occur with the script RECOVER_DISK_FULL, the following message must definitely be observed.

By default, the compressed tar files are stored in the system directory /var/opt/fujitsu/robar. The system support must make sure that there is always sufficient storage capacity available in the directory for all operational situations that might occur, in particular for the use of ROBAR. If this is not the case, the system may become blocked because logging, for instance, is not possible.

You can select a different directory using the configuration parameter ROBAR_BACKUP_DIR (see "Global configuration file for ROBAR-SV") in the global ROBAR-SV configuration file. The selected directory must be situated in a different file system than the ROBAR-SV installation. Here, too, the system support must make sure that sufficient storage capacity is available at all times.

You can use the cron daemon to automate deletion and to make sure that only older data is deleted.

Trace files saved by the RECOVER_DISK_FULL script are unconditionally deleted when a ROBAR-SV instance is started by specifying the Delete trace files parameter. They are deleted periodically and when a ROBAR-SV instance is started if they are older than the number of days defined in the global configuration file of ROBAR-SV using the max_trace_file_age parameter (see section "max_trace_file_age configuration parameter" in chapter "Global configuration file for ROBAR-SV").

The /var/lib/robar_abba/<instance>/RECOVER_DISK_FULL.TRACE file informs you of how far the RECOVER_DISK_FULL script has progressed.

All RECOVER_DISK_FULL jobs write their logs into this file. It should therefore be cleared on a regular basis in order to avoid space problems.

If the RECOVER_DISK_FULL job cannot be completed successfully due to insufficient resources, an entry with the time and reason of the abort is written into the RECOVER_DISK_FULL.TRACE file. Relevant entries are also made in the operating system's logging files. The ROBAR-SV tasks are terminated.

One reason for insufficient resources may be that too many files need to be saved. In this case you have to delete superfluous files manually.

If RECOVER_DISK_FULL jobs cannot be terminated successfully and the automatic restart of the ROBAR-SV instance (see section "Automatic restart of ROBAR-SV instances" in chapter "Automating the behavior of ROBAR-SV") is enabled, restarting a ROBAR-SV instance may continually fail due to a lack of resources.