Creating Backups¶

Disaster recovery is an important role for all lab admins. If anyone cares about the data within the instance and/or the availability of the instance, then steps need to be taken, in advance, to prepare for the worst. Additionally, these steps will also help with growing your lab by making it easier to deploy an initial working setup onto a new master or worker without going through all the steps manually.

Backups may take up a lot of space, and take a long time to perform. This is a particular problem with the master where the added load could impact on overall performance. Investigate methods of writing backups to external storage without causing undue latency.

Backups may also contain sensitive information like database passwords, authentication and encryption certificates and keys. Protect your backups from access by anyone except the admins.

Any backup which has not been tested to successfully restore the previous service is not worth having as a backup. It may be infeasible to test every backup but intermittently, one backup needs to be fully tested and all backups then checked that the equivalent files exist in each.

Avoid backup solutions which try to image the entire drive. For such systems to work reliably with a database, the database server would have to be stopped. That first requires that all lava-server services are stopped and all running TestJobs are canceled. i.e. A backup like this would require a maintenance window (scheduled downtime) every time the backup was to be performed.

If you use configuration management, as recommended in Best practice, then this forms part of your backup and recovery plan.

# apt-get download lava-server=2016.12-1

$ dpkg-query -W -f '${version}\n' 'lava-server'

# apt-get download -t jessie-backports lava-server=2016.12-1~bpo8+1

Restoring a master from a backup¶

1. Disable access to the system while restoring. For example, set up routing to prevent a newly installed apache service from responding on the expected IP address and virtual host details to avoid confusing users. Place a holding page elsewhere until the restoration is fully complete and tested.

   Caution

   Users must not be allowed to access the instance during the restore. There must be no database accesses outside the explicit control of the admin performing the restore.

   Some admins may already have alternative apache configuration which can replace the LAVA interface with a maintenance page whilst still allowing admins to access the LAVA interface, e.g. on a different hostname or IP address, to do some of the steps below. Such arrangements can be very useful but are outside the scope of the LAVA documentation, so command line operations are shown instead. The goal is the same in each case - cancel test jobs already running and in the queue and set all devices to Maintenance. Make sure that the instance is in full maintenance before re-enabling the LAVA UI.

2. If you are restoring multiple machines, start with the master and only start to restore workers when the master is fully restored but whilst the master remains invisible to users.

3. Prepare the base system and ensure all packages installed at this stage are up to date with the Debian mirrors.

4. If using backports, add the backports apt source and run apt update to populate the apt lists.

5. Install lava-server as per the documentation. Select a version which is slightly earlier or the same as the one installed when the backup was made. Avoid installing any version of lava-server newer than the one which was running when the backup was created. This installation will use an empty database and this is expected.

6. Make sure that this instance actually works. On the command line, you can use:

   $ sudo lava-server manage check --deploy
   

7. Stop all LAVA services - the new installation will have automatically started all services using the empty database but until the database state can be updated, there must be no attempt to reserve devices for jobs in the queue or add test jobs to the queue.

   • service lava-server-gunicorn stop
     • If your local configuration permits only admins to see the LAVA UI, then this one LAVA service can be left running.
   • service lava-master stop
   • service lava-logs stop
   • service lava-slave stop
   • service lava-publisher stop

8. Dump the (empty) initial database and restore the database from the backup.

   See also

   Migrating postgresql versions for how to drop the initial cluster and replace with the cluster from the backup.

9. Using the command line, change the health of all devices which are not Retired to Maintenance.

   Device health can be changed by looping over each device:

   $ sudo lava-server manage devices update --health MAINTENANCE --hostname ${HOSTNAME}
   

   or, with 2018.12 and newer, you can use the updated maintenance helper:

   $ sudo lava-server manage maintenance --force
   

   Check the status of all devices. There should be no devices in the following listings:

   $ sudo lava-server manage devices list --health GOOD
   $ sudo lava-server manage devices list --health LOOPING
   $ sudo lava-server manage devices list --health UNKNOWN
   

10. Restore the other configuration on the master:

    • Any template changes from the packaged defaults
    • Device dictionaries
    • Per-dispatcher configuration
    • Test job log files from your backup
    • Other elements, as required.

11. Start all LAVA services

    • service lava-master start

    • service lava-logs start

    • service lava-slave start

    • service lava-publisher start

    • service lava-server-gunicorn restart

      • If your local configuration cannot restrict the LAVA UI to only admins, then take note that this will restore visibility to users. Before restarting lava-server-gunicorn:

        • Check that the other services are all running correctly

        • Check that the health for all devices is set to MAINTENANCE or RETIRED.

        • Fail any running test jobs:

          $ lava-server manage jobs fail <job_id>
          

12. Check the logs to ensure that all services are running without errors.

13. If there are any devices on the master, put some of those devices online and run some health checks. If not, do as much of a check as possible on the master and then move to restoring the workers, if that is necessary.

14. Once all workers are restored and all devices are both online and have passed a health check, the restoration is complete. If a holding page was used, it can be taken down and the normal access to the instance restored to users.

Restoring a worker from backups¶

This is a much simpler process than a master (or a V1 worker which is arguably more complex to restore than a master). Workers should only be restored after the master has been restored and whilst all devices are still in maintenance.

The only critical LAVA element for a worker to be restored from backup is the ZMQ communication back to the master. This is retained in /etc/lava-dispatcher/lava-slave.

Other files may be required by specific labs and may already be handled by configuration management, e.g.:

• /etc/ser2net.conf.
• Local PDU scripts.
• udev rules for particular devices or services.

Once the base system has been restored and lava-dispatcher has been installed at the same version as previously, the ZMQ configuration can simply be put back into place and lava-slave restarted.

$ sudo service lava-slave restart

The worker will now be able to respond to test job messages sent by the master.