Backups updated - now only using duplicity

My previous post made use of duply as a wrapper around duplicity, but for some reason everything started going a bit wrong, with the cause being *something* that changed in the Python boto library config, but I couldn't work out what.

The solution was to stop using duply and to use duplicity directly. There are many guides that explain how to setup duplicity to use Amazon S3 storage, so I'll not reproduce that information here (Google is your friend), but it's useful to show the script I use to initiate a backup:

export PASSPHRASE="Add your GPG passphrase here"
export AWS_ACCESS_KEY_ID="Add your AWS access key here"
export AWS_SECRET_ACCESS_KEY="Add your AWS secret access key here"

duplicity -v6 --encrypt-key AddYourGPGkeyHere  \
--exclude='**/Junk/*' \
--include='**/Music/*' \
--exclude='**/Documents/temp' \
--include='**/Documents/*' \
--exclude='**/.*' \
--exclude='**/' \
--exclude='**' \
--rsync-options="--size-only" \
/base/dir/to/backup/eg/home/patrick s3://s3-eu-west-1.amazonaws.com/path/to/s3/bucket/or/dir/in/bucket

The above script sets up the authentication variables and then performs a backup of the base directory. I've split the "duplicity" invocation over several lines to make it clearer what's going on.

Note that the final three "exclude" statements will exclude everything from the backup, so it's important to add "include" statements for anything you want backing up.

In the example above, just the "Music" and "Documents" directories in the specified base directory (/base/dir/to/backup/eg/home/patrick) will be backed up, and within the "Documents" directory, the "temp" directory will be excluded.

When deciding what to include or exclude, duplicity will use the first successful include/exclude match, so it's important that the "**/Documents/temp" exclude statement comes before the "**/Documents/*" include statement.

Restoring from backup
Again, there are plenty of guides on the web describing how to do a restore. The only thing to remember is that you will need the three environment variables, shown in the above script, to be setup before performing any duplicity commands. You'll also need to include the "--encrypt-key" argument in the duplicity invocation.

Handy tip
I store my backup script, including all the passphrases and keys, in a LastPass secure note in order that I always have all details needed in the event that some kind of total disaster has occurred.

Using Cygwin on Citrix XenApp to host thin-client Unix applications

These three scripts enable Citrix XenApp to be used as a means to host remote Unix applications, avoiding the need to use something like Opentext VDI Desktop and the like.

This enables legacy Unix apps to be run on a centralised X11 infrastructure, with users getting the benefit of being able to access the running Unix app via the bandwidth-efficient Citrix ICA protocol. This has the added benefit that users can disconnect their Citrix session and leave the Unix desktop session application still running until the user wants to re-connect.

The scripts enable a user to run a published Citrix app which starts X11 on the XenApp server, launches a login panel running on the remote Unix server and, following successful login via SSH using the supplied credentials, launches the remote Unix application on the application server, tunnelling the X11 graphics back to the XenApp server.

The entire Unix desktop and Unix application is rendered by the X11 server on the XenApp server, and all graphics then are relayed to the user's Citrix client via Citrix ICA in the usual way.

The script actually runs the remote Xsession on a different Unix server to the Unix application, but you might not need this.

In fact, the scripts are fairly specific to a consulting job from a few years ago, but no doubt will be useful to someone, especially for:

• Figuring out how to safely and successfully launch and use Cygwin/X11 on XenApp;
• Running an interactive and customised logon panel for a Unix desktop/application session using Expect and Tcl;
• Automating the backend connection from Citrix to the application via SSH.

Scripts can be found here: https://drive.google.com/open?id=0B-QhLh7MxUu1VU0zZVJzQ2NaQjg

Screenshots
TCL/Expect-driven login window. This, including the surrounding Cygwin window decoration, is what the user sees when they launch the XenApp published app (which simply runs the xxxx.sh script on the XenApp server).

Resulting desktop session, ready to run the application. Again, this is exactly what the user sees, including the surrounding Cygwin window decoration.

How to hack VLAN support into Symantec/Veritas Netbackup BMR

These are rough notes. I might update them in the future to make them more useful.

The original issue was that when using Symantec (or Veritas, whatever they are these days) and Bare Metal Restore (BMR) on a network connection that is 802.1q (i.e. trunked VLANs on a single physical connection), it would fail because BMR requires a non-trunked connection.

This hack adds the ability to support BMR using a VLAN that is 802.1q-trunked. In a nutshell, I added a call to vconfig in the BMR initiation script that is part of the BMR boot image, allowing the network connection to be properly configured with the required VLAN.

 ------------------------------------------------

Updated to rc.stage1 file required

This was modified to enable VLAN support. We added an extra question to collect the required VLAN number, and then added this VLAN to each interface that BMR tries to use.


Solution: We needed to add the vconfig binary to the SRT to enable setting of the VLAN by the script. This needs to reside in /sbin in the initrd.img file contained in the SRT.


------------------------------------------------

Missing/incorrect hpahcisr module

This module (the single hpahcisr.ko file) needs to be added to the BMR ISO boot image. Make sure it's exactly the right version for the Linux kernel you're using for the BMR ISO. It will fail to load and abort the BMR process if it's the wrong version.



You will need to add the hpahcisr.ko from either the base OS image that the BMR ISO is based on, or get it from somewhere like 

http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/redhat/5.5Server/packages/i386/



Potential problems are a mis-match between the kernel version of the Linux OS you are using and the kernel version that the module wants. Make sure you get the 32-bit/64-bit version as required.

You can check the kernel version the module wants with:

modinfo hpahcisr.ko

Look for the "vermagic" line, and note any specifics, such as "PAE". This can make all the difference!

You can run the modinfo command from a command line within the BMR session to find out what the module is expecting, and then compare it against the output of "uname -a" to make sure they match.

(For our config, the kernel version in use for the BMR ISO was 2.6.18-194.el5, but the original hpahcisr.ko module reported itself as needing 2.6.18-194.el5PAE)

Downloading the module

Get the hpahcisr module from the link above - you will need the generic "rhel.x86" version for RHEL 5.5 and below,

or the version specific to the RHEL version you are using for later versions of RHEL . You need to unpack it and

extract the hpahcisr.ko file, and then add this file to the BMR ISO image.


(For our config we used

http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/redhat/5.5Server/packages/i386/hpahcisr-

1.2.4-14.rhel5.i686.rpm since 1.2.4-14 was the version of the original PAE-based module)

Extract the downloaded RPM file into the current directory (Using any Linux or Cygwin system) with:

rpm2cpio hpahcisr-1.2.4-14.rhel5.i686.rpm | cpio --make-directories --extract

Find the tar archive within the extracted files, unpack it and find the appropriate hpahcisr.ko file for the target kernel on the BMR ISO image.



(For our config, this was ./hpahcisr-1.2.4_2010_04_08/lib/modules/2.6.18-194.el5/kernel/drivers/scsi/hpahcisr/hpahcisr.ko)


Solution: Add this module into the BMR ISO SRT.

---------------------------------------------------------------------------

Missing bmrc binary

This caused file retrieval from the BMR server to fail:

<graphic goes here>

Solution: Seemed to be caused by an error on our part. Copied over the bin directory from a previous attempt and all was well.


-------------------------------------------------------------------------

Disk partitioning problem


<graphic goes here>


Solution: /usr/openv/bin/vnet not present! Added the bin directory to the SRT.



This didn't fix the LVM problem. This was resolved by applying a Symantec fix.

How to do high-quality CD rips using Linux

1. Install cdparanoia
2. Install ripit
3. Put the following into a script call rip.sh:

drive=$1
echo Using sr${drive}
ripit --outputdir ~/Music/iTunes\ Music/Music --device /dev/sr${drive} --quality 0 --vbrmode new --preset insane --coverart 1 --dirtemplate '"$artist/$album"' --loop 1 --threads 4
4. Do as many concurrent rips as you have CD drives using:

sh rip.sh n

Where n is the CD drive number.

Using duply (and thus duplicity) to backup my Linux server

Aim is to backup selected contents of my Linux server to Amazon S3. This is a work in progress. Linux box is a standard Ubuntu 15.04 desktop install.

1. Install and configure duply as described here: http://old.blog.phusion.nl/2013/11/11/duplicity-s3-easy-cheap-encrypted-automated-full-disk-backups-for-your-servers/ .
   
   I used the version in the default repository rather than adding the PPA
   
   
2. Added the following lines to the ~/.duply/home/conf file ("home" is the name of my duply backup profile) since I'm in Europe:
export S3_USE_SIGV4="True"
export DUPLY_PARAMS="$DUPLY_PARAMS --s3-use-rrs --s3-use-multiprocessing --s3-use-new-style --s3-european-buckets"
3. My boot drive is a small-ish SSD and all my data (music, photos, everything I use) resides on a separate non-SSD drive, with symlinks from the SSD-based home directory to directories on the non-SSD drive. This is so I can install a new OS without wiping all my data. duply/duplicity doesn't like backing up symlinks, so I needed to set the SOURCE variable in the ~/.duply/home/conf file to the location of my data on the non-SSD drive. Setting it to ~/ on the SSD drive results in no backups being taken!
4. I wanted only to backup selected directories. I needed to ensure that duply ignored all fies and directories in the $SOURCE directory that I didn't list explicitly:
- **/dir1/dontbackup
+ **/dir1
- **/*
- **/.*
- **This backs up dir1 but omits the sub-dir dontbackup. It also omits all other files and directories in $SOURCE, including hidden files and directories. In this example, only dir1 (minus the contents of dontbackup) will be backed up.
5. Backup is initiated manually with:
   sudo duply home backup

First post!

This blog will be nothing more than a place for me to publish things that either I or others might find useful.