Difference between revisions of "Backup scripts"

From OpenKM Documentation
Jump to: navigation, search
m (→‎Remote backup with rdiff-backup)
Line 145: Line 145:
 
* [http://wiki.rdiff-backup.org/wiki/index.php/Main_Page rdiff-backup wiki]
 
* [http://wiki.rdiff-backup.org/wiki/index.php/Main_Page rdiff-backup wiki]
  
−
== Local backup to USB disk ==
+
== Local backup to USB disk with rdiff-backup ==
 
USB disk mount point can be defined in ''/etc/fstab'' as:
 
USB disk mount point can be defined in ''/etc/fstab'' as:
  
Line 222: Line 222:
 
   sleep 5; echo ".";
 
   sleep 5; echo ".";
 
done
 
done
 +
</source>
 +
 +
== Backup with duplicity ==
 +
Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.
 +
 +
Duplicity can be installed in Debian / Ubuntu as simple as:
 +
 +
  $ sudo aptitude install duplicity
 +
 +
But it not in the CentOS / RedHat default repositories, so you need to install from another source. This script will help in this installation process:
 +
 +
<source lang="bash">
 +
#!/bin/bash
 +
#ARCH=i386
 +
ARCH=x86_64
 +
 +
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/duplicity-0.6.14-1.el5.$ARCH.rpm
 +
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/ncftp-3.2.2-1.el5.$ARCH.rpm
 +
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/librsync-0.9.7-13.el5.$ARCH.rpm
 +
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-GnuPGInterface-0.3.2-2.el5.noarch.rpm
 +
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-boto-1.9b-6.el5.noarch.rpm
 +
 +
rpm -Uvh duplicity-0.6.14-1.el5.$ARCH.rpm ncftp-3.2.2-1.el5.$ARCH.rpm librsync-0.9.7-13.el5.$ARCH.rpm python-GnuPGInterface-0.3.2-2.el5.noarch.rpm python-boto-1.9b-6.el5.noarch.rpm
 
</source>
 
</source>
  
 
[[Category: Installation Guide]]
 
[[Category: Installation Guide]]
 
[[Category: OKM Network]]
 
[[Category: OKM Network]]

Revision as of 11:24, 24 October 2011

These backup scripts use rsync to minimize network load and creates incremental backups, preserving last four backups. For more info, read http://www.mikerubel.org/computers/rsync_snapshots/.

To install the cron job, run: 

$ sudo crontab -e

And add these lines according to your personal configuration: 

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee -a /root/backup.log

Or, if you want to separate log reports by date: 

MAILTO=nomail@openkm.com
@weekly /root/backup.sh | tee /root/backup.$(date +\%Y.\%m.\%d_\%H.\%M.\%S).log

Nota clasica.png MAILTO may also be used to direct mail to multiple recipients by separating recipient users with a comma.

For more information take a look at Crontab quick reference

Remote backup with rsync

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
FILES="/home/openkm"
BACKUP_DIR="/mnt/backup"
## END CONFIG ##
echo -e "### BEGIN: $(date +"%x %X") ###\n"

# Stop JBoss
/etc/init.d/jboss stop
while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
  sleep 5; echo ".";
done

# Copy to backup server
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded $FILES backup@server:$BACKUP_DIR/$HOST

# Start JBoss
/etc/init.d/jboss start
echo -e "\n### END: $(date +"%x %X") ###"

Remote backup with rsync and rotation

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
FILES="/home/openkm"
BACKUP_DIR="/mnt/backup"
## END CONFIG ##
echo -e "### BEGIN: $(date +"%x %X") ###\n"

# Stop JBoss
/etc/init.d/jboss stop

# Copy to backup server
ssh backup@server "cd $HOST; rm -rf backup.3; mv backup.2 backup.3; mv backup.1 backup.2; mv backup.0 backup.1"
rsync -apzhR --stats --delete --exclude=*~ --delete-excluded --link-dest="$BACKUP_DIR/$HOST/backup.1" \
$FILES backup@server:$BACKUP_DIR/$HOST/backup.0

# Start JBoss
/etc/init.d/jboss start
echo -e "\n### END: $(date +"%x %X") ###"

Local backup with rsync and rotation

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
FILES="/home/openkm"
BACKUP_DIR="/mnt/backup"
## END CONFIG ##
echo -e "### BEGIN: $(date +"%x %X") ###\n"

# Stop JBoss
/etc/init.d/jboss stop

# Calculate snapshot
LAST_SNAPSHOT=`ls -ltr $BACKUP_DIR | tail -1 | awk {'print $8'} | cut -d . -f 2`
NEW_SNAPSHOT=$((LAST_SNAPSHOT+1))

# Copy to backup server
rsync -apzhR --stats --delete --exclude=*~ --exclude="$JBOSS_HOME/cache" --delete-excluded \
--link-dest="$BACKUP_DIR/$HOST/backup.$LAST_SNAPSHOT" $FILES "$BACKUP_DIR/$HOST/backup.$NEW_SNAPSHOT"

# Start JBoss
/etc/init.d/jboss start
echo -e "\n### END: $(date +"%x %X") ###"

# Status
echo "=================================";
du -hs $BACKUP_DIR
echo "*********************************";
du -hs --time $BACKUP_DIR/*
echo "=================================";
df -h | grep "$BACKUP_DIR"
echo "=================================";

Remote backup with rdiff-backup


Nota clasica.png rdiff-backup from Debian throws a warning due to a deprecated method [os.popen2 is deprecated], which can be hidden following the steps at How to shut up Python deprecation warnings. 

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
FILES="/home/openkm"
BACKUP_DIR="/mnt/backup"
## END CONFIG ##
echo -e "### BEGIN: $(date +"%x %X") ###\n"

# Stop JBoss
/etc/init.d/jboss stop

# Backup and purge old backups
rdiff-backup --remove-older-than 30B backup@server::$BACKUP_DIR/$HOST
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
  --include $FILES --exclude '**' / backup@server::$BACKUP_DIR/$HOST

# Start JBoss
/etc/init.d/jboss start
echo -e "\n### END: $(date +"%x %X") ###"

# Status
echo "=================================";
rdiff-backup --list-increment-sizes backup@server::$BACKUP_DIR/$HOST
echo "=================================";

More info about rdiff-backup:

Local backup to USB disk with rdiff-backup

USB disk mount point can be defined in /etc/fstab as: 

 /dev/sdb1       /mnt/backup     ext4    defaults        0       0

#!/bin/bash
#
## BEGIN CONFIG ##
HOST=$(uname -n)
MYSQL_PASS=""
OPENKM_HOME="/home/openkm"
JBOSS_HOME="$OPENKM_HOME/jboss-4.2.3.GA"
DATABASE_EXP="$OPENKM_HOME/db"
BACKUP_DIR="/mnt/backup"
## END CONFIG ##

if [ $(id -u) != 0 ]; then echo "You should run this script as root"; exit; fi

echo -e "### BEGIN: $(date +"%x %X") ###\n"
rm -rf $DATABASE_EXP
mkdir -p $DATABASE_EXP

# Mount disk
if mount | grep "$BACKUP_DIR type" > /dev/null; then
  echo "$BACKUP_DIR already mounted";
else
  mount $BACKUP_DIR;
fi

# Stop JBoss
/etc/init.d/jboss stop

# Clean logs
echo "Clean JBoss temporal files."
rm -rf $JBOSS_HOME/server/default/log
rm -rf $JBOSS_HOME/server/default/tmp
rm -rf $JBOSS_HOME/server/default/work

# Backup de MySQL
if [ -n "$MYSQL_PASS" ]; then
  MYSQL_DBS=$(mysqlshow -h localhost -u root -p$MYSQL_PASS | awk '(NR > 2) && (/[a-zA-Z0-9]+[ ]+[|]/) && ($2 != "mysql") && ($2 != "test") && ($2 != "information_schema") { print $2 }');
  
  for DB in $MYSQL_DBS ; do
    echo "* Backuping MySQL data from $DB...";
    mysqldump -h localhost -u root -p$MYSQL_PASS $DB > $DATABASE_EXP/mysql_$DB.sql
  done
  echo "-------------------------------------";
fi

# Backup and purge old backups
rdiff-backup --remove-older-than 30B $BACKUP_DIR/$HOST
rdiff-backup -v 3 --print-statistics --exclude /media --exclude /mnt \
  --include $OPENKM_HOME --exclude '**' / $BACKUP_DIR/$HOST

# Start JBoss
/etc/init.d/jboss start
echo -e "\n### END: $(date +"%x %X") ###"

# Status
echo "=================================";
rdiff-backup --list-increment-sizes $BACKUP_DIR/$HOST
echo "*********************************";
df -h | grep "$BACKUP_DIR"
echo "=================================";

# Umount disk
sync
umount $BACKUP_DIR

Be sure to have the last JBoss startup script, otherwise use this piece of code to ensure that JBoss is stopped before backuping the files: 

while [ "$(ps -ef | grep java | grep jboss | wc -l)" -gt "0" ]; do
  sleep 5; echo ".";
done

Backup with duplicity

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

Duplicity can be installed in Debian / Ubuntu as simple as: 

 $ sudo aptitude install duplicity

But it not in the CentOS / RedHat default repositories, so you need to install from another source. This script will help in this installation process: 

#!/bin/bash
#ARCH=i386
ARCH=x86_64

wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/duplicity-0.6.14-1.el5.$ARCH.rpm
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/ncftp-3.2.2-1.el5.$ARCH.rpm
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/librsync-0.9.7-13.el5.$ARCH.rpm
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-GnuPGInterface-0.3.2-2.el5.noarch.rpm
wget http://download.fedora.redhat.com/pub/epel/5/$ARCH/python-boto-1.9b-6.el5.noarch.rpm

rpm -Uvh duplicity-0.6.14-1.el5.$ARCH.rpm ncftp-3.2.2-1.el5.$ARCH.rpm librsync-0.9.7-13.el5.$ARCH.rpm python-GnuPGInterface-0.3.2-2.el5.noarch.rpm python-boto-1.9b-6.el5.noarch.rpm
Retrieved from "https://www.openkm.com/wiki/index.php?title=Backup_scripts&oldid=5375"