12

How to Schedule Daily Rolling EBS Snapshots

Posted (Updated ) in Linux

Update 2016-08-02: Some AWS commands no longer return any responses. Updated the script accordingly.

I want to schedule backups of my Ubuntu EC2’s EBS on a daily rolling schedule – ie a backup will occur once each day, and after 7 days the oldest snapshot is deleted – so there will always be 1 weeks worth of backups.

Prerequisites

First we need an IAM user with permissions only to do what our backup script requires. Create one in the IAM section of AWS console (I named mine snapshot-backup) and in the Inline Policies area give it the following policy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

 

For the script itself we’ll need AWS command line tools:

1
2
3
4
5
6
7
8

sudo apt-get install awscli
 
# Follow the config documentation http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
aws configure --profile=ssbackup
AWS Access Key ID [None]: AAAAAAAAAAAAAAAAAA
AWS Secret Access Key [None]: bbbbbbbbbbbbbbbbbbbbbbbbbbbb
Default region name [None]: ap-southeast-2
Default output format [None]: json

 

The Script

My script will perform a snapshot of each volume passed to it, then delete any snapshots older than the given number of days. It will only delete snapshots that it created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

#!/usr/bin/env python3
#
# Creats a new snapshot for all passed volumes deletes old snapshots
#
# http://jayaraj.sosblogs.com/The-first-blog-b1/AWS-Automated-EBS-Snapshot-Script-b1-p3.htm
#
# Usage:
# python3 ssbackup.py --volume-ids=vol-1a23bcd4 --volume-ids=vol-2b34cde5 --expiry-days=7
#
 
import argparse
import subprocess
import json
import logging
import time, datetime, dateutil.parser
 
profile = 'ssbackup'       # Your AWS CLI profile
region = 'ap-southeast-2'  # AWS region volumes/snapshots are located
 
def bash(command):
	process = subprocess.Popen(command, stdout=subprocess.PIPE)
	return process.communicate()[0].decode('utf-8')
 
def getOurSnapshots():
	"""
		Return a list of snapshot Dicts created with this plugin.
	"""
	return json.loads(bash([
			"aws", "ec2", "describe-snapshots",
			"--filters", "Name=tag-key,Values=Group", "Name=tag-value,Values=ssbackup",
			"--profile", profile,
			"--region", region,
			"--output=json"
		]))['Snapshots']
 
def createSnapshots(volumeIds):
	"""
		Return True if snapshots of the given volumes are created, else False
 
		Keyword arguments:
		volumeIds -- List of EBS volume IDs
	"""
	# Create the snapshots
	snapshots = []
	for volumeId in volumeIds:
		snapshots.append(createSnapshotForVolume(volumeId))
 
	# Add Name and Group tags to the snapshot
	if len(snapshots):
		snapshotIds = []
		date = time.strftime("%Y-%m-%d")
 
		for snapshot in snapshots:
			snapshotIds.append(snapshot['SnapshotId'])
 
		# create-tags returns no output now so just perform the command and
		# return True
		bash([
			"aws", "ec2", "create-tags",
			"--resources", ' '.join(snapshotIds),
			"--tags", "Key=Name,Value='Backup "+date+"'", "Key=Group,Value=ssbackup",
			"--profile", profile,
			"--region", region,
			"--output=json"
		])
 
		return True
 
	return False
 
def createSnapshotForVolume(volumeId):
	"""
		Return a Dict of a created snapshot for the given EBS volume
 
		Keyword arguments:
		volumeId -- An EBS volume ID
	"""
 
	date = time.strftime("%Y-%m-%d")
	message = "Creating snapshot for volume "+volumeId+"..."
	response = json.loads(bash([
		"aws", "ec2", "create-snapshot",
		"--volume-id", volumeId,
		"--description", "Backup "+date,
		"--profile", profile,
		"--region", region,
		"--output=json"
	]))
	message += response['SnapshotId']
	logging.info(message)
 
	return response
 
def deleteOldSnapshots(snapshots, max_age):
	"""
		Delete all listed snapshots older than max_age
	"""
	snapshotIds = []
	date = datetime.datetime.now()
 
	for snapshot in snapshots:
		snapshotDate = dateutil.parser.parse(snapshot['StartTime']).replace(tzinfo=None)
		dateDiff = date - snapshotDate
 
		if dateDiff.days >= max_age:
			message = "Deleting snapshot "+snapshot['SnapshotId']+" ("+str(dateDiff.days)+" days old)..."
			# delete-snapshot no longer returns any output
			bash([
				"aws", "ec2", "delete-snapshot",
				"--snapshot-id", snapshot['SnapshotId'],
				"--profile", profile,
				"--region", region,
				"--output=json"
			])
 
			message += "done"
			logging.info(message)
 
if __name__ == '__main__':
	parser = argparse.ArgumentParser(description='ADD YOUR DESCRIPTION HERE')
	parser.add_argument('-i','--volume-ids', help='EBS volume ID', required=True)
	parser.add_argument('-d','--delete-old', help='Delete old snapshots?', required=False, type=bool, default=True)
	parser.add_argument('-x','--expiry-days', help='Number of days to keep snapshots', required=False, type=int, default=7)
	args = parser.parse_args()
 
	logging.basicConfig(filename='ssbackup.log', level=logging.DEBUG, format='%(asctime)s:  %(message)s', datefmt='%Y-%m-%d %I:%M:%S%p')
 
	# Get all active volumes
	volumeIds = args.volume_ids.split(',')
	# Create the snapshots
	if len(volumeIds):
		snapshots = createSnapshots(volumeIds)
		pass
 
	# Delete snapshots older than expiry-days
	if args.delete_old:
		deleteOldSnapshots(getOurSnapshots(), args.expiry_days)

 

Put it to Work

To run this script automatically each day, add it to your crontab like so:

1
2

# m   h  dom mon dow   command
  5   0    *   *   *   python3 /home/user/ssbackup.py --volume-ids=vol-a1bcd2e3 --expiry-days=7

 

The script will automatically create an ssbackup.log file that sits next to it describing exactly what it did on last run. For example here’s my log when running a backup with expiry-days=0 a couple of times:

1
2
3
4

2015-06-18 01:09:50AM:  Creating snapshot for volume vol-a1bcd2e3...snap-a123b4cd
2015-06-18 01:09:52AM:  Deleting snapshot snap-b123c4de (0 days old)...done
2015-06-18 02:36:28AM:  Creating snapshot for volume vol-a1bcd2e3...snap-c012345d
2015-06-18 02:36:30AM:  Deleting snapshot snap-a123b4cd (0 days old)...done