12

How to Schedule Daily Rolling EBS Snapshots

Posted (Updated ) in Linux

Update 2016-08-02: Some AWS commands no longer return any responses. Updated the script accordingly.

I want to schedule backups of my Ubuntu EC2’s EBS on a daily rolling schedule – ie a backup will occur once each day, and after 7 days the oldest snapshot is deleted – so there will always be 1 weeks worth of backups.

Prerequisites

First we need an IAM user with permissions only to do what our backup script requires. Create one in the IAM section of AWS console (I named mine snapshot-backup) and in the Inline Policies area give it the following policy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

 

For the script itself we’ll need AWS command line tools:

1
2
3
4
5
6
7
8
sudo apt-get install awscli
 
# Follow the config documentation http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
aws configure --profile=ssbackup
AWS Access Key ID [None]: AAAAAAAAAAAAAAAAAA
AWS Secret Access Key [None]: bbbbbbbbbbbbbbbbbbbbbbbbbbbb
Default region name [None]: ap-southeast-2
Default output format [None]: json

 

The Script

My script will perform a snapshot of each volume passed to it, then delete any snapshots older than the given number of days. It will only delete snapshots that it created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
#!/usr/bin/env python3
#
# Creats a new snapshot for all passed volumes deletes old snapshots
#
# http://jayaraj.sosblogs.com/The-first-blog-b1/AWS-Automated-EBS-Snapshot-Script-b1-p3.htm
#
# Usage:
# python3 ssbackup.py --volume-ids=vol-1a23bcd4 --volume-ids=vol-2b34cde5 --expiry-days=7
#
 
import argparse
import subprocess
import json
import logging
import time, datetime, dateutil.parser
 
profile = 'ssbackup'       # Your AWS CLI profile
region = 'ap-southeast-2'  # AWS region volumes/snapshots are located
 
def bash(command):
	process = subprocess.Popen(command, stdout=subprocess.PIPE)
	return process.communicate()[0].decode('utf-8')
 
def getOurSnapshots():
	"""
		Return a list of snapshot Dicts created with this plugin.
	"""
	return json.loads(bash([
			"aws", "ec2", "describe-snapshots",
			"--filters", "Name=tag-key,Values=Group", "Name=tag-value,Values=ssbackup",
			"--profile", profile,
			"--region", region,
			"--output=json"
		]))['Snapshots']
 
def createSnapshots(volumeIds):
	"""
		Return True if snapshots of the given volumes are created, else False
 
		Keyword arguments:
		volumeIds -- List of EBS volume IDs
	"""
	# Create the snapshots
	snapshots = []
	for volumeId in volumeIds:
		snapshots.append(createSnapshotForVolume(volumeId))
 
	# Add Name and Group tags to the snapshot
	if len(snapshots):
		snapshotIds = []
		date = time.strftime("%Y-%m-%d")
 
		for snapshot in snapshots:
			snapshotIds.append(snapshot['SnapshotId'])
 
		# create-tags returns no output now so just perform the command and
		# return True
		bash([
			"aws", "ec2", "create-tags",
			"--resources", ' '.join(snapshotIds),
			"--tags", "Key=Name,Value='Backup "+date+"'", "Key=Group,Value=ssbackup",
			"--profile", profile,
			"--region", region,
			"--output=json"
		])
 
		return True
 
	return False
 
def createSnapshotForVolume(volumeId):
	"""
		Return a Dict of a created snapshot for the given EBS volume
 
		Keyword arguments:
		volumeId -- An EBS volume ID
	"""
 
	date = time.strftime("%Y-%m-%d")
	message = "Creating snapshot for volume "+volumeId+"..."
	response = json.loads(bash([
		"aws", "ec2", "create-snapshot",
		"--volume-id", volumeId,
		"--description", "Backup "+date,
		"--profile", profile,
		"--region", region,
		"--output=json"
	]))
	message += response['SnapshotId']
	logging.info(message)
 
	return response
 
def deleteOldSnapshots(snapshots, max_age):
	"""
		Delete all listed snapshots older than max_age
	"""
	snapshotIds = []
	date = datetime.datetime.now()
 
	for snapshot in snapshots:
		snapshotDate = dateutil.parser.parse(snapshot['StartTime']).replace(tzinfo=None)
		dateDiff = date - snapshotDate
 
		if dateDiff.days >= max_age:
			message = "Deleting snapshot "+snapshot['SnapshotId']+" ("+str(dateDiff.days)+" days old)..."
			# delete-snapshot no longer returns any output
			bash([
				"aws", "ec2", "delete-snapshot",
				"--snapshot-id", snapshot['SnapshotId'],
				"--profile", profile,
				"--region", region,
				"--output=json"
			])
 
			message += "done"
			logging.info(message)
 
if __name__ == '__main__':
	parser = argparse.ArgumentParser(description='ADD YOUR DESCRIPTION HERE')
	parser.add_argument('-i','--volume-ids', help='EBS volume ID', required=True)
	parser.add_argument('-d','--delete-old', help='Delete old snapshots?', required=False, type=bool, default=True)
	parser.add_argument('-x','--expiry-days', help='Number of days to keep snapshots', required=False, type=int, default=7)
	args = parser.parse_args()
 
	logging.basicConfig(filename='ssbackup.log', level=logging.DEBUG, format='%(asctime)s:  %(message)s', datefmt='%Y-%m-%d %I:%M:%S%p')
 
	# Get all active volumes
	volumeIds = args.volume_ids.split(',')
	# Create the snapshots
	if len(volumeIds):
		snapshots = createSnapshots(volumeIds)
		pass
 
	# Delete snapshots older than expiry-days
	if args.delete_old:
		deleteOldSnapshots(getOurSnapshots(), args.expiry_days)

 

Put it to Work

To run this script automatically each day, add it to your crontab like so:

1
2
# m   h  dom mon dow   command
  5   0    *   *   *   python3 /home/user/ssbackup.py --volume-ids=vol-a1bcd2e3 --expiry-days=7

 

The script will automatically create an ssbackup.log file that sits next to it describing exactly what it did on last run. For example here’s my log when running a backup with expiry-days=0 a couple of times:

1
2
3
4
2015-06-18 01:09:50AM:  Creating snapshot for volume vol-a1bcd2e3...snap-a123b4cd
2015-06-18 01:09:52AM:  Deleting snapshot snap-b123c4de (0 days old)...done
2015-06-18 02:36:28AM:  Creating snapshot for volume vol-a1bcd2e3...snap-c012345d
2015-06-18 02:36:30AM:  Deleting snapshot snap-a123b4cd (0 days old)...done

 

  • William Turner

    When you get a chance, you might want to re-write using the native boto (aws) library.

  • Ali Gouta

    Nice article. I am getting this error when I run your script:

    Traceback (most recent call last):

    File “update-snapshot”, line 91, in

    snapshots = createSnapshots(volumeIds)

    File “update-snapshot”, line 47, in createSnapshots

    “–region”, region

    File “/usr/lib64/python2.7/json/__init__.py”, line 338, in loads

    return _default_decoder.decode(s)

    File “/usr/lib64/python2.7/json/decoder.py”, line 366, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

    File “/usr/lib64/python2.7/json/decoder.py”, line 384, in raw_decode

    raise ValueError(“No JSON object could be decoded”)

    ValueError: No JSON object could be decoded

    I am running Python 2.7.10 ..

    • Omar Tanti

      Nice Article and very helpful. Thanks.

      Regarding your issue it seems that create-tags and delete-snapshot no longer return a reponse. So you have to edit this:

      response = json.loads(bash([
      “aws”, “ec2”, “create-tags”,
      “–resources”, ‘ ‘.join(snapshotIds),
      “–tags”, “Key=Name,Value=’Backup “+date+”‘”, “Key=Group,Value=ssbackup”,
      “–profile”, profile,
      “–region”, region
      ]))

      into this:

      bash([
      “aws”, “ec2”, “create-tags”,
      “–resources”, ‘ ‘.join(snapshotIds),
      “–tags”, “Key=Name,Value=’Backup “+date+”‘”, “Key=Group,Value=ssbackup”,
      “–profile”, profile,
      “–region”, region
      ])

      and also this:

      response = json.loads(bash([
      “aws”, “ec2”, “delete-snapshot”,
      “–snapshot-id”, snapshot[‘SnapshotId’],
      “–profile”, profile,
      “–region”, region
      ]))

      into this:

      bash([
      “aws”, “ec2”, “delete-snapshot”,
      “–snapshot-id”, snapshot[‘SnapshotId’],
      “–profile”, profile,
      “–region”, region
      ])

      • flynsarmy

        Thanks for that. I actually fixed this myself the other day but never updated my article. I’ve done so now.

  • Brian Deitte

    I hope you don’t mind, but I’ve expanded a bit on what you’ve done here for what has worked for me and written about it. And given you credit of course! Thanks for your help, as this was very helpful for me. http://deitte.com/scheduled-ebs-sna/

  • ajay

    Script is nice but It should be modified to give informative description abut the associated volume. With the present script it’s difficult to identify associated volumes in case you have multiple volumes and instances.

  • Vignesh SK

    Thank you for the script, the region code is hard coded and hence I had to change it, would be better if this could take the region code from the aws configuration.

  • sai

    An error occurred (InvalidVolume.NotFound) when calling the CreateSnapshot operation: The volume ‘vol-0a414117686c2da87’ does not exist.
    Traceback (most recent call last):
    File “ssbackup.py”, line 132, in
    snapshots = createSnapshots(volumeIds)
    File “ssbackup.py”, line 46, in createSnapshots
    snapshots.append(createSnapshotForVolume(volumeId))
    File “ssbackup.py”, line 87, in createSnapshotForVolume
    “–output=json”
    File “/usr/lib64/python2.7/json/__init__.py”, line 338, in loads
    return _default_decoder.decode(s)
    File “/usr/lib64/python2.7/json/decoder.py”, line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File “/usr/lib64/python2.7/json/decoder.py”, line 383, in raw_decode
    raise ValueError(“No JSON object could be decoded”)
    ValueError: No JSON object could be decoded

    • flynsarmy

      Sounds like you entered an incorrect volume ID.

  • Mahender Mahi

    An error occurred (InvalidVolume.NotFound) when calling the CreateSnapshot operation: The volume ‘vol-0a414117686c2da87’ does not exist.

    [root@CentOs selinux]# curl “https://s3.amazonaws.com/aws-cli/awscli-bundle.zip” -o “awscli-bundle.zip”
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 7284k 100 7284k 0 0 258k 0 0:00:28 0:00:28 –:–:– 973k
    [root@CentOs selinux]# cd
    [root@CentOs ~]# unzip awscli-bundle.zip
    unzip: cannot find or open awscli-bundle.zip, awscli-bundle.zip.zip or awscli-b undle.zip.ZIP.
    [root@CentOs ~]# curl “https://s3.amazonaws.com/aws-cli/awscli-bundle.zip” -o “a wscli-bundle.zip”
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 7284k 100 7284k 0 0 438k 0 0:00:16 0:00:16 –:–:– 1294k
    [root@CentOs ~]# unzip awscli-bundle.zip
    Archive: awscli-bundle.zip
    inflating: awscli-bundle/install
    inflating: awscli-bundle/packages/awscli-1.11.11.tar.gz
    inflating: awscli-bundle/packages/jmespath-0.9.0.tar.gz
    inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz
    inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz
    inflating: awscli-bundle/packages/s3transfer-0.1.9.tar.gz
    inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz
    inflating: awscli-bundle/packages/futures-3.0.5.tar.gz
    inflating: awscli-bundle/packages/docutils-0.12.tar.gz
    inflating: awscli-bundle/packages/colorama-0.3.7.zip
    inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz
    inflating: awscli-bundle/packages/pyasn1-0.1.9.tar.gz
    inflating: awscli-bundle/packages/virtualenv-13.0.3.tar.gz
    inflating: awscli-bundle/packages/botocore-1.4.68.tar.gz
    inflating: awscli-bundle/packages/python-dateutil-2.5.3.tar.gz
    inflating: awscli-bundle/packages/six-1.10.0.tar.gz
    [root@CentOs ~]# ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
    Running cmd: /usr/bin/python virtualenv.py –python /usr/bin/python /usr/local/aws
    Running cmd: /usr/local/aws/bin/pip install –no-index –find-links file:///root/a wscli-bundle/packages awscli-1.11.11.tar.gz
    You can now run: /usr/local/bin/aws –version
    [root@CentOs ~]# aws help
    [root@CentOs ~]# sudo curl -o /usr/local/bin/ssbackup.py https://raw.githubusercon tent.com/bdeitte/scheduled-snapshots/master/scripts/ssbackup.py
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 4340 100 4340 0 0 2758 0 0:00:01 0:00:01 –:–:– 2757
    [root@CentOs ~]# less /usr/local/bin/ssbackup.py
    root@CentOs ~]# aws configure –profile=ssbackup AWS Access Key ID [****************PT2A]: AKIAJPTLZECDAU3U6Z5A
    AWS Secret Access Key [****************umtl]: 0i+ZJv4BKEDYcXtQJqzQtlrHRBqzDX0GnI5c tSzG
    Default region name [us-west-2]: us-west-2
    Default output format [json]: json
    [root@CentOs ~]# python /usr/local/bin/ssbackup.py –volume-ids=vol-0c956be48ce14e 9c5–expiry-days=7
    Creating snapshots…

    An error occurred (InvalidParameterValue) when calling the CreateSnapshot operatio n: Value (vol-0c956be48ce14e9c5–expiry-days=7) for parameter volumeId is invalid. Expected: ‘vol-…’.
    Traceback (most recent call last):
    File “/usr/local/bin/ssbackup.py”, line 122, in
    snapshots = createSnapshots(volumeIds)
    File “/usr/local/bin/ssbackup.py”, line 42, in createSnapshots
    snapshots.append(createSnapshotForVolume(volumeId))
    File “/usr/local/bin/ssbackup.py”, line 80, in createSnapshotForVolume
    “–profile”, profile
    File “/usr/lib64/python2.7/json/__init__.py”, line 338, in loads
    return _default_decoder.decode(s)
    File “/usr/lib64/python2.7/json/decoder.py”, line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File “/usr/lib64/python2.7/json/decoder.py”, line 383, in raw_decode
    raise ValueError(“No JSON object could be decoded”)
    ValueError: No JSON object could be decoded