Uploading Directly to Amazon S3 with Dropzone and PHP

Posted March 16th, 2018 (Updated 13 Aug 2019) in Javascript, PHP

Dropzone is a really nice lightweight javascript upload library with drag drop support and a high level of customisability. The documentation for direct uploads to Amazon S3 with a PHP backend are entirely lacking though so I’ve whipped up a complete tutorial on how to make it happen. For this tutorial I’ll be using Laravel but the script is very simple and any PHP backend will work.

What You’ll Need

AWS Access Key
AWS Secret Key
AWS Region
AWS Bucket Name

As part of Laravel I’m using dotenv to store these values and retrieving them with getenv() but they can be stored as config variables or any other method you like.

The Frontend

For the most part there isn’t anything particularly complicated about the frontend markup but there are a few things to note:

<!doctype html>
<html>
    <head>
        <meta charset="utf-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <meta name="csrf-token" content="<?= csrf_token() ?>">
 
        <title>Dropzone Uploader Example</title>
        <link rel="stylesheet" href="<?= asset('css/app.css') ?>">
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js"></script>
        <script type="text/javascript">
            // Make AJAX POST requests work with laravel. See https://stackoverflow.com/a/47806189
            $.ajaxSetup({
                beforeSend: function(xhr, type) {
                    if (!type.crossDomain) {
                        xhr.setRequestHeader('X-CSRF-Token', $('meta[name="csrf-token"]').attr('content'));
                    }
                },
            });
 
            jQuery(document).ready(function($) {
                var uploader = new Dropzone('#s3dropzone', {
                    url: $('#s3dropzone').attr('action'),
                    method: "post",
                    autoProcessQueue: true,
                    maxfiles: 5,
                    timeout: null,
                    parallelUploads: 3,
                    maxThumbnailFilesize: 8, // 3MB
                    thumbnailWidth: 150,
                    thumbnailHeight: 150,
                    /**
                     *
                     * @param object   file https://developer.mozilla.org/en-US/docs/Web/API/File
                     * @param function done Use done('my error string') to return an error
                     */
                    accept: function(file, done) {
                        file.postData = [];
                        $.ajax({
                            url: "<?= route('request-s3-file-signature') ?>",
                            data: {
                                filename: file.name
                            },
                            type: 'POST',
                            dataType: 'json',
                            success: function(response) {
                                if ( !response.success )
                                    done(response.message);
 
                                delete response.success;
                                file.custom_status = 'ready';
                                file.postData = response;
                                file.s3 = response.key;
                                $(file.previewTemplate).addClass('uploading');
                                done();
                            },
                            error: function(response) {
                                file.custom_status = 'rejected';
                                if (response.responseText) {
                                    response = JSON.parse(response.responseText);
                                }
                                if (response.message) {
                                    done(response.message);
                                    return;
                                }
                                done('error preparing the upload');
                            }
                        });
                    },
                    /**
                     * Called just before each file is sent.
                     * @param object   file https://developer.mozilla.org/en-US/docs/Web/API/File
                     * @param object   xhr
                     * @param object   formData https://developer.mozilla.org/en-US/docs/Web/API/FormData
                     */
                    sending: function(file, xhr, formData) {
                        $.each(file.postData, function(k, v) {
                            formData.append(k, v);
                        });
                        formData.append('Content-type', 'application/octet-stream');
                        // formData.append('Content-length', '');
                        formData.append('acl', 'public-read');
                        formData.append('success_action_status', '200');
                    }
                });
            });
        </script>
    </head>
    <body>
        <!-- URL taken from https://stackoverflow.com/a/14061033 -->
        <form action="https://s3-<?= $AWS_REGION ?>.amazonaws.com/<?= $AWS_S3_BUCKET ?>/" id="s3dropzone">
            <div class="fallback">
                <input name="file" type="file" multiple />
            </div>
        </form>
    </body>
</html>

The important parts here are:

Avoiding Timeouts

1	timeout: null,

Without this Dropzone option our uploads are limited to 30 seconds and then time out.

Signing upload files

This is where the real work is done. Each time a file is added to dropzone, we need to grab a signature and policy document from our backend and pass that along with the file to S3.

In Dropzone’s accept callback we perform an AJAX request passing the filename to upload. This returns a JSON response containing our AWS Access Key, filepath in S3 to upload to, policy document and encrypted signature document. All of this information is attached to the File object for later use in Dropzone’s sending callback where it’s attached to the form data to be sent to S3.

This solution will work with a backend of any language at all provided it returns in the format

{
    success: 1|0,
    AWSAccessKeyId: string,
    key: string,
    policy: string,
    signature: string
}

Our Upload URL

The documentation as of the time of writing uses the URL structure http://<bucket>.s3.amazonaws.com/ however there are a couple of issues with this.

It’s not HTTPS
It doesn’t work with buckets with dots in the name

Instead thanks to user Nate on StackOverflow we instead use this URL https://s3-<region>.amazonaws.com/<bucket>/.

Creating Signature and Policy Documents

Here is the backend PHP for creating our signature and policy documents.

<?php
Route::post('/ajax/request-s3-file-signature', function() {
	// Name of the file uploaded
	$filename = Request::post('filename', '');
 
	// Validate file name
    if ( !preg_match("/^[ a-zA-Z0-9\._-]+$/", $filename) )
        return response()->json([
            'success' => 0,
            'message' => "Filename must contain only letters, numbers, dots, underscores and dashes.",
        ]);
 
    // Path in our S3 bucket to upload to
    $filepath = 'tests/';
 
    // Set up our policy generator being careful to add conditions that match
    // what our HTML form will be submitting to S3
    $policy = (new App\AwsPostPolicy(getenv('AWS_ACCESS'), getenv('AWS_SECRET')))
    	->addCondition('', 'acl', 'public-read')
    	->addCondition('starts-with', '$key', $filepath)
    	->addCondition('', 'bucket', getenv('AWS_S3_BUCKET'))
    	->addCondition('', 'success_action_status', 200)
    	->addCondition('', 'content-type', "application/octet-stream");
 
    return response()->json([
        'success' => 1,
        'AWSAccessKeyId' => getenv('AWS_ACCESS'),
        'key' => $filepath . $filename,
        'policy' => $policy->getPolicy(),
        'signature' => $policy->getSignedPolicy(),
    ]);
})->name('request-s3-file-signature');
 
 
 
<?php
 
namespace App;
 
/**
 * @category    Amazon
 * @package     AWS
 * @copyright   Copyright 2009 Amazon Technologies, Inc.
 * @link        http://aws.amazon.com
 * @license     http://aws.amazon.com/apache2.0  Apache License, Version 2.0
 * @author      Michael@AWS
 */
/**
 * Create an Amazon S3 POST policy.
 *
 * @package AWS
 */
class AwsPostPolicy {
    /**
     * AWS Secret Access Key.
     *
     * @var string
     */
    protected $_awsSecretAccessKey = '';
    /**
     * AWS Access Key ID.
     *
     * @var string
     */
    protected $_awsAccessKeyId = '';
    /**
     * Amazon S3 bucket.
     *
     * @var string
     */
    protected $_bucket = '';
    /**
     * Array of conditions.
     *
     * @var array
     */
    protected $_conditions = array();
    /**
     * Duration in seconds that the policy is valid.  Default is 24 hours.
     *
     * @var int
     */
    protected $_duration = 86400;
    /**
     * Maintain a cache of un-encoded policy so that multiple requests to get
     * the policy will not incur unnecessary computation.
     *
     * @var string
     */
    protected $_policyCache = '';
    /**
     * Construct a new POST policy object.
     *
     * @param string $awsSecretAccessKey
     *      Your AWS Secret Access Key.
     *
     * @param string $bucket
     *      Amazon S3 bucket name.
     *
     * @param int $duration
     *      The number of seconds for which the policy is valid.
     */
    public function __construct($awsAccessKeyId, $awsSecretAccessKey, $bucket = '', $duration = 86400) {
        $this->_awsAccessKeyId = $awsAccessKeyId;
        $this->_awsSecretAccessKey = $awsSecretAccessKey;
        $this->_bucket = $bucket;
        $this->_duration = $duration;
        $this->_expireCache();
    }
    /**
     * Add a policy condition.
     *
     * @param string $condition
     *      Condition type.  Possible values are 'eq', 'starts-with',
     *      and 'content-length-range'.  Pass null or an empty string to set a
     *      condition that uses the {"field": "match"} syntax.  Passing anything
     *      in the $condition parameter will cause the condition to use the
     *      [condition, field, match] syntax.
     *
     * @param string $field
     *      The field name.
     *
     * @param string $match
     *      String to match.
     *
     * @return Aws_S3_PostPolicy
     *      Returns a reference to the object.
     *
     * <code>
     * $policy->addCondition('', 'acl', 'public-read');
     * // Produces: {"acl": "public-read"}
     *
     * $policy->addCondition('eq', 'acl', 'public-read');
     * // Produces: ["eq", "$acl", "public-read"]
     *
     * $policy->addCondition('starts-with', '$key', 'user/betty/');
     * // Produces: ["starts-with", "$key", "user/betty/"]
     *
     * $policy->addCondition('content-length-range', 1048579, 10485760);
     * // Produces: ["content-length-range", 1048579, 10485760]
     * </code>
     */
    public function addCondition($condition = 'eq', $field, $match = '') {
        $this->_conditions[] = array($condition, $field, $match);
        $this->_expireCache();
        return $this;
    }
    /**
     * Get the AWS Access Key ID associated with this POST policy.
     *
     * @return string
     *      Returns the AWS Access Key ID.
     *
     */
    public function getAwsAccessKeyId() {
        return $this->_awsAccessKeyId;
    }
    /**
     * Get the bucket associated with the policy.
     *
     * @return string
     *      Returns the Amazon S3 bucket name.
     */
    public function getBucket() {
        return $this->_bucket;
    }
    /**
     * Get a spefic permission based on the field name.
     *
     * @param string $field
     *      Field name to retrieve.
     *
     * @param bool $fullCondition
     *      Whether or not to return the entire condition as an array or just
     *      retrieve the match value of the condition.  By default, this is set
     *      to false so that it will only retrieve the match value of the
     *      condition.
     *
     * @return array|string
     *      If $fullCondition is set to true, then this method will return an
     *      indexed array of data about the condition:
     *      array(condition, field, match).
     *      Returns an empty array if the field is not found.
     *
     *      If $fullCondition is set to false, then this method will only return the
     *      match field of the condition.
     */
    public function getCondition($field, $fullCondition = false) {
        if (count($this->_conditions)) {
            foreach ($this->_conditions as $condition) {
                if ($condition[1] === $field) {
                    return ($fullCondition) ? $condition : $condition[2];
                }
            }
        }
        return ($fullCondition) ? array() : null;
    }
    /**
     * Get all of the conditions associated with the policy.
     *
     * @return array
     *      Returns an indexed array of conditions.
     */
    public function getConditions() {
        return $this->_conditions;
    }
    /**
     * Retrieve the POST policy.
     *
     * @param bool $encode
     *      Set to true to retrieve the encoded policy.
     *
     * @return string
     *      Returns an un-signed POST policy.
     */
    public function getPolicy($encode = false) {
        if (!$this->_policyCache) {
            $policy = sprintf('{ "expiration": "%s"', gmdate('Y-n-d\TH:i:s.000\Z', time() + $this->_duration));
            if (count($this->_conditions)) {
                $policy .= ', "conditions": [';
                $first = true;
                foreach ($this->_conditions as $condition) {
                    if (!$first) {
                        $policy .= ', ';
                    }
                    if ($condition[0]) {
                        $policy .= sprintf('["%s", "%s", "%s"]', $condition[0], $condition[1], $condition[2]);
                    } else {
                        $policy .= sprintf('{"%s": "%s"}', $condition[1], $condition[2]);
                    }
                    $first = false;
                }
                $policy .= ']';
            }
            $policy .= '}';
            $this->_policyCache = $policy;
        } else {
            $policy = $this->_policyCache;
        }
        if (!$encode) {
            return base64_encode($policy);
        } else {
            return base64_encode(utf8_encode(preg_replace('/\s\s+|\\f|\\n|\\r|\\t|\\v/', '', $policy)));
        }
    }
    /**
     * Get a signed POST policy.
     *
     * @return string
     *      Returns a signed POST policy.
     */
    public function getSignedPolicy() {
        return base64_encode(hash_hmac('sha1', $this->getPolicy(), $this->_awsSecretAccessKey, true));
    }
    /**
     * Expire the policy cache.
     *
     * @return Aws_S3_Policy
     *      Returns a reference to the object.
     */
    protected function _expireCache() {
        $this->_policyCache = '';
        return $this;
    }
    /**
     * Reset the policy by clearing the conditions, removing the bucket name,
     * and making the duration 86400 seconds (24 hours).
     *
     * @return Aws_S3_PostPolicy
     *      Returns a reference to the object.
     */
    public function reset() {
        $this->_conditions = array();
        $this->_bucket = '';
        $this->_duration = 0;
        $this->_expireCache();
        return $this;
    }
    /**
     * Set the Amazon S3 bucket.
     *
     * @param string $bucket
     *      Amazon S3 bucket name.
     *
     * @return Aws_S3_PostPolicy
     *      Returns a reference to the object.
     */
    public function setBucket($bucket) {
        $this->_bucket = $bucket;
        return $this;
    }
    /**
     * Set the number of seconds the policy is valid.
     *
     * @param int $seconds
     *      The number of seconds the policy should be valid.
     *
     * @return Aws_S3_PostPolicy
     *      Returns a reference to the object.
     */
    public function setDuration($seconds) {
        $this->_duration = $seconds;
        $this->_expireCache();
        return $this;
    }
}

The AwsPostPolicy class handles all the heavy lifting for us – we just need to plug in our Access and Secret keys as well as adding a condition to match each field our HTML Form is passing to S3 during the upload.