Posted in PHP

In a recent project I had to download and process a bunch of CSVs. Initially I had an extremely ugly exec() call to linux’s wget command for reasons I won’t go into but obviously a better, PHP-based solution was required. I had previous experience with Guzzle and its pooled requests so it was the obvious place to go.

Below is the script I ended up with. It takes an array of files and downloads them all to a __FILE__.’/downloads/’ directory. Not the cleanest thing in the world but it did the trick and you should be able to adapt it as you need.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
<?php
 
ini_set('display_errors', true);
require __DIR__.'/vendor/autoload.php';
 
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Response;
 
$files = array(
	'http://mysite.com/1.csv',
	'http://mysite.com/2.csv',
	'http://mysite.com/3.csv',
	'http://mysite.com/4.csv',
);
 
// Track redirects so our Pool's fulfilled closure knows which URL the current
// download is for.
$client = new Client([
	'allow_redirects' => ['track_redirects' => true]
]);
 
$requests = function($total) use ($client, $files, $import_path) {
    foreach ( $files as $file )
    {
        yield function($poolOpts) use ($client, $file) {
            $reqOpts = array_merge($poolOpts, [
            	// Sink option specifies the download path for this file
                'sink' => __DIR__.'/downloads/' . basename($file)
            ]);
 
            return $client->getAsync($file, $reqOpts);
        };
    }
};
 
$pool = new Pool($client, $requests(100), [
    'concurrency' => 3,
    'fulfilled' => function(Response $response, $index) use ($files) {
        // Grab the URLs this file redirected through to download in chronological order.
        $urls = $response->getHeader(\GuzzleHttp\RedirectMiddleware::HISTORY_HEADER);
 
        echo "Downloaded ", end($urls), "<br/>\n";
    },
    'rejected' => function(Exception $reason, $index) use (&$import_errors) {
        $url = (string)$reason->getRequest()->getUri();
 
        echo "Failed to download ", $url, ": ", $reason->getMessage(), "<br/>\n";
    },
]);
 
$pool->promise()->wait();
 
echo "Finished downloading.<br/>";

Read More »

Posted (Updated ) in Database, Linux, PHP

After suffering some pretty bad issues with MAMP, I decided to set everything up with homebrew instead. The result was surprisingly a much faster and (in my opinion) easier to configure setup.

As a tl;dr, we’ll be setting up Homebrew MySQL and PHP and using OSX’s built in Apache.

In this tutorial I’m using the subl command which will open a file for editing in Sublime Text. If you don’t use Sublime Text, replace subl with nano or vi or any other app you use to edit text/config files.

 

Homebrew Setup

Homebrew is a package manager for OSX. It makes installation of a wide variety of useful apps super easy.

Installation instructions are on the homebrew homepage but you can also just run the following:

1
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

 

MySQL

I lied! We’re installing MariaDB instead! At the time of writing MySQL version 8.0.11 has just changed its default authentication method to caching_sha2_password which isn’t supported in PHP. It’s a huge hassle so we’ll just use the drop-in replacement MariaDB instead.

Install and configure MariaDB.

1
2
3
4
# Install MariaDB
brew install mariadb
# Open my.cnf config file for editing
subl /usr/local/etc/my.cnf

Add the following to the end of the file to add support for large imports:

1
2
max_allowed_packet = 2G
innodb_file_per_table = 1

Make MySQL start when you log in:

1
brew services start mariadb

The default installation comes with a passwordless root user. So secure it with:

1
mysql_secure_installation

 

SSL

Like all developers I like working on a custom subdomain – in this case localhost.com. We need to create a self-signed wildcard SSL certificate and get Chrome accepting it.

Create a folder /Users/your_username/Sites/certs and inside it run the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Generate a temporary OpenSSL config file
cat > openssl.cnf <<-EOF
  [req]
  distinguished_name = req_distinguished_name
  x509_extensions = v3_req
  prompt = no
  [req_distinguished_name]
  CN = *.localhost.com
  [v3_req]
  keyUsage = keyEncipherment, dataEncipherment
  extendedKeyUsage = serverAuth
  subjectAltName = @alt_names
  [alt_names]
  DNS.1 = *.localhost.com
  DNS.2 = localhost.com
EOF
 
# Generate the certificates
openssl req \
  -new \
  -newkey rsa:2048 \
  -sha1 \
  -days 3650 \
  -nodes \
  -x509 \
  -keyout ssl.key \
  -out server.crt \
  -config openssl.cnf
 
# Delete the temporary config file
rm openssl.cnf

This should have created two files – server.crt and server.key which will be used in the apache config below to get HTTPS up and running.

But first, because this certificate is self-signed, it’ll result in a This site’s security certificate is not trusted! error in Chrome. That can be fixed through adding the cert to OSX’s keychain app.

  • 1
    
    open /Applications/Utilities/Keychain\ Access.app /Users/your_username/Sites/certs/server.crt
  • Set the Keychain dropdown to System and click Add
  • Now in the Certificates section of Keychain find your newly added cert, double click it, expand the Trust section and set everything to Always Trust
  • These changes will only take effect after a browser restart.

 

Apache and PHP

OSX 10.13 High Sierra comes (at the time of writing) with Apache 2.4.33.

To configure apache (with SSL):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Install PHP 7.1
brew install php@7.1
brew link --overwrite --force php@7.1
# Open httpd.conf for editing
subl /etc/apache2/httpd.conf
 
# Enable the PHP and SSL modules by removing the # at the start of the line
LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so
LoadModule ssl_module libexec/apache2/mod_ssl.so
LoadModule php7_module /usr/local/opt/php@7.1/lib/httpd/modules/libphp7.so
# A few extras I like to have enabled
LoadModule deflate_module libexec/apache2/mod_deflate.so
LoadModule expires_module libexec/apache2/mod_expires.so
LoadModule headers_module libexec/apache2/mod_headers.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
 
# Point the document root to a htdocs folder in your home directory and enable .htaccess
# I've removed all the comments for succinctness but feel free to leave them in
DocumentRoot "/Users/your_username/htdocs"
<Directory "/Users/your_username/htdocs">
    Options FollowSymLinks Multiviews
    MultiviewsMatch Any
 
    AllowOverride All
 
    Require all granted
</Directory>
 
# Add PHP to your default file list
<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>
 
# And make it work
<FilesMatch \.php$>
    SetHandler application/x-httpd-php
</FilesMatch>
 
# As with content, we want to load all site definitions from a Sites folder in our 
# home directory. At the bottom replace the following:
# Include /private/etc/apache2/other/*.conf
IncludeOptional /Users/your_username/Sites/*.conf
 
# The error and custom logs too
CustomLog "/Users/your_username/Sites/logs/apache2/access_log" common
ErrorLog "/Users/your_username/Sites/logs/apache2/error_log"
 
# Uncomment to load the SSL config
Include /private/etc/apache2/extra/httpd-ssl.conf

Now configure the default SSL options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Open httpd.conf for editing
subl /etc/apache2/extra/httpd-ssl.conf
 
# Point to our same document root as before
DocumentRoot "/Users/your_username/htdocs"
 
# Update log file locations
ErrorLog "/Users/your_username/Sites/logs/apache2/error_log"
TransferLog "/Users/your_username/Sites/logs/apache2/access_log"
CustomLog "/Users/your_username/Sites/logs/apache2/ssl_request_log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
 
# Point to the certs we created
SSLCertificateFile "/Users/your_username/Sites/certs/server.crt"
SSLCertificateKeyFile "/Users/your_username/Sites/certs/server.key"

Since this is a development machine, you’ll probably also want to enable the ever popular xdebug which luckily for us comes pre-compiled with OSX. What OSX doesn’t come with, however, is a default php.ini though it does have a sample file. We can use that:

1
sudo cp /etc/php.ini.default /etc/php.ini

Then simply add extension=xdebug.so below all the extension= lines in your new /etc/php.ini file.

VirtualHosts

I like to split virtualhosts up into one for each site and store them all in /Users/your_username/Sites/ folder.

Create a file /Users/your_username/Sites/mysite.localhost.com.conf and add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName mysite.localhost.com
  ServerAlias mysite.localhost.com
  DocumentRoot /Users/your_username/htdocs/mysite.com
 
  ErrorLog /Users/your_username/Sites/logs/mysite.com.error.log
  LogLevel warn
  CustomLog /Users/your_username/Sites/logs/mysite.com.access.log varnishcombined
 
  <Directory /Users/your_username/htdocs/mysite.com/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>
 
<IfModule ssl_module>
  <VirtualHost *:443>
    ServerAdmin webmaster@localhost
    ServerName mysite.localhost.com
    ServerAlias mysite.localhost.com
    DocumentRoot /Users/your_username/htdocs/mysite.com
 
    ErrorLog /Users/your_username/Sites/logs/mysite.com.error.log
    LogLevel warn
    CustomLog /Users/your_username/Sites/logs/mysite.com.access.log varnishcombined
 
    <Directory /Users/flynsarmy/htdocs/work/qpsmedia/qpsstats/>
      Options Indexes FollowSymLinks
      AllowOverride All
      Require all granted
    </Directory>
 
    SSLEngine on
    SSLCertificateFile    /Users/your_username/Sites/certs/server.crt
    SSLCertificateKeyFile /Users/your_username/Sites/certs/server.key
 
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
      SSLOptions +StdEnvVars
    </FilesMatch>
  </VirtualHost>
</IfModule>

 

Finally, restart apache and you should be good to go!

1
sudo apachectl restart

 

Resources

Read More »

Posted (Updated ) in Javascript, PHP

Dropzone is a really nice lightweight javascript upload library with drag drop support and a high level of customisability. The documentation for direct uploads to Amazon S3 with a PHP backend are entirely lacking though so I’ve whipped up a complete tutorial on how to make it happen. For this tutorial I’ll be using Laravel but the script is very simple and any PHP backend will work.

 

What You’ll Need

  • AWS Access Key
  • AWS Secret Key
  • AWS Region
  • AWS Bucket Name

As part of Laravel I’m using dotenv to store these values and retrieving them with getenv() but they can be stored as config variables or any other method you like.

Read More »

Posted in PHP

Say you’re pooling requests in Guzzle and want the responses to have access to data such as the request URL. Guzzle doesn’t currently allow for this but that’s all fixed with the help of a simple middleware class.

 

Usage

When requesting a URL with GuzzleHttp\Client, pass a RESPONSE_META array in your requests second argument like so:

1
2
3
4
5
6
$client->get($url, [
    'RESPONSE_META' => [
        'url' => $url,
        'some_data' => 'foo',
    ],
]);

The data is attached as a header to the response. Access it like so: (Remembering to prepend X-GUZZLE-META- to your key)

1
$some_data = $response->getHeaderLine('X-GUZZLE-META-some_data');

 

Complete Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$names = ['Alpha', 'Beta', 'Charlie', 'Delta'];
$stack = GuzzleHttp\HandlerStack::create();
$stack->push(GuzzleResponseMetaMiddleware::middleware());$client = new GuzzleHttp\Client([
    'handler' => $stack]);
$requests = function() use ($client, $names) {
    foreach ( $names as $name )
    {
        $url = "Http://mydomain.com/?name=".$name;
 
        yield function() use ($client, $url, $name) {
            return $client->getAsync($url, [
                'RESPONSE_META' => [                    'url' => $url,                    'name' => $name,                ],            ]);
        };
    }
};
$pool = new GuzzleHttp\Pool($client, $requests(), [
    'concurrency' => 5,
    'fulfilled' => function (Psr\Http\Message\ResponseInterface $response, $index) {
        $url = $response->getHeaderLine('X-GUZZLE-META-url');        $name = $response->getHeaderLine('X-GUZZLE-META-name'); 
        // do something with this info
    },
]);
 
// Initiate the transfers and create a promise
$promise = $pool->promise();
 
// Force the pool of requests to complete.
$promise->wait();

Read More »

Posted in Linux, PHP

Isn’t it annoying when you want to connect to your home network while out and about but don’t know what your IP is? Sick of dynamic DNS sites with arbitrary restrictions on their free tiers? Well look no further! This tutorial demonstrates how to point your home IP to a subdomain of your website using a simple PHP script.

 

The Concept

  • Set up a Route 53 subdomain for pointing to your home
  • A device in your home uses a scheduled task to ping a URL on your website
  • That URL grabs the IP hitting it and points your subdomain to the IP.

Read More »

Posted in PHP

WordPress offers no way out of the box to grab your sidebar widget settings and use them somewhere else in your site. Here’s a function to allow you to do it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
/**
 * Find a given widget in a given sidebar and return its settings.
 * 
 * Example usage:
 * $options = [];
 * try {
 *    $options = get_sidebar_widget_options('sidebar-1', 'recent-comments');
 * } catch (Exception $e) {}
 *
 * @param $sidebar_id    The ID of the sidebar. Defined in your register_sidebar() call
 * @param $widget_type   Widget type specified in register_sidebar()
 * @return array         Saved options
 * @throws Exception     "Widget not found in sidebar" or "Widget has no saved options"
 */
function get_sidebar_widget_options($sidebar_id, $widget_type)
{
    // Grab the list of sidebars and their widgets
    $sidebars = wp_get_sidebars_widgets();
    // Just grab the widgets for our sidebar
    $widgets = $sidebars[$sidebar_id];
 
    // Get the ID of our widget in this sidebar
    $widget_id = 0;
    foreach ( $widgets as $widget_details )
    {
        // $widget_details is of the format $widget_type-$id - we just want the id part
        if ( preg_match("/^{$widget_type}\-(?P<id>\d+)$/", $widget_details, $matches) )
        {
            $widget_id = $matches['id'];
            break;
        }
    }
 
    // If we didn't find the given widget in the given sidebar, throw an error
    if ( !$widget_id )
        throw new Exception("Widget not found in sidebar");
 
    // Grab the options of each instance of our $widget_type from the DB
    $options = get_option('widget_' . $widget_type);
 
    // Ensure there are settings to return
    if ( !isset($options[$widget_id]) )
        throw new Exception("Widget has no saved options");
 
    // Grab the settings
    $widget_options = $options[$widget_id];
 
    return $widget_options;
}

Drop the function in your functions.php. Note that it will find the first occurrence of your given widget in the given sidebar and return its settings.

Read More »

Posted (Updated ) in PHP

Welcome back to my simple to-do application tutorial for Laravel 5. This tutorial is relatively long so I’ve broken it up into multiple posts.

  1. Part 1 – Installation, Database and Routes
  2. Part 2 – Listing Projects and Tasks
  3. Part 3 – Create/Edit/Delete
  4. Part 4 – Validation

The source for each part can also be found on GitHub.

So far we’ve learned how to install and set up Laravel, set up some project and task nested resources and display them to the user. Create, edit and delete functionality has also been implemented. In this chapter we’ll finish things off by adding form validation.

Read More »

Posted (Updated ) in PHP

Welcome back to my simple to-do application tutorial for Laravel 5. This tutorial is relatively long so I’ve broken it up into multiple posts.

  1. Part 1 – Installation, Database and Routes
  2. Part 2 – Listing Projects and Tasks
  3. Part 3 – Create/Edit/Delete
  4. Part 4 – Validation

The source for each part can also be found on GitHub.

So far we’ve learned how to install and set up Laravel, set up some project and task resources and displayed them to the user. In this chapter we’ll learn how to set up create, edit and delete pages/actions.

Read More »

Posted (Updated ) in PHP

Welcome back to my simple to-do application tutorial for Laravel 5. This tutorial is relatively long so I’ve broken it up into multiple posts.

  1. Part 1 – Installation, Database and Routes
  2. Part 2 – Listing Projects and Tasks
  3. Part 3 – Create/Edit/Delete
  4. Part 4 – Validation

The source for each part can also be found on GitHub.

So far we have a working database complete with seed data and a bunch of routes for displaying, editing and deleting our projects and tasks. In this second chapter I’ll cover controllers, models (with relationships), views (including the blade templating language and layouts) and route model binding.

Read More »

Posted (Updated ) in PHP

With the release of Laravel 5, there have been a bunch of backwards incompatible changes, new features added as well as the usual influx of new users so I thought I’d take the time to redo my basic to-do application with Laravel 5. The app covers a wide range of concepts, links to relevant learning material where possible and should make for a great introduction to the framework.

This tutorial is relatively long so I’ve broken it up into multiple posts.

  1. Part 1 – Installation, Database and Routes
  2. Part 2 – Listing Projects and Tasks
  3. Part 3 – Create/Edit/Delete
  4. Part 4 – Validation

The source for each part can also be found on GitHub.

Today will cover installation, configuration, artisan, migration, seeding and routes.

Read More »