Posted (Updated ) in Database, Linux, PHP

After suffering some pretty bad issues with MAMP, I decided to set everything up with homebrew instead. The result was surprisingly a much faster and (in my opinion) easier to configure setup.

As a tl;dr, we’ll be setting up Homebrew MySQL and PHP and using OSX’s built in Apache.

In this tutorial I’m using the subl command which will open a file for editing in Sublime Text. If you don’t use Sublime Text, replace subl with nano or vi or any other app you use to edit text/config files.

 

Homebrew Setup

Homebrew is a package manager for OSX. It makes installation of a wide variety of useful apps super easy.

Installation instructions are on the homebrew homepage but you can also just run the following:

1
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

 

MySQL

I lied! We’re installing MariaDB instead! At the time of writing MySQL version 8.0.11 has just changed its default authentication method to caching_sha2_password which isn’t supported in PHP. It’s a huge hassle so we’ll just use the drop-in replacement MariaDB instead.

Install and configure MariaDB.

1
2
3
4
# Install MariaDB
brew install mariadb
# Open my.cnf config file for editing
subl /usr/local/etc/my.cnf

Add the following to the end of the file to add support for large imports:

1
2
max_allowed_packet = 2G
innodb_file_per_table = 1

Make MySQL start when you log in:

1
brew services start mariadb

The default installation comes with a passwordless root user. So secure it with:

1
mysql_secure_installation

 

SSL

Like all developers I like working on a custom subdomain – in this case localhost.com. We need to create a self-signed wildcard SSL certificate and get Chrome accepting it.

Create a folder /Users/your_username/Sites/certs and inside it run the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Generate a temporary OpenSSL config file
cat > openssl.cnf <<-EOF
  [req]
  distinguished_name = req_distinguished_name
  x509_extensions = v3_req
  prompt = no
  [req_distinguished_name]
  CN = *.localhost.com
  [v3_req]
  keyUsage = keyEncipherment, dataEncipherment
  extendedKeyUsage = serverAuth
  subjectAltName = @alt_names
  [alt_names]
  DNS.1 = *.localhost.com
  DNS.2 = localhost.com
EOF
 
# Generate the certificates
openssl req \
  -new \
  -newkey rsa:2048 \
  -sha1 \
  -days 3650 \
  -nodes \
  -x509 \
  -keyout ssl.key \
  -out server.crt \
  -config openssl.cnf
 
# Delete the temporary config file
rm openssl.cnf

This should have created two files – server.crt and server.key which will be used in the apache config below to get HTTPS up and running.

But first, because this certificate is self-signed, it’ll result in a This site’s security certificate is not trusted! error in Chrome. That can be fixed through adding the cert to OSX’s keychain app.

  • 1
    
    open /Applications/Utilities/Keychain\ Access.app /Users/your_username/Sites/certs/server.crt
  • Set the Keychain dropdown to System and click Add
  • Now in the Certificates section of Keychain find your newly added cert, double click it, expand the Trust section and set everything to Always Trust
  • These changes will only take effect after a browser restart.

 

Apache and PHP

OSX 10.13 High Sierra comes (at the time of writing) with Apache 2.4.33.

To configure apache (with SSL):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Install PHP 7.1
brew install php@7.1
brew link --overwrite --force php@7.1
# Open httpd.conf for editing
subl /etc/apache2/httpd.conf
 
# Enable the PHP and SSL modules by removing the # at the start of the line
LoadModule socache_shmcb_module libexec/apache2/mod_socache_shmcb.so
LoadModule ssl_module libexec/apache2/mod_ssl.so
LoadModule php7_module /usr/local/opt/php@7.1/lib/httpd/modules/libphp7.so
# A few extras I like to have enabled
LoadModule deflate_module libexec/apache2/mod_deflate.so
LoadModule expires_module libexec/apache2/mod_expires.so
LoadModule headers_module libexec/apache2/mod_headers.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so
 
# Point the document root to a htdocs folder in your home directory and enable .htaccess
# I've removed all the comments for succinctness but feel free to leave them in
DocumentRoot "/Users/your_username/htdocs"
<Directory "/Users/your_username/htdocs">
    Options FollowSymLinks Multiviews
    MultiviewsMatch Any
 
    AllowOverride All
 
    Require all granted
</Directory>
 
# Add PHP to your default file list
<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>
 
# And make it work
<FilesMatch \.php$>
    SetHandler application/x-httpd-php
</FilesMatch>
 
# As with content, we want to load all site definitions from a Sites folder in our 
# home directory. At the bottom replace the following:
# Include /private/etc/apache2/other/*.conf
IncludeOptional /Users/your_username/Sites/*.conf
 
# The error and custom logs too
CustomLog "/Users/your_username/Sites/logs/apache2/access_log" common
ErrorLog "/Users/your_username/Sites/logs/apache2/error_log"
 
# Uncomment to load the SSL config
Include /private/etc/apache2/extra/httpd-ssl.conf

Now configure the default SSL options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Open httpd.conf for editing
subl /etc/apache2/extra/httpd-ssl.conf
 
# Point to our same document root as before
DocumentRoot "/Users/your_username/htdocs"
 
# Update log file locations
ErrorLog "/Users/your_username/Sites/logs/apache2/error_log"
TransferLog "/Users/your_username/Sites/logs/apache2/access_log"
CustomLog "/Users/your_username/Sites/logs/apache2/ssl_request_log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
 
# Point to the certs we created
SSLCertificateFile "/Users/your_username/Sites/certs/server.crt"
SSLCertificateKeyFile "/Users/your_username/Sites/certs/server.key"

Since this is a development machine, you’ll probably also want to enable the ever popular xdebug which luckily for us comes pre-compiled with OSX. What OSX doesn’t come with, however, is a default php.ini though it does have a sample file. We can use that:

1
sudo cp /etc/php.ini.default /etc/php.ini

Then simply add extension=xdebug.so below all the extension= lines in your new /etc/php.ini file.

VirtualHosts

I like to split virtualhosts up into one for each site and store them all in /Users/your_username/Sites/ folder.

Create a file /Users/your_username/Sites/mysite.localhost.com.conf and add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ServerName mysite.localhost.com
  ServerAlias mysite.localhost.com
  DocumentRoot /Users/your_username/htdocs/mysite.com
 
  ErrorLog /Users/your_username/Sites/logs/mysite.com.error.log
  LogLevel warn
  CustomLog /Users/your_username/Sites/logs/mysite.com.access.log varnishcombined
 
  <Directory /Users/your_username/htdocs/mysite.com/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>
 
<IfModule ssl_module>
  <VirtualHost *:443>
    ServerAdmin webmaster@localhost
    ServerName mysite.localhost.com
    ServerAlias mysite.localhost.com
    DocumentRoot /Users/your_username/htdocs/mysite.com
 
    ErrorLog /Users/your_username/Sites/logs/mysite.com.error.log
    LogLevel warn
    CustomLog /Users/your_username/Sites/logs/mysite.com.access.log varnishcombined
 
    <Directory /Users/flynsarmy/htdocs/work/qpsmedia/qpsstats/>
      Options Indexes FollowSymLinks
      AllowOverride All
      Require all granted
    </Directory>
 
    SSLEngine on
    SSLCertificateFile    /Users/your_username/Sites/certs/server.crt
    SSLCertificateKeyFile /Users/your_username/Sites/certs/server.key
 
    <FilesMatch "\.(cgi|shtml|phtml|php)$">
      SSLOptions +StdEnvVars
    </FilesMatch>
  </VirtualHost>
</IfModule>

 

Finally, restart apache and you should be good to go!

1
sudo apachectl restart

 

Resources

Read More »

Posted in Linux, Uncategorized

I have a bunch of sites in /var/www and need individual user logins with access to their respective sites. In this tutorial I’ll go over how to create a user, chroot jail them and allow access to specific folders (in our case web directories).

For reference I’m using a standard LAMP server on Ubuntu:

1
2
sudo apt-get install -y tasksel
sudo tasksel install lamp-server

but this tutorial will work for any web server configuration.

 

1. Create User, Assign Web Group

1
2
3
4
5
6
7
# Create the user setting group to www-data
sudo useradd -Ng www-data myuser
sudo passwd myuser
 
# Restrict login to SFTP only
sudo groupadd sftp-only
sudo usermod myuser -G sftp-only

 

Create their web directory and provide access

With the new user created, make a directory matching their website’s name and mount the real website folder to it:

1
2
3
4
5
6
7
8
9
# Create chroot directory and set permissions
mkdir -p /home/myuser/mysite.com/html
chmod 755 /home/myuser/mysite.com/html
 
# Mount the destination directory at the directory we just created
mount --bind /var/www/mysite.com/html /home/myuser/mysite.com/html
 
# Add the above command to /etc/rc.local to mount it on boot
nano /etc/rc.local

 

Restrict the user to SFTP Only

We only want to allow SFTP access for this user. First open /etc/passwd and make sure the end of the line has /bin/false like so:

1
2
tail -n1 /etc/passwd
# myuser:x:1001:33::/home/myuser:/bin/false

Now edit /etc/sshd/sshd_config to allow only SFTP myuser:

1
2
3
4
5
Match User myuser
  ChrootDirectory /home/myuser
  ForceCommand internal-sftp
  AllowTcpForwarding no
  X11Forwarding no

Restart the SSHD service:

1
sudo service sshd restart

Now when you try to SSH in with this user you’ll get the error:

This service allows sftp connections only.

 

That’s it! They should now be able to SFTP in and will only have a mysite.com directory with access to their web files.

 

Further Reading

mihai.ile’s post on Stack Overflow – How can I chroot sftp-only SSH users into their homes?

Read More »

Posted in Linux

I’ve been trying to clone a private git repository from BitBucket and getting the response:

$ git clone git@bitbucket.org:my/repo.git
Cloning into ‘repo’…
Permission denied (publickey).
fatal: Could not read from remote repository.

There are two things that need to be done to fix this.

 

Add your SSH Key to BitBucket

Firstly, make sure your git server has your SSH key. I’m using BitBucket so as per their documentation:

1
2
ssh-keygen
cat ~/.ssh/id_rsa.pub

Copy and add your key to Settings – SSH Keys area in BitBucket.

 

Add your Key to the SSH Agent

If this still isn’t enough to fix the above error you may need to add your new key to your machines SSH agent.

1
2
3
4
5
# Make sure SSH agent is running
eval `ssh-agent -s`
 
# Add your key to the agent
ssh-add ~/.ssh/id_rsa

 

Give it a try now and you should be all good. Thanks to Srikanth Kondaparthy and user456814 for their helpful posts on Stack Overflow.

Read More »

Posted in Linux, PHP

Isn’t it annoying when you want to connect to your home network while out and about but don’t know what your IP is? Sick of dynamic DNS sites with arbitrary restrictions on their free tiers? Well look no further! This tutorial demonstrates how to point your home IP to a subdomain of your website using a simple PHP script.

 

The Concept

  • Set up a Route 53 subdomain for pointing to your home
  • A device in your home uses a scheduled task to ping a URL on your website
  • That URL grabs the IP hitting it and points your subdomain to the IP.

Read More »

Posted (Updated ) in Linux

I want to schedule backups of my Ubuntu EC2’s EBS on a daily rolling schedule – ie a backup will occur once each day, and after 7 days the oldest snapshot is deleted – so there will always be 1 weeks worth of backups.

Read on for the implementation.

Read More »

Posted in Linux

When you create an Amazon EC2 instance, you’re given a .PEM private key allowing for passwordless entry to your server. Losing this key can be pretty costly but below I’ll show how to get you back in again.

The Problem

We’ve lost our PEM key or the one we have isn’t working:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$  ssh -vvv -i /path/to/my.pem ubuntu@host.com
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
...
debug2: key: /path/to/my.pem (0x0), explicit
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/me/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /path/to/my.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA 99:99:aa:9a:aa:99:99:a9:aa:99:99:99:99:9a:99:aa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

 

The Plan

We need to set a new authorized_key on our server. To do this we’ll:

  • Create a temporary new EC2 instance (E2) with a new keypair
  • Mount our servers EBS volume to E2
  • Set the authorized_key in our EBS volume to use our new key
  • Reattach the EBS to our original EC2 and log in.

 

The Implementation

I don’t like big wordy tutorials so here’s a tl;dr of all steps involved:

  • Create a snapshot of your EC2’s (E) EBS volume (V)
  • Create a new volume (V2) from the snapshot
  • Start new t2.micro EC2 Ubuntu instance (E2), using a new key pair
  • Attach V2 to E2, as /dev/xvdf (or /dev/sdf)
  • SSH in to E2
  • 1
    2
    3
    
    sudo mount /dev/xvdf1 /mnt/tmp -t ext4
    cp ~/.ssh/authorized_keys /mnt/tmp/home/ubuntu/.ssh/authorized_keys
    sudo umount /mnt/tmp
  • Detach V2 from E2
  • Stop E
  • Detach V from E
  • Attach V2 to E as /dev/sda1
  • Start E
  • Login as before, using your new .pem file
  • If all is well and you’re in, delete E2 and V

In my personal case, the above didn’t help and I was still getting the error Permission denied (publickey). I had to also copy E2‘s sshd_config because I’d borked E‘s and it was the actual reason I couldn’t SSH in.

So before the umount line above, also do:

1
2
3
sudo cp /etc/ssh/sshd_config /mnt/tmp/etc/ssh/sshd_config
mkdir /mnt/tmp/home/ubuntu/.ssh/bak
mv /mnt/tmp/home/ubuntu/.ssh/id_rsa /mnt/tmp/home/ubuntu/.ssh/id_rsa.pub /mnt/tmp/home/ubuntu/.ssh/known_hosts /mnt/tmp/home/ubuntu/.ssh/bak

Hope this helps.

Read More »

Posted in Linux

For those lucky Kodi users still owning an NYXBoard Hybrid, wake on USB is essential for a seamless HTPC experience. Below I’ll explain each step in getting that happening as well as some skin customisations to make things a little nicer down the road.

Wake on USB comes in 2 stages:

  1. Enabling it in the BIOS
  2. Enabling it in the OS

For this tutorial I’m running an Intel NUC 54250WYK with OpenELEC.

Read More »

Posted in Linux

Today I had a folder of files like so:

1
2
3
4
5
6
7
My Video - 01.ass
My Video - 01.mkv
My Video - 02.ass
My Video - 02.mkv
My Video - 03.ass
My Video - 03.mkv
...

I wanted to add the .ass subtitle files to my .mkv containers but I had about 100 videos and didn’t want to do each one manually.

Using mkvtoolnix you can combine the video and subtitles like so (courtesy of Super User):

1
mkvmerge -o output.mkv input.mkv subs.srt

Because our files are all nicely named we can use a for loop to iterate this command over them all simply replacing the .mkv extension with .ass in the third argument (See How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?):

1
for f in *.mkv; do mkvmerge -o "./muxed/$f" "$f" "${f%.mkv}.ass"; done

and that’s it! The fixed up files will be in your muxed subdirectory.

Read More »

Posted in Linux

One of the most annoying things about SABnzbd is it’s failure to correctly repair broken downloads – especially when you download multiple repairable files in a single combined NZB. This often stems from SABnzbd not including all relevant par2 and part files during its repair – so the par2 command assumes far more missing files than there actually are. Below I’ll explain a method for easily including all relevant files for a (hopefully) more successful repair.

 

The Par2 Command

First let’s look at the par2 command:

1
2
3
4
5
6
7
8
$ par2
...
Usage:
 
  par2 c(reate) [options] <par2 file> [files] : Create PAR2 files
  par2 v(erify) [options] <par2 file> [files] : Verify files using PAR2 file
  par2 r(epair) [options] <par2 file> [files] : Repair files using PAR2 files
...

Based on the above we need to call

1
par2 r <list of par2 files> <list of mkv parts>

But how do we get those lists?

 

Your Download

Let’s say I’ve (legally) downloaded of episodes 8 and 9 of MyShow!. My failed download folder might include the following files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
MyShow! - 08 [720p].mkv.001
MyShow! - 08 [720p].mkv.002
MyShow! - 08 [720p].mkv.003
MyShow! - 08 [720p].mkv.par2
MyShow! - 08 [720p].mkv.vol000+01.par2
MyShow! - 08 [720p].mkv.vol001+02.par2
MyShow! - 09 [720p].mkv.001
MyShow! - 09 [720p].mkv.002
MyShow! - 09 [720p].mkv.003
MyShow! - 09 [720p].mkv.004
MyShow! - 09 [720p].mkv.par2
MyShow! - 09 [720p].mkv.vol000+01.par2
MyShow! - 09 [720p].mkv.vol001+02.par2
MyShow! - 09 [720p].mkv.vol003+04.par2

We want to repair one episode at a time starting with episode 8. Here’s how I’d retrieve lists of just par2 then just the part files:

1
2
3
4
5
6
7
# Episode 8
ls *"- 08"*.par2 # Par2 files
ls *"- 08"*.[0-9][0-9][0-9] # Part files
 
# Episode 9
ls *"- 09"*.par2 # Par2 files
ls *"- 09"*.[0-9][0-9][0-9] # Part files

 

Putting it All Together

Combining par2 with my above lists results in:

1
2
par2 r *"- 08"*.par2 *"- 08"*.[0-9][0-9][0-9] # Ep 8
par2 r *"- 09"*.par2 *"- 09"*.[0-9][0-9][0-9] # Ep 9

You may also need to include *.mkv if that file exists but more often than not I’ve found it doesn’t.

Hope your repairs go a little smoother with this quick tip!

Read More »

Posted in Linux

If you’ve installed the MediaServer or PhotoStation packages on your Synology NAS you’ve probably noticed @eaDir directories popping up everywhere. These are “hidden” folders equivalent to thumbs.db on Windows where the package stores thumbnail files associated with iTunes support. If you’re not using iTunes you don’t need these directories. You can remove them in two steps:

Disable the Service Creating Them

SSH in as root and run the following:

1
2
cd /usr/syno/etc.defaults/rc.d/
chmod 000 S66fileindexd.sh S66synoindexd.sh S77synomkthumbd.sh S88synomkflvd.sh S99iTunes.sh

Remove the existing directories

Again in SSH use the following to locate them (cd to your volume root first):

1
find . -type d -name "@eaDir"

and if you’re feeling adventurous you can automatically delete them like so:

1
find . -type d -name "@eaDir" -print0 | xargs -0 rm -rf

Read More »