Cache::FileCache Thoughts

If using perl Cache::FileCache, some comments offering a helpful starting point to get a cache that works in a relatively well behaved and unsurprising way:

use Cache::FileCache;
use File::Spec;

my $cache = Cache::FileCache->new({
    # "The namespace associated with this
    # cache."
    namespace => 'BJD-TESTING-app_id_cache_id',
    # "The default expiration time for
    # objects place in the cache."
    # This is in seconds.
    default_expires_in => 1,
    # "Sets the auto purge interval. If this
    # option is set to a particular time ( in
    # the same format as the expires_in ),
    # then the purge( ) routine will be
    # called during the first set after the
    # interval expires. The interval will
    # then be reset."
    #
    # XXX To work needs at least one of
    # auto_purge_on_set or auto_purge_on_get
    # to be set to true
    auto_purge_interval => 5,
    auto_purge_on_set => 1,
    # restrict access to the cache to just this
    # user (data security)
    #
    # NOTE that if you set directory_umask
    # but no cache_root you can end up clashing
    # with other users who will also be trying
    # to use (and maybe set the umask) on
    # /tmp/FileCache
    #
    # If we care about data security set
    # cache_root as well to put the files in
    # out own private directory:
    directory_umask => 0077,
    cache_root =>
      File::Spec->tmpdir()
      . "/bjdean-perl-Cache-FileCache",
});

/usr/bin/base64 – copying and pasting code / patches betweeen terminals

The Scenario

A couple of terminals open, connected to a mix of your own workstation, local development servers and remote servers running on a different network and frequently behind a variety of security barriers.

You want to copy a smallish chunk of code from a file across the network, or the output of a diff to apply a patch – but characters like whitespace and newlines which should not change will frequently be modified through the copy. You end up with whitespace changes where you don’t want them (perhaps later causing source code control merges to fail, and patches will fail straight away).

base64 / base64 -d to the rescue

Use /usr/bin/base64 to make sure the data you copy between terminals remains unchanged.

For example – call base64 on a file to give you a safe-to-copy chunk of ASCII:

$ base64 example001
aGVsbG8gd29ybGQK

Which you can then get back with “base64 -d”:

$ echo aGVsbG8gd29ybGQK | base64 -d
hello world

Or you can just transfer a patch, for example:

$ cat example001
hello world
$ cat example002
hello all
$ diff -u example001 example002 
--- example001  2016-08-09 16:20:52.383813529 +1000
+++ example002  2016-08-09 16:21:08.675813172 +1000
@@ -1 +1 @@
-hello world
+hello all
$ diff -u example001 example002 | base64 
LS0tIGV4YW1wbGUwMDEJMjAxNi0wOC0wOSAxNjoyMDo1Mi4zODM4MTM1MjkgKzEwMDAKKysrIGV4
YW1wbGUwMDIJMjAxNi0wOC0wOSAxNjoyMTowOC42NzU4MTMxNzIgKzEwMDAKQEAgLTEgKzEgQEAK
LWhlbGxvIHdvcmxkCitoZWxsbyBhbGwK

# And then in another terminal...
$ cat | base64 -d | patch
LS0tIGV4YW1wbGUwMDEJMjAxNi0wOC0wOSAxNjoyMDo1Mi4zODM4MTM1MjkgKzEwMDAKKysrIGV4
YW1wbGUwMDIJMjAxNi0wOC0wOSAxNjoyMTowOC42NzU4MTMxNzIgKzEwMDAKQEAgLTEgKzEgQEAK
LWhlbGxvIHdvcmxkCitoZWxsbyBhbGwK
patching file example001
$ cat example001 
hello all

So long as the text is relatively small this allows you to safely transfer exactly the data you intended between terminals.

In a text editor like vi you can even just copy the base64 encoded data into the file you are editing, and then pipe the range including that base64 data through “base64 -d” to get the code chunk you need right where you need it.

Perl Gotcha : xor precedence

Perl has a logical xor operator but it happens to be at the very bottom of the precedence priority list , importantly below the humble comma. So this innocent looking piece of code:

foobar( $a xor $b, $c );

Is interpreted as:

foobar( $a xor ($b, $c) );

And note unlike the not, and and or low-precedence logical operators,  there is no high-precedence xor operator.

So it’s a good idea to wrap xors in parentheses to guarantee intended behaviour:

foobar( ($a xor $b), $c );

Avoiding Custom Internet Explorer Error Pages

A bit more of a short and to the point comment rather than an actual article.

Various web browsers used to display a “helpful” page saying “something is broken” when a website responded with a HTTP 4xx or 5xx error response. At a time when those pages were frequently the apache default error page these custom pages were still not a very good idea, but these days useful information is frequently included in error pages (in particular for errors like 410 Gone).

Most browsers have switched to showing the actual error page sent by the web server, but Internet Explorer keeps a version of this custom error page alive. Fortunately there’s a work-around which is that Internet Explorer does this with a silent switch when the error page has at least 512 bytes in the initial response (ie not including any linked content like scripts, styles and images).

The upshot of this is that these pages will display a custom error page on Internet Explorer:

While these pages should display the actual error pages:

References:

Docker FROM debian:latest

So in my last article I created a Docker container with nothing but a statically compiled helloworld in it. That demonstrated how it’s possible to define a very very (very very) simple container.

Here I define a similarly simple container with more complex dependencies – a hello world perl script requiring the Modern::Perl perl module. Now I don’t even need to build the executable binary, but all of a sudden I need to include enough of an environment to support my simple perl script.

As far as working with Docker this is not much more complex than changing “FROM scratch” to “FROM debian:latest”. The result is a much bigger container which will take more resources to run, but other than providing the computing resources to support that I really don’t have to care about it that much.

The Dockerfile now looks like this:

FROM debian:latest

# Get base module requirements
RUN apt-get --assume-yes update &&\
    apt-get --assume-yes install libmodern-perl-perl

# Copy the script in place and start running
COPY hello.pl /
CMD ["/hello.pl"]

But the container build process is no longer a couple of lines:

$ make docker
docker build --rm -t bradhelloperl .
Sending build context to Docker daemon 4.096 kB
Sending build context to Docker daemon 
Step 0 : FROM debian:latest
latest: Pulling from debian
902b87aaaec9: Already exists 
9a61b6b1315e: Already exists 
Digest: sha256:b42e664a14b4ed96f7891103a6d0002a6ae2f09f5380c3171566bc5b6446a8ce
Status: Downloaded newer image for debian:latest
 ---> 9a61b6b1315e
Step 1 : RUN apt-get --assume-yes update &&    apt-get --assume-yes install libmodern-perl-perl
 ---> Running in afe73c43d3a8
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:2 http://security.debian.org jessie/updates/main amd64 Packages [165 kB]
Get:3 http://httpredir.debian.org jessie InRelease [134 kB]
Get:4 http://httpredir.debian.org jessie-updates InRelease [123 kB]
Get:5 http://httpredir.debian.org jessie/main amd64 Packages [9038 kB]
Get:6 http://httpredir.debian.org jessie-updates/main amd64 Packages [3614 B]
Fetched 9527 kB in 40s (237 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libalgorithm-c3-perl libarchive-extract-perl libcgi-fast-perl libcgi-pm-perl
  libclass-c3-perl libclass-c3-xs-perl libcpan-meta-perl libdata-optlist-perl
  libdata-section-perl libfcgi-perl libgdbm3 liblog-message-perl
  liblog-message-simple-perl libmodule-build-perl libmodule-pluggable-perl
  libmodule-signature-perl libmro-compat-perl libpackage-constants-perl
  libparams-util-perl libpod-latex-perl libpod-readme-perl
  libregexp-common-perl libsoftware-license-perl libsub-exporter-perl
  libsub-install-perl libterm-ui-perl libtext-soundex-perl
  libtext-template-perl perl perl-modules rename
Suggested packages:
  perl-doc libterm-readline-gnu-perl libterm-readline-perl-perl make
  libb-lint-perl libcpanplus-dist-build-perl libcpanplus-perl
  libfile-checktree-perl libobject-accessor-perl
Recommended packages:
  libarchive-tar-perl
The following NEW packages will be installed:
  libalgorithm-c3-perl libarchive-extract-perl libcgi-fast-perl libcgi-pm-perl
  libclass-c3-perl libclass-c3-xs-perl libcpan-meta-perl libdata-optlist-perl
  libdata-section-perl libfcgi-perl libgdbm3 liblog-message-perl
  liblog-message-simple-perl libmodern-perl-perl libmodule-build-perl
  libmodule-pluggable-perl libmodule-signature-perl libmro-compat-perl
  libpackage-constants-perl libparams-util-perl libpod-latex-perl
  libpod-readme-perl libregexp-common-perl libsoftware-license-perl
  libsub-exporter-perl libsub-install-perl libterm-ui-perl
  libtext-soundex-perl libtext-template-perl perl perl-modules rename
0 upgraded, 32 newly installed, 0 to remove and 1 not upgraded.
Need to get 6606 kB of archives.
After this operation, 37.3 MB of additional disk space will be used.
Get:1 http://httpredir.debian.org/debian/ jessie/main libgdbm3 amd64 1.8.3-13.1 [30.0 kB]
Get:2 http://httpredir.debian.org/debian/ jessie/main libalgorithm-c3-perl all 0.09-1 [11.9 kB]
Get:3 http://httpredir.debian.org/debian/ jessie/main libarchive-extract-perl all 0.72-1 [24.8 kB]
Get:4 http://httpredir.debian.org/debian/ jessie/main perl-modules all 5.20.2-3+deb8u1 [2551 kB]
Get:5 http://httpredir.debian.org/debian/ jessie/main perl amd64 5.20.2-3+deb8u1 [2629 kB]
Get:6 http://httpredir.debian.org/debian/ jessie/main libcgi-pm-perl all 4.09-1 [213 kB]
Get:7 http://httpredir.debian.org/debian/ jessie/main libparams-util-perl amd64 1.07-2+b1 [23.5 kB]
Get:8 http://httpredir.debian.org/debian/ jessie/main libclass-c3-perl all 0.26-1 [22.9 kB]
Get:9 http://httpredir.debian.org/debian/ jessie/main libsub-exporter-perl all 0.986-1 [49.9 kB]
Get:10 http://httpredir.debian.org/debian/ jessie/main libdata-optlist-perl all 0.109-1 [10.6 kB]
Get:11 http://httpredir.debian.org/debian/ jessie/main libclass-c3-xs-perl amd64 0.13-2+b1 [15.2 kB]
Get:12 http://httpredir.debian.org/debian/ jessie/main libmodern-perl-perl all 1.20140107-1 [9964 B]
Get:13 http://httpredir.debian.org/debian/ jessie/main libmodule-build-perl all 0.421000-2 [265 kB]
Get:14 http://httpredir.debian.org/debian/ jessie/main libmodule-signature-perl all 0.73-1+deb8u2 [30.4 kB]
Get:15 http://httpredir.debian.org/debian/ jessie/main libmodule-pluggable-perl all 5.1-1 [25.0 kB]
Get:16 http://httpredir.debian.org/debian/ jessie/main libpackage-constants-perl all 0.04-1 [5820 B]
Get:17 http://httpredir.debian.org/debian/ jessie/main libpod-latex-perl all 0.61-1 [34.7 kB]
Get:18 http://httpredir.debian.org/debian/ jessie/main libregexp-common-perl all 2013031301-1 [173 kB]
Get:19 http://httpredir.debian.org/debian/ jessie/main libpod-readme-perl all 0.11-1 [15.3 kB]
Get:20 http://httpredir.debian.org/debian/ jessie/main libfcgi-perl amd64 0.77-1+b1 [39.1 kB]
Get:21 http://httpredir.debian.org/debian/ jessie/main libsoftware-license-perl all 0.103010-3 [119 kB]
Get:22 http://httpredir.debian.org/debian/ jessie/main libcgi-fast-perl all 1:2.04-1 [10.9 kB]
Get:23 http://httpredir.debian.org/debian/ jessie/main libcpan-meta-perl all 2.142690-1 [125 kB]
Get:24 http://httpredir.debian.org/debian/ jessie/main rename all 0.20-3 [12.4 kB]
Get:25 http://httpredir.debian.org/debian/ jessie/main libsub-install-perl all 0.928-1 [11.4 kB]
Get:26 http://httpredir.debian.org/debian/ jessie/main libmro-compat-perl all 0.12-1 [13.2 kB]
Get:27 http://httpredir.debian.org/debian/ jessie/main libdata-section-perl all 0.200006-1 [13.4 kB]
Get:28 http://httpredir.debian.org/debian/ jessie/main liblog-message-perl all 0.8-1 [26.0 kB]
Get:29 http://httpredir.debian.org/debian/ jessie/main liblog-message-simple-perl all 0.10-2 [8126 B]
Get:30 http://httpredir.debian.org/debian/ jessie/main libtext-template-perl all 1.46-1 [53.1 kB]
Get:31 http://httpredir.debian.org/debian/ jessie/main libterm-ui-perl all 0.42-1 [19.1 kB]
Get:32 http://httpredir.debian.org/debian/ jessie/main libtext-soundex-perl amd64 3.4-1+b2 [13.7 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 6606 kB in 16s (395 kB/s)
Selecting previously unselected package libgdbm3:amd64.
(Reading database ... 7528 files and directories currently installed.)
Preparing to unpack .../libgdbm3_1.8.3-13.1_amd64.deb ...
Unpacking libgdbm3:amd64 (1.8.3-13.1) ...
Selecting previously unselected package perl-modules.
Preparing to unpack .../perl-modules_5.20.2-3+deb8u1_all.deb ...
Unpacking perl-modules (5.20.2-3+deb8u1) ...
Selecting previously unselected package perl.
Preparing to unpack .../perl_5.20.2-3+deb8u1_amd64.deb ...
Unpacking perl (5.20.2-3+deb8u1) ...
Selecting previously unselected package libalgorithm-c3-perl.
Preparing to unpack .../libalgorithm-c3-perl_0.09-1_all.deb ...
Unpacking libalgorithm-c3-perl (0.09-1) ...
Selecting previously unselected package libarchive-extract-perl.
Preparing to unpack .../libarchive-extract-perl_0.72-1_all.deb ...
Unpacking libarchive-extract-perl (0.72-1) ...
Selecting previously unselected package libcgi-pm-perl.
Preparing to unpack .../libcgi-pm-perl_4.09-1_all.deb ...
Unpacking libcgi-pm-perl (4.09-1) ...
Selecting previously unselected package libfcgi-perl.
Preparing to unpack .../libfcgi-perl_0.77-1+b1_amd64.deb ...
Unpacking libfcgi-perl (0.77-1+b1) ...
Selecting previously unselected package libcgi-fast-perl.
Preparing to unpack .../libcgi-fast-perl_1%3a2.04-1_all.deb ...
Unpacking libcgi-fast-perl (1:2.04-1) ...
Selecting previously unselected package libclass-c3-perl.
Preparing to unpack .../libclass-c3-perl_0.26-1_all.deb ...
Unpacking libclass-c3-perl (0.26-1) ...
Selecting previously unselected package libclass-c3-xs-perl.
Preparing to unpack .../libclass-c3-xs-perl_0.13-2+b1_amd64.deb ...
Unpacking libclass-c3-xs-perl (0.13-2+b1) ...
Selecting previously unselected package libcpan-meta-perl.
Preparing to unpack .../libcpan-meta-perl_2.142690-1_all.deb ...
Unpacking libcpan-meta-perl (2.142690-1) ...
Selecting previously unselected package libparams-util-perl.
Preparing to unpack .../libparams-util-perl_1.07-2+b1_amd64.deb ...
Unpacking libparams-util-perl (1.07-2+b1) ...
Selecting previously unselected package libsub-install-perl.
Preparing to unpack .../libsub-install-perl_0.928-1_all.deb ...
Unpacking libsub-install-perl (0.928-1) ...
Selecting previously unselected package libdata-optlist-perl.
Preparing to unpack .../libdata-optlist-perl_0.109-1_all.deb ...
Unpacking libdata-optlist-perl (0.109-1) ...
Selecting previously unselected package libmro-compat-perl.
Preparing to unpack .../libmro-compat-perl_0.12-1_all.deb ...
Unpacking libmro-compat-perl (0.12-1) ...
Selecting previously unselected package libsub-exporter-perl.
Preparing to unpack .../libsub-exporter-perl_0.986-1_all.deb ...
Unpacking libsub-exporter-perl (0.986-1) ...
Selecting previously unselected package libdata-section-perl.
Preparing to unpack .../libdata-section-perl_0.200006-1_all.deb ...
Unpacking libdata-section-perl (0.200006-1) ...
Selecting previously unselected package liblog-message-perl.
Preparing to unpack .../liblog-message-perl_0.8-1_all.deb ...
Unpacking liblog-message-perl (0.8-1) ...
Selecting previously unselected package liblog-message-simple-perl.
Preparing to unpack .../liblog-message-simple-perl_0.10-2_all.deb ...
Unpacking liblog-message-simple-perl (0.10-2) ...
Selecting previously unselected package libmodern-perl-perl.
Preparing to unpack .../libmodern-perl-perl_1.20140107-1_all.deb ...
Unpacking libmodern-perl-perl (1.20140107-1) ...
Selecting previously unselected package libmodule-build-perl.
Preparing to unpack .../libmodule-build-perl_0.421000-2_all.deb ...
Adding 'diversion of /usr/bin/config_data to /usr/bin/config_data.diverted by libmodule-build-perl'
Adding 'diversion of /usr/share/man/man1/config_data.1.gz to /usr/share/man/man1/config_data.diverted.1.gz by libmodule-build-perl'
Unpacking libmodule-build-perl (0.421000-2) ...
Selecting previously unselected package libmodule-pluggable-perl.
Preparing to unpack .../libmodule-pluggable-perl_5.1-1_all.deb ...
Unpacking libmodule-pluggable-perl (5.1-1) ...
Selecting previously unselected package libmodule-signature-perl.
Preparing to unpack .../libmodule-signature-perl_0.73-1+deb8u2_all.deb ...
Unpacking libmodule-signature-perl (0.73-1+deb8u2) ...
Selecting previously unselected package libpackage-constants-perl.
Preparing to unpack .../libpackage-constants-perl_0.04-1_all.deb ...
Unpacking libpackage-constants-perl (0.04-1) ...
Selecting previously unselected package libpod-latex-perl.
Preparing to unpack .../libpod-latex-perl_0.61-1_all.deb ...
Adding 'diversion of /usr/bin/pod2latex to /usr/bin/pod2latex.bundled by libpod-latex-perl'
Adding 'diversion of /usr/share/man/man1/pod2latex.1.gz to /usr/share/man/man1/pod2latex.bundled.1.gz by libpod-latex-perl'
Unpacking libpod-latex-perl (0.61-1) ...
Selecting previously unselected package libregexp-common-perl.
Preparing to unpack .../libregexp-common-perl_2013031301-1_all.deb ...
Unpacking libregexp-common-perl (2013031301-1) ...
Selecting previously unselected package libpod-readme-perl.
Preparing to unpack .../libpod-readme-perl_0.11-1_all.deb ...
Unpacking libpod-readme-perl (0.11-1) ...
Selecting previously unselected package libtext-template-perl.
Preparing to unpack .../libtext-template-perl_1.46-1_all.deb ...
Unpacking libtext-template-perl (1.46-1) ...
Selecting previously unselected package libsoftware-license-perl.
Preparing to unpack .../libsoftware-license-perl_0.103010-3_all.deb ...
Unpacking libsoftware-license-perl (0.103010-3) ...
Selecting previously unselected package libterm-ui-perl.
Preparing to unpack .../libterm-ui-perl_0.42-1_all.deb ...
Unpacking libterm-ui-perl (0.42-1) ...
Selecting previously unselected package libtext-soundex-perl.
Preparing to unpack .../libtext-soundex-perl_3.4-1+b2_amd64.deb ...
Unpacking libtext-soundex-perl (3.4-1+b2) ...
Selecting previously unselected package rename.
Preparing to unpack .../archives/rename_0.20-3_all.deb ...
Unpacking rename (0.20-3) ...
Setting up libgdbm3:amd64 (1.8.3-13.1) ...
Setting up perl-modules (5.20.2-3+deb8u1) ...
Setting up perl (5.20.2-3+deb8u1) ...
update-alternatives: using /usr/bin/prename to provide /usr/bin/rename (rename) in auto mode
Setting up libalgorithm-c3-perl (0.09-1) ...
Setting up libarchive-extract-perl (0.72-1) ...
Setting up libcgi-pm-perl (4.09-1) ...
Setting up libfcgi-perl (0.77-1+b1) ...
Setting up libcgi-fast-perl (1:2.04-1) ...
Setting up libclass-c3-perl (0.26-1) ...
Setting up libclass-c3-xs-perl (0.13-2+b1) ...
Setting up libcpan-meta-perl (2.142690-1) ...
Setting up libparams-util-perl (1.07-2+b1) ...
Setting up libsub-install-perl (0.928-1) ...
Setting up libdata-optlist-perl (0.109-1) ...
Setting up libmro-compat-perl (0.12-1) ...
Setting up libsub-exporter-perl (0.986-1) ...
Setting up libdata-section-perl (0.200006-1) ...
Setting up liblog-message-perl (0.8-1) ...
Setting up liblog-message-simple-perl (0.10-2) ...
Setting up libmodern-perl-perl (1.20140107-1) ...
Setting up libmodule-build-perl (0.421000-2) ...
Setting up libmodule-pluggable-perl (5.1-1) ...
Setting up libmodule-signature-perl (0.73-1+deb8u2) ...
Setting up libpackage-constants-perl (0.04-1) ...
Setting up libpod-latex-perl (0.61-1) ...
Setting up libregexp-common-perl (2013031301-1) ...
Setting up libpod-readme-perl (0.11-1) ...
Setting up libtext-template-perl (1.46-1) ...
Setting up libsoftware-license-perl (0.103010-3) ...
Setting up libterm-ui-perl (0.42-1) ...
Setting up libtext-soundex-perl (3.4-1+b2) ...
Setting up rename (0.20-3) ...
update-alternatives: using /usr/bin/file-rename to provide /usr/bin/rename (rename) in auto mode
Processing triggers for libc-bin (2.19-18) ...
 ---> 5df0148ab594
Removing intermediate container afe73c43d3a8
Step 2 : COPY hello.pl /
 ---> 73621b2beb17
Removing intermediate container 073772d52012
Step 3 : CMD /hello.pl
 ---> Running in 32701bc83e03
 ---> 2ba9eec0fbfa
Removing intermediate container 32701bc83e03
Successfully built 2ba9eec0fbfa

And in this case I’ve create a perl script which loops forever to show what happens when a container keeps running:

#!/usr/bin/env perl

use Modern::Perl;

# autoflush output
$| = 1;

my $loop = 0;
while (1) {
    say "[${loop}] Hello World!";
    $loop++;
    sleep 1;
}

Now I start the container and ask it to run in the background:

$ make run
docker run -d bradhelloperl
e4cb410b97f18b70f18ba1a4b4ded3e310bc63f85dbaf58b776463ec4a61ab64

And to see what is being printed I view the container log:

 $ docker logs e4cb410b97f18b70f18ba1a4b4ded3e310bc63f85dbaf58b776463ec4a61ab64
[0] Hello World!
[1] Hello World!
[2] Hello World!
[3] Hello World!
[4] Hello World!
[5] Hello World!
[6] Hello World!
[7] Hello World!

And finally, I’ll have to kill this container off now as it’s going to run forever. To show a different way this can be done I’ll look up the human readable ID and use it instead:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
e4cb410b97f1        bradhelloperl       "/hello.pl"         4 minutes ago       Up 4 minutes                            silly_sinoussi 

$ docker kill silly_sinoussi
silly_sinoussi

$ docker rm silly_sinoussi
silly_sinoussi

Note I needed to both kill and remove the container. While the container was killed but not removed I could still view the logs (and see the final stdout output before the kill was run).

Docker FROM scratch

Docker is a framework which makes it easy to wrap linux applications in “containers” – a sort chroot’d jail but with tools to take away the pain of setting up dependencies and also providing tools to help automate setting up dependencies between those containers.

The Docker Hub provides a bunch of pre-defined containers from major projects like Ubuntu, mysql, Redis, nginx, wordpress, postgres and java. It’s also a central store (like github, but for Docker) of publicly created and licensed Docker containers.

What’s the simplest quickest way to start? Well there is a special base container called SCRATCH, which is used in the Docker official hello world container. A container based on SCRATCH has nothing in it, so whatever you add has to be fully self sufficient. The Docker official hello world does this with some assembly code compiled by nasm which certainly covers the angle of a stand-alone executable but I think is arguably more complex than a statically compiled hello world – so here we go:

helloworld

Doesn’t really need much in the way of an explanation, but I’m going to write it in c and I’m including a bit of code to grab an environment variable to show how these can be passed into a Docker container to configure the way a container operates.

Here’s the helloworld c code:

#include<stdio.h>
#include<stdlib.h>

int main(int argc, char** argv) {
  // Vars
  char *envVar;

  // do it
  printf("Hello From Brad\n");

  // Check for special environment variable, print it present
  envVar = getenv("BJDTEST");
  if ( envVar == NULL ) {
    printf("No special environment variable set - set BJDTEST to see this work\n");
  }
  else {
    printf("Special environment BJDTEST set to [%s]\n", envVar);
  }

  return 0;
}

That’s built statically the usual way – with this bit of Makefile:

helloworld: helloworld.c
    gcc -static -o helloworld helloworld.c

Note that without the static build the Docker container will fail because the linked libraries will not exist inside the Docker container.

The simplest of containers – FROM scratch

To defined a docker container you write a Dockerfile – this is just about as simple as it gets:

FROM scratch
COPY helloworld /
CMD ["/helloworld"]

Basically, start with the no-op scratch container, copy our statically compiled binary to a known location and run it.

Building this with docker is very fast, because there’s not much there and scratch is very small. Here’s another Makefile snipped:

docker: helloworld
      docker build --rm -t bradhello .

And here’s how that looks:

$ make docker
docker build --rm -t bradhello .
Sending build context to Docker daemon 885.2 kB
Sending build context to Docker daemon 
Step 0 : FROM scratch
 ---> 
Step 1 : COPY helloworld /
 ---> Using cache
 ---> a4a757585183
Step 2 : CMD /helloworld
 ---> Using cache
 ---> b32df200f4ed
Successfully built b32df200f4ed

And here’s how that runs, deleting the container after running otherwise it will leave an empty container waiting around to be removed:

$ docker run --rm=true bradhello
Hello From Brad
No special environment variable set - set BJDTEST to see this work

And finally, as you can see above my environment variable was not set. But I can do that – either a bunch of variables defined in a file and used with –env-file=<filename> or individually with –env=”foo=bar”.

$ docker run --rm=true --env="BJDTEST=value for demo" bradhello
Hello From Brad
Special environment BJDTEST set to [value for demo]

 

Getting WordPress Up and Going

Setting up WordPress server there were a couple of minor wrinkles to sort out. I’ve run a blog before before and that fell by the wayside when I started using a personal wiki instead. But this seems like a good opportunity to see how one of the very popular blogging platforms works and what’s involved in keeping that running under the hood.

I work primarily with Debian systems, so that was a natural place to start. The wordpress package makes it very easy to get the base dependencies going with a known supported version, so if you’re running a recent release of Debian that seems like a reasonable place to start as well. That said, this of course means that the package is reconfigured along Debian guidelines and I found that I needed to spend a little time working out how this was done before it made sense.

Directory Structure – Symlinks and the Filesystem Hierarchy Standard

Probably the most important thing to know is the way the content directories have been set up to adhere to the Debian Filesystem Hierarchy Standard - that is:

/srv/www/wp-content/<blog hostname>/

  • This directory contains all per-blog custom changes to the global WordPress installation. Symlinks are used to bring in content from /var/lib/wordpress/wp-content (and then in turn from /usr/share/wordpress).
  • Shared resources are symlinked from: /var/lib/wordpress/wp-content/

/var/lib/wordpress/wp-content/

  • Server-specific server-wide customization
  • Base level files/configuration are symlinked from: /usr/share/wordpress/

/usr/share/wordpress/

  • System-level files – treat as read only. These files are managed by the Debian package.
  • This is the Apache DocumentRoot of all WordPress instances on the server, the only content hosted from the previous two directories are under the wp-content sub-path of the blogs.
  • Note that .htaccess is symlinked to /etc/wordpress/htaccess (again to allow local customization without breaking the standard filesystem hierarchy).

There’s also some useful documentation at the Debian Wiki: WordPress

configuration Script – setup-mysql

The Debian package comes with a helper script to create the base database, configuration files and expected directory structure – this is /usr/share/doc/wordpress/examples/setup-mysql. Once used the apache configuration still needs to be manually updated, but the first step in setting up a new blog looks like:

sudo bash /usr/share/doc/wordpress/examples/setup-mysql -n brad_wordpress blog.bjdean.id.au

Apache configuration

To activate the blog instance(s) create an apache config file along the lines of:

### The default virtual host should not point to wordpress
<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  UseCanonicalName Off
  VirtualDocumentRoot
  /var/www/html
  Options All
</VirtualHost>

# For this configuration to work you'll also need to have mod_rewrite #and mod_vhost_alias loaded and working in Apache.
# To enable these modules run
# # a2enmod rewrite && a2enmod vhost_alias && /etc/init.d/apache2 restart

# Allow wordpress overrides in shared .htaccess
<Directory /usr/share/wordpress>
  AllowOverride All
</Directory>

# Config for wordpress
<Directory /srv/>
  Options FollowSymLinks
  AllowOverride None
  Require all granted
</Directory>
<Directory /var/lib/wordpress/>
  Options FollowSymLinks
  AllowOverride None
  Require all granted
</Directory>

# To add new WordPress blog copy and adjust this template:
#<VirtualHost *:80>
#  ServerAdmin XXXBlogHostNameXXX@example.org
#  ServerName XXXBlogHostNameXXX
#  ErrorLog ${APACHE_LOG_DIR}/XXXBlogHostNameXXX-error_log
#  TransferLog ${APACHE_LOG_DIR}/XXXBlogHostNameXXX-access_log
#
#  # Shared WordPress docroot
#  DocumentRoot /usr/share/wordpress
#
#  # wp-content in /srv/www/wp-content/$0
#  RewriteEngine On
#  RewriteRule ^/wp-content/(.*)$ /srv/www/wp-content/%{HTTP_HOST}/$1
#</VirtualHost>

# Current Blogs
<VirtualHost *:80>
  ServerAdmin admin@example.org
  ServerName blog.bjdean.id.au
  ErrorLog ${APACHE_LOG_DIR}/blog.bjdean.id.au-error_log
  TransferLog ${APACHE_LOG_DIR}/blog.bjdean.id.au-access_log

  # Shared WordPress docroot
  DocumentRoot /usr/share/wordpress

  # wp-content in /srv/www/wp-content/$0
  RewriteEngine On
  RewriteRule ^/wp-content/(.*)$ /srv/www/wp-content/%{HTTP_HOST}/$1
</VirtualHost>

What About Multi-Site?

WordPress introduced multi-site hosting within a single instance in version 3.0 but there is manual work to set up independent blog hostnames (ie blogs that are not in a subdirectory or subdomain) and for the purposes of setting updown this blog server I was more interested in keeping the blogs separate to give more flexibility down the track.

User-Friendly URLs

To support user-friendly URLs (a switch in the configuration of each blog instance) the /etc/wordpress/htaccess file needs to be updated (uncomment the first <IfModule block).

Protecting Joomla : User-Registration Spam Relay

The Problem: A Default Setting

By default user registration is enabled.

It’s important to realise that even though links to the user registration page may not have been included in the design of a Joomla site the components are still present and they will be regularly targetted by automatic spiders searching for vulnerable sites. Check access logs  for requests to paths like:

  • /index.php/shop-login
  • /index.php/shop-login?view=registration&layout=complete
  • /index.php/component/users/?view=registration
  • /index.php/component/user/?task=register
  • /index.php?option=com_user&view=register

With user registration enabled scripts can use a Joomla site as an open mail relay by registering users with target email addresses and inserting spam/attack payload into the user details. The Joomla site will send a confirmation email to the target email address, and any email tracing of the source of the email will lead directly to the weakened Joomla server.

Disable User Registration

The method varies depending on the version of Joomla, in later versions it is done as follows:

“Navigate to Users  User Manager then click on User Manager. Once in User Manager Screen, click on ‘Options’ Toolbar button to show the pop up window, as in image above. Click to fill bubble next to Allow user registration, then click save.”

Reference : Joomla Documentation: Disabling User Registration

If a site has had to be disabled due to this problem the Joomla Control Panel will not be available so the above steps are not useful. It is possible to disable user registration manually by updating the chvnp_extensions table – for name=’com_users’ (updating the JSON in the params column).

User Registration Is Still Possible

Site administrator can still add new accounts using the Joomla Control Panel under /administrator/.

Restrict Access To /adminstrator/

Speaking of Control Panel, /adminstrator/ is a known target on every Joomla website. There is no need for anyone without administrative access to have any access to that directory so at the very least add basic authentication to that directory.

Better yet, protect accesses to that directory with HTTPS to encrypt both your authentication credentials and your administrative traffic.

Site Cleanup

Once the attack vector is blocked the site will still have the new accounts left over from the attacks. Depending on how long the attack has been going on there may be many (hundreds of thousands, perhaps more) of these accounts.

Unfortunately the Joomla Control Panel does not handle a large number of accounts (read more than a couple of hundred) at all well. The default maximum number of accounts  listed “per page” is 100 and while this can be manually overridden by editing the site URL (eg. “&list[limit]=2000″) this doesn’t get you far with hundreds of thousands of accounts.

Also the process of deleting a user from a Joomla site is relatively involved with rows being deleted from a number of tables with a couple of different foreign keys in play, so just deleting the user manually is not an easy approach either.

Fortunately it’s quite easy to write a script to interact with the Control Panel (I used WWW::Mechanize) – select the user IDs that you don’t want to delete and then write a script which will remove all the others!

I’ve attached the perl script I used here: delete_joomla_users.txt

Site Modules Are Known Targets

As a footnote, and one which I’ve yet to fully explore – modules are known targets. This is not a Joomla-specific problem – once an attacker can identify the underlying system which provides a website (in particular content management systems which allow dynamic updates to content and so have more attack vectors than a static website) then known vulnerabilities can be targeted.

Other than keeping site software up to date, applying security updates and configuring the site such that it is not vulnerable (as described in this article) it’s important to keep auditing both the public contents (eg. web page contents, images, hidden embedded scripting) and private contents (eg. extra users registered).

So, This is Life

Marvin once said:

Life, don’t talk to me about life

But really, what was the problem. Life’s not so bad and after all : there are puppies, and fish, and ponies, and food, and walks in the forest, and walks through the grass, and cycling, and talking, and meeting and travelling and all those other things without which Life would not be Life.

So here’s a soothing image instead:

snoooooozing

Soothing Image No. 123132512233