Posts

I wrote about using netcat to measure bandwidth between servers which works perfectly well in a minimal sort of way (and in particular between servers where the relatively common netcat is installed). For a slightly more user-friendly approach consider iperf. Once installed on both servers (let’s call them serverA and serverB): start iperf to listen on one server (add a port particularly if there are firewall restrictions in place which need to be adjusted/worked-with): <span id="line-58" class="anchor"></span>serverA$ iperf -s -p 12345 start iperf to send data from the other server: <span id="line-61" class="anchor"></span>serverB$ iperf -c serverA -p 12345 iperf displays results / status on both servers: serverA$ iperf -s -p 12345 ------------------------------------------------------------ Server listening on TCP port 12345 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.1 port 12345 connected with 10.0.0.2 port 48728 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 989 MBytes 829 Mbits/sec serverB$ iperf -c serverA -p 12345 ------------------------------------------------------------ Client connecting to ServerA, TCP port 12345 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.2 port 48728 connected with 10.0.0.1 port 12345 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 989 MBytes 830 Mbits/sec Run this in both directions a few times to get a good feeling for the bandwidth between the servers. There are other options (eg. parallel dual-direction testing) to consider, so the man page is worth a read.

It’s important to read the documentation, and then read it again once or twice. PHP’s date_parse function parses a datetime string and returns an “array with information about the parsed date on success or FALSE on failure.” This makes it tempting to use the function something like: $datetime = date_parse($datetime_string); if ( $datetime === FALSE ) { # Failure - deal with the bad $datetime_string } else { # Success - use the $datetime } However in this case failure means the the date_parse function has failed to work at a fairly fundamental level and reading the docs further you find “In case the date format has an error, the element ’errors’ will contains the error messages.” And indeed: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR apply to micro, small and medium-sized enterprises (SMEs)? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. ...

This article starts with a conversation about adjusting compiled perl regular expressions and a sudden realisation - which went a little along the lines of: OtherPerson> I can force a regular expression to change, for example: OtherPerson> DB<1> $a = qr{Abc} OtherPerson> DB<2> $a =~ s/A/a/ OtherPerson> DB<3> x $a OtherPerson> 0 '(?^:abc)' Me> No that doesn't work does it, oh wait <strong>it does</strong>, why? It took me a few moments of poking this myself in the debugger to notice the key change - which was that the compiled regular expression had become a string. See in the below example that $a starts as a RegExp and then becomes a scalar (at which point the ref function returns the empty string): ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Revisited: see Revisiting how the GDPR affects my setting of cookies (for security : a legitimate interest) Question: How does the GDPR affect my setting of cookies? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. There is a fundamental change for cookies between the previous directive (and it’s modernising follow-up directive Directive 2002/58/EC) and the GDPR: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: Is my personal project subject to the GDPR? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. So you have a personal project which includes storage and processing of personal data, perhaps online (but perhaps it’s not even a computer based system - this regulation covers any data processing which has data in a filing system). First things first - your project is not covered by the GDPR unless the personal data in it includes EU residents - for more information please see http://blog.bjdean.id.au/2018/05/territorial-scope-of-the-gdpr/ . If your project is covered based on Territorial Scope then (just for personal projects) if your project is run by a “natural person” and if the nature of the work of the project is that it is “in the course of a purely personal or household activity” then it is explicitly not covered by the GDPR in Article 2. “Material Scope” Paragraph (2c). The nature of a “household activity” is not clearly defined - certainly it’s not in the Definitions of the regulation. Some vague examples are given in Recital (18) : “Personal or household activities could include correspondence and the holding of addresses, or social networking and online activity undertaken within the context of such activities.”. That the burden of proof is going to be on the controller (you, for your personal project). Exactly what qualifies as a household activity would seem to be something that would vary depending on the nature of the household, but you would be hard-pressed to claim that your particular household typically harvests, stores and processes EU resident personal data in bulk. In addition Recital (18) does specify that the activity should have “no connection to a professional or commercial activity” so even if your household is all about data processing, as it’s a professional and/or commercial activity it’s still going to be covered by the GDPR. ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR cover my data processing of EU resident personal data if I’m not an EU organisation? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. The GDPR unambiguously covers all processing of personal data of “data subjects” (defined as natural persons) who are EU residents by organisations and people who are in the EU. However the regulation also protects EU residents when using non-EU services in certain cases. For service providers outside the EU this means: ...

It’s often the case when analysing system logs that you want to create a summary bringing together data from different lines of a log - for example capturing an initial and final state of a transaction, or (as in this example) capturing a date-time stamp and some data from a multi-line log entry. In this example I have a log file containing periodic extracts of ‘mysqladmin extended-status’ with a date-time line to record when the status was taken - for example (removing most of the lines with “…” for brevity): ...

I’m at a conference, I have my ChromeBook (vanilla, no local linux installs or any such thing), I have internet and I’ve set up a VM out there in the cloud, I’m regularly closing my laptop and wondering between talks, so it would be handy to be able to resume my ssh session without having to restart and without having to type “screen -r” every time. My ChromeBook ssh client doesn’t let me set up a “ssh here and run this command” so instead, given that I’m only logging into my VM with a single session at a time I can add this to the end of my ~/.bashrc file: ...

I was entertained by Numberphile’s dice-based generation of an approximate of pi so thought I’d throw together a quick script to try this out: random_vals_pi The results look pretty good: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 100000000 100000; done 3.14130098162 3.13774230925 3.14174020697 3.13769082316 3.14274853819 The theory is based on having large integers, so reducing this reduces the accuracy of the approximation: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 10 100000; done 3.07962120525 3.0938112197 3.08601801522 3.08910857173 3.08599352408 And similarly reducing the number of samples does the same: ...

When you have more than one public key set up to be able to access a single account (ie more than one public key listed in the authorized_keys you may want to check which public key was used to make a login). Since openssh 6.3 (released 2013) the public key fingerprint is logged - for example the below shows a set of made up “Accepted publickey” entries from an ssh auth.log: ...

For plain bytes/second bandwidth testing - ie without taking things like encryption overhead and compression improvements into account - the netcat command-line utility is pretty handy. Once installed on both servers (let’s call them serverA and serverB): start netcat to listen on one server and pipe the output through wc -c to both count the bytes (for confirmation) but also so that the bytes are not written to a filesystem or the terminal (which would cause a bottleneck and likely reduce the apparent bandwidth). By default nc will quit when the first network connection it accepts is closed: <span id="line-58" class="anchor"></span>serverA$ nc -l -p 12345 | wc -c start netcat to send data from the other server - using dd to send as quickly as possible (using /dev/zero is fast). Using -q 0 will cause netcat to quit as soon as it sees an end of file (EOF): <span id="line-61" class="anchor"></span>serverB$ dd if=/dev/zero bs=$((2**20)) count=$((2**10)) | nc -q 0 serverA 12345 On the sending server (serverB) the output will show the number of bytes transmitted and the time it took to do that: <span id="line-66" class="anchor"></span>1048576+0 records in <span id="line-67" class="anchor"></span>1048576+0 records out <span id="line-68" class="anchor"></span>1073741824 bytes (1.1 GB) copied, 9.68678 s, 111 MB/s And on the other server (serverA) the number of bytes will be printed (confirming the transmission was complete): 1073741824 Run this in both directions a few times to get a good feeling for the bandwidth between the servers.

If using perl Cache::FileCache, some comments offering a helpful starting point to get a cache that works in a relatively well behaved and unsurprising way: use Cache::FileCache; use File::Spec; my $cache = Cache::FileCache->new({ # "The namespace associated with this # cache." namespace => 'BJD-TESTING-app_id_cache_id', # "The default expiration time for # objects place in the cache." # This is in seconds. default_expires_in => 1, # "Sets the auto purge interval. If this # option is set to a particular time ( in # the same format as the expires_in ), # then the purge( ) routine will be # called during the first set after the # interval expires. The interval will # then be reset." # # XXX To work needs at least one of # auto_purge_on_set or auto_purge_on_get # to be set to true auto_purge_interval => 5, auto_purge_on_set => 1, # restrict access to the cache to just this # user (data security) # # NOTE that if you set directory_umask # but no cache_root you can end up clashing # with other users who will also be trying # to use (and maybe set the umask) on # /tmp/FileCache # # If we care about data security set # cache_root as well to put the files in # out own private directory: directory_umask => 0077, cache_root => File::Spec->tmpdir() . "/bjdean-perl-Cache-FileCache", });

The Scenario A couple of terminals open, connected to a mix of your own workstation, local development servers and remote servers running on a different network and frequently behind a variety of security barriers. You want to copy a smallish chunk of code from a file across the network, or the output of a diff to apply a patch - but characters like whitespace and newlines which should not change will frequently be modified through the copy. You end up with whitespace changes where you don’t want them (perhaps later causing source code control merges to fail, and patches will fail straight away). ...

Perl has a logical xor operator but it happens to be at the very bottom of the precedence priority list , importantly below the humble comma. So this innocent looking piece of code: foobar( $a xor $b, $c ); Is interpreted as: foobar( $a xor <strong>($b, $c)</strong> ); And note unlike the not, and and or low-precedence logical operators, there is no high-precedence xor operator. So it’s a good idea to wrap xors in parentheses to guarantee intended behaviour: ...

A bit more of a short and to the point comment rather than an actual article. Various web browsers used to display a “helpful” page saying “something is broken” when a website responded with a HTTP 4xx or 5xx error response. At a time when those pages were frequently the apache default error page these custom pages were still not a very good idea, but these days useful information is frequently included in error pages (in particular for errors like 410 Gone). Most browsers have switched to showing the actual error page sent by the web server, but Internet Explorer keeps a version of this custom error page alive. Fortunately there’s a work-around which is that Internet Explorer does this with a silent switch when the error page has at least 512 bytes in the initial response (ie not including any linked content like scripts, styles and images). The upshot of this is that these pages will display a custom error page on Internet Explorer: ...

So in my last article I created a Docker container with nothing but a statically compiled helloworld in it. That demonstrated how it’s possible to define a very very (very very) simple container. Here I define a similarly simple container with more complex dependencies - a hello world perl script requiring the Modern::Perl perl module. Now I don’t even need to build the executable binary, but all of a sudden I need to include enough of an environment to support my simple perl script. As far as working with Docker this is not much more complex than changing “FROM scratch” to “FROM debian:latest”. The result is a much bigger container which will take more resources to run, but other than providing the computing resources to support that I really don’t have to care about it that much. The Dockerfile now looks like this: ...

Docker is a framework which makes it easy to wrap linux applications in “containers” - a sort chroot’d jail but with tools to take away the pain of setting up dependencies and also providing tools to help automate setting up dependencies between those containers. The Docker Hub provides a bunch of pre-defined containers from major projects like Ubuntu, mysql, Redis, nginx, wordpress, postgres and java. It’s also a central store (like github, but for Docker) of publicly created and licensed Docker containers. What’s the simplest quickest way to start? Well there is a special base container called SCRATCH, which is used in the Docker official hello world container. A container based on SCRATCH has nothing in it, so whatever you add has to be fully self sufficient. The Docker official hello world does this with some assembly code compiled by nasm which certainly covers the angle of a stand-alone executable but I think is arguably more complex than a statically compiled hello world - so here we go: ...

Setting up WordPress server there were a couple of minor wrinkles to sort out. I’ve run a blog before before and that fell by the wayside when I started using a personal wiki instead. But this seems like a good opportunity to see how one of the very popular blogging platforms works and what’s involved in keeping that running under the hood. I work primarily with Debian systems, so that was a natural place to start. The wordpress package makes it very easy to get the base dependencies going with a known supported version, so if you’re running a recent release of Debian that seems like a reasonable place to start as well. That said, this of course means that the package is reconfigured along Debian guidelines and I found that I needed to spend a little time working out how this was done before it made sense. ...

The Problem: A Default Setting By default user registration is enabled. It’s important to realise that even though links to the user registration page may not have been included in the design of a Joomla site the components are still present and they will be regularly targetted by automatic spiders searching for vulnerable sites. Check access logs for requests to paths like: /index.php/shop-login /index.php/shop-login?view=registration&layout=complete /index.php/component/users/?view=registration /index.php/component/user/?task=register /index.php?option=com_user&view=register With user registration enabled scripts can use a Joomla site as an open mail relay by registering users with target email addresses and inserting spam/attack payload into the user details. The Joomla site will send a confirmation email to the target email address, and any email tracing of the source of the email will lead directly to the weakened Joomla server. ...