Posts

When testing network connections to a server using the TLS SNI extension to allow a single IP address to respond with different certificates the openssl s_client program supports this with the -servername command-line option: -servername name Set the TLS SNI (Server Name Indication) extension in the ClientHello message to the given value. If -servername is not provided, the TLS SNI extension will be populated with the name given to -connect if it follows a DNS name format. If -connect is not provided either, the SNI is set to “localhost”. This is the default since OpenSSL 1.1.1. Even though SNI should normally be a DNS name and not an IP address, if -servername is provided then that name will be sent, regardless of whether it is a DNS name or not. This option cannot be used in conjunction with -noservername. ...

The Problem Working with copying text between applications which use the X11 PRIMARY (eg. the quick-copy of selected text and pasting by clicking the middle mouse button) and those which use the CLIPBOARD (eg. usually GUI applications using Ctrl-C to copy and Ctrl-V to paste). The CLIPBOARD is also the buffer used (for example) when web browser javascript automatically copies selected text (which is frequently a hassle because it is added to make difficult to select text easy to copy, but it is not then available to middle-click paste into a terminal). For more detailed information see the X11 documentation. ...

A list of commands / references I’ve found useful. Also see my old wiki page. stdbuf - Run COMMAND, with modified buffering operations for its standard streams See stdbuf - Run COMMAND, with modified buffering operations for its standard streams Tracing the DNS glue record for a domain To find the glue records (if any) for a domain use (for example): dig +trace +additional positive-internet.com NS This will give a full trace on how the NS records for the domain were found, and if they end up using a glue record it will be visible (only if +additional is given in the command) - for example in the lookup above we start with the global servers, then find the servers for .com. and then the next response contains the information from the .com. servers as to where to find positive-internet.com. data and this includes glue records: ...

You want to use pip to install a module from PyPI. However your server distribution in a little old (at the time of writing this did not need to be all that old, so the problem has come up for me a number of times): example@localhost:~/piptest$ pip install mkdocs Downloading/unpacking mkdocs Cannot fetch index base URL http://pypi.python.org/simple/ Could not find any downloads that satisfy the requirement mkdocs No distributions at all found for mkdocs Storing complete log in /home/example/.pip/pip.log A first but not great fix is to simply override the install prefix using: ...

I wrote about using netcat to measure bandwidth between servers which works perfectly well in a minimal sort of way (and in particular between servers where the relatively common netcat is installed). For a slightly more user-friendly approach consider iperf. Once installed on both servers (let’s call them serverA and serverB): start iperf to listen on one server (add a port particularly if there are firewall restrictions in place which need to be adjusted/worked-with): <span id="line-58" class="anchor"></span>serverA$ iperf -s -p 12345 start iperf to send data from the other server: <span id="line-61" class="anchor"></span>serverB$ iperf -c serverA -p 12345 iperf displays results / status on both servers: serverA$ iperf -s -p 12345 ------------------------------------------------------------ Server listening on TCP port 12345 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.1 port 12345 connected with 10.0.0.2 port 48728 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 989 MBytes 829 Mbits/sec serverB$ iperf -c serverA -p 12345 ------------------------------------------------------------ Client connecting to ServerA, TCP port 12345 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.2 port 48728 connected with 10.0.0.1 port 12345 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 989 MBytes 830 Mbits/sec Run this in both directions a few times to get a good feeling for the bandwidth between the servers. There are other options (eg. parallel dual-direction testing) to consider, so the man page is worth a read.

It’s important to read the documentation, and then read it again once or twice. PHP’s date_parse function parses a datetime string and returns an “array with information about the parsed date on success or FALSE on failure.” This makes it tempting to use the function something like: $datetime = date_parse($datetime_string); if ( $datetime === FALSE ) { # Failure - deal with the bad $datetime_string } else { # Success - use the $datetime } However in this case failure means the the date_parse function has failed to work at a fairly fundamental level and reading the docs further you find “In case the date format has an error, the element ’errors’ will contains the error messages.” And indeed: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR apply to micro, small and medium-sized enterprises (SMEs)? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. ...

This article starts with a conversation about adjusting compiled perl regular expressions and a sudden realisation - which went a little along the lines of: OtherPerson> I can force a regular expression to change, for example: OtherPerson> DB<1> $a = qr{Abc} OtherPerson> DB<2> $a =~ s/A/a/ OtherPerson> DB<3> x $a OtherPerson> 0 '(?^:abc)' Me> No that doesn't work does it, oh wait <strong>it does</strong>, why? It took me a few moments of poking this myself in the debugger to notice the key change - which was that the compiled regular expression had become a string. See in the below example that $a starts as a RegExp and then becomes a scalar (at which point the ref function returns the empty string): ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Revisited: see Revisiting how the GDPR affects my setting of cookies (for security : a legitimate interest) Question: How does the GDPR affect my setting of cookies? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. There is a fundamental change for cookies between the previous directive (and it’s modernising follow-up directive Directive 2002/58/EC) and the GDPR: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: Is my personal project subject to the GDPR? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. So you have a personal project which includes storage and processing of personal data, perhaps online (but perhaps it’s not even a computer based system - this regulation covers any data processing which has data in a filing system). First things first - your project is not covered by the GDPR unless the personal data in it includes EU residents - for more information please see http://blog.bjdean.id.au/2018/05/territorial-scope-of-the-gdpr/ . If your project is covered based on Territorial Scope then (just for personal projects) if your project is run by a “natural person” and if the nature of the work of the project is that it is “in the course of a purely personal or household activity” then it is explicitly not covered by the GDPR in Article 2. “Material Scope” Paragraph (2c). The nature of a “household activity” is not clearly defined - certainly it’s not in the Definitions of the regulation. Some vague examples are given in Recital (18) : “Personal or household activities could include correspondence and the holding of addresses, or social networking and online activity undertaken within the context of such activities.”. That the burden of proof is going to be on the controller (you, for your personal project). Exactly what qualifies as a household activity would seem to be something that would vary depending on the nature of the household, but you would be hard-pressed to claim that your particular household typically harvests, stores and processes EU resident personal data in bulk. In addition Recital (18) does specify that the activity should have “no connection to a professional or commercial activity” so even if your household is all about data processing, as it’s a professional and/or commercial activity it’s still going to be covered by the GDPR. ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR cover my data processing of EU resident personal data if I’m not an EU organisation? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. The GDPR unambiguously covers all processing of personal data of “data subjects” (defined as natural persons) who are EU residents by organisations and people who are in the EU. However the regulation also protects EU residents when using non-EU services in certain cases. For service providers outside the EU this means: ...

It’s often the case when analysing system logs that you want to create a summary bringing together data from different lines of a log - for example capturing an initial and final state of a transaction, or (as in this example) capturing a date-time stamp and some data from a multi-line log entry. In this example I have a log file containing periodic extracts of ‘mysqladmin extended-status’ with a date-time line to record when the status was taken - for example (removing most of the lines with “…” for brevity): ...

I’m at a conference, I have my ChromeBook (vanilla, no local linux installs or any such thing), I have internet and I’ve set up a VM out there in the cloud, I’m regularly closing my laptop and wondering between talks, so it would be handy to be able to resume my ssh session without having to restart and without having to type “screen -r” every time. My ChromeBook ssh client doesn’t let me set up a “ssh here and run this command” so instead, given that I’m only logging into my VM with a single session at a time I can add this to the end of my ~/.bashrc file: ...

I was entertained by Numberphile’s dice-based generation of an approximate of pi so thought I’d throw together a quick script to try this out: random_vals_pi The results look pretty good: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 100000000 100000; done 3.14130098162 3.13774230925 3.14174020697 3.13769082316 3.14274853819 The theory is based on having large integers, so reducing this reduces the accuracy of the approximation: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 10 100000; done 3.07962120525 3.0938112197 3.08601801522 3.08910857173 3.08599352408 And similarly reducing the number of samples does the same: ...

When you have more than one public key set up to be able to access a single account (ie more than one public key listed in the authorized_keys you may want to check which public key was used to make a login). Since openssh 6.3 (released 2013) the public key fingerprint is logged - for example the below shows a set of made up “Accepted publickey” entries from an ssh auth.log: ...

For plain bytes/second bandwidth testing - ie without taking things like encryption overhead and compression improvements into account - the netcat command-line utility is pretty handy. Once installed on both servers (let’s call them serverA and serverB): start netcat to listen on one server and pipe the output through wc -c to both count the bytes (for confirmation) but also so that the bytes are not written to a filesystem or the terminal (which would cause a bottleneck and likely reduce the apparent bandwidth). By default nc will quit when the first network connection it accepts is closed: <span id="line-58" class="anchor"></span>serverA$ nc -l -p 12345 | wc -c start netcat to send data from the other server - using dd to send as quickly as possible (using /dev/zero is fast). Using -q 0 will cause netcat to quit as soon as it sees an end of file (EOF): <span id="line-61" class="anchor"></span>serverB$ dd if=/dev/zero bs=$((2**20)) count=$((2**10)) | nc -q 0 serverA 12345 On the sending server (serverB) the output will show the number of bytes transmitted and the time it took to do that: <span id="line-66" class="anchor"></span>1048576+0 records in <span id="line-67" class="anchor"></span>1048576+0 records out <span id="line-68" class="anchor"></span>1073741824 bytes (1.1 GB) copied, 9.68678 s, 111 MB/s And on the other server (serverA) the number of bytes will be printed (confirming the transmission was complete): 1073741824 Run this in both directions a few times to get a good feeling for the bandwidth between the servers.

If using perl Cache::FileCache, some comments offering a helpful starting point to get a cache that works in a relatively well behaved and unsurprising way: use Cache::FileCache; use File::Spec; my $cache = Cache::FileCache->new({ # "The namespace associated with this # cache." namespace => 'BJD-TESTING-app_id_cache_id', # "The default expiration time for # objects place in the cache." # This is in seconds. default_expires_in => 1, # "Sets the auto purge interval. If this # option is set to a particular time ( in # the same format as the expires_in ), # then the purge( ) routine will be # called during the first set after the # interval expires. The interval will # then be reset." # # XXX To work needs at least one of # auto_purge_on_set or auto_purge_on_get # to be set to true auto_purge_interval => 5, auto_purge_on_set => 1, # restrict access to the cache to just this # user (data security) # # NOTE that if you set directory_umask # but no cache_root you can end up clashing # with other users who will also be trying # to use (and maybe set the umask) on # /tmp/FileCache # # If we care about data security set # cache_root as well to put the files in # out own private directory: directory_umask => 0077, cache_root => File::Spec->tmpdir() . "/bjdean-perl-Cache-FileCache", });

The Scenario A couple of terminals open, connected to a mix of your own workstation, local development servers and remote servers running on a different network and frequently behind a variety of security barriers. You want to copy a smallish chunk of code from a file across the network, or the output of a diff to apply a patch - but characters like whitespace and newlines which should not change will frequently be modified through the copy. You end up with whitespace changes where you don’t want them (perhaps later causing source code control merges to fail, and patches will fail straight away). ...

Perl has a logical xor operator but it happens to be at the very bottom of the precedence priority list , importantly below the humble comma. So this innocent looking piece of code: foobar( $a xor $b, $c ); Is interpreted as: foobar( $a xor <strong>($b, $c)</strong> ); And note unlike the not, and and or low-precedence logical operators, there is no high-precedence xor operator. So it’s a good idea to wrap xors in parentheses to guarantee intended behaviour: ...

A bit more of a short and to the point comment rather than an actual article. Various web browsers used to display a “helpful” page saying “something is broken” when a website responded with a HTTP 4xx or 5xx error response. At a time when those pages were frequently the apache default error page these custom pages were still not a very good idea, but these days useful information is frequently included in error pages (in particular for errors like 410 Gone). Most browsers have switched to showing the actual error page sent by the web server, but Internet Explorer keeps a version of this custom error page alive. Fortunately there’s a work-around which is that Internet Explorer does this with a silent switch when the error page has at least 512 bytes in the initial response (ie not including any linked content like scripts, styles and images). The upshot of this is that these pages will display a custom error page on Internet Explorer: ...