Posts

This is really a quick reminder about a perl function which does a little more than you’d perhaps expect. Need to convert a binary (or hex or octal) string to an integer? The perl documentation for the oct(EXPR) function starts out with: Interprets EXPR as an octal string and returns the corresponding value. Then includes the comment: (If EXPR happens to start off with 0x, interprets it as a hex string. If EXPR starts off with 0b, it is interpreted as a binary string. Leading whitespace is ignored in all three cases.) ...

A very quick and simple comment on building single-quoted strings in shell scripts which include single quotes. Note that it’s not possible to include a single quote in a single-quoted string - for example the bash man page: Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash. And dash: Enclosing characters in single quotes preserves the literal meaning of all the characters (except single quotes, making it impossible to put single-quotes in a single-quoted string). ...

Ever wondered why you can read the $LINES and $COLUMNS environment variables from your text shell and have them seemingly aware (or indeed, actually aware) of the size of the graphical terminal in which that shell is running? Enter SIGWINCH - a signal sent to processes when a window size changes. This signal causes a process to retrieve it’s current window size. For example in linux it is done though an ioctl call in termios.h: ...

The Arctic Death Spiral describes events unfolding such that we have a feedback loop whereby the the increasing amount of ice melting over summer is reducing the “heat mirroring” effect of the arctic and allowing the planet to absorb more heat, which in turn melts more ice the next summer… For more information see: Arctic Death Spiral

Scenario For a certain code-base it’s decided it would be useful to be able to trigger an exception on-demand (mid-way through a process working). Why? Perhaps we have a process which loops over a repeated task which takes a reasonable amount of time, and we know that from time to time the input data to this process might need to be quickly updated and we’d like to abort whatever the current loop is doing to read in the latest data. Meanwhile for another process we’d simply like some diagnostic information (which might be slightly expensive to generate) to be available on demand. We might also have a standard application design whereby all otherwise unhandled exceptions will be captured at the last moment (before causing a process to fail) and signal sysadmins with emails or management systems with failure states (rather than simply aborting a process with a kill and having to communicate that this has been done). Signals are helpful here, and we even have SIGUSR1 and SIGUSR2 (see “Standard Signals” in your handy signal(7) man page) allocated for user-defined purposes. Rather than having to build some other signalling method with our application, we can use one of those to trigger an exception by setting up a signal handler. ...

When testing network connections to a server using the TLS SNI extension to allow a single IP address to respond with different certificates the openssl s_client program supports this with the -servername command-line option: -servername name Set the TLS SNI (Server Name Indication) extension in the ClientHello message to the given value. If -servername is not provided, the TLS SNI extension will be populated with the name given to -connect if it follows a DNS name format. If -connect is not provided either, the SNI is set to “localhost”. This is the default since OpenSSL 1.1.1. Even though SNI should normally be a DNS name and not an IP address, if -servername is provided then that name will be sent, regardless of whether it is a DNS name or not. This option cannot be used in conjunction with -noservername. ...

The Problem Working with copying text between applications which use the X11 PRIMARY (eg. the quick-copy of selected text and pasting by clicking the middle mouse button) and those which use the CLIPBOARD (eg. usually GUI applications using Ctrl-C to copy and Ctrl-V to paste). The CLIPBOARD is also the buffer used (for example) when web browser javascript automatically copies selected text (which is frequently a hassle because it is added to make difficult to select text easy to copy, but it is not then available to middle-click paste into a terminal). For more detailed information see the X11 documentation. ...

A list of commands / references I’ve found useful. Also see my old wiki page. stdbuf - Run COMMAND, with modified buffering operations for its standard streams See stdbuf - Run COMMAND, with modified buffering operations for its standard streams Tracing the DNS glue record for a domain To find the glue records (if any) for a domain use (for example): dig +trace +additional positive-internet.com NS This will give a full trace on how the NS records for the domain were found, and if they end up using a glue record it will be visible (only if +additional is given in the command) - for example in the lookup above we start with the global servers, then find the servers for .com. and then the next response contains the information from the .com. servers as to where to find positive-internet.com. data and this includes glue records: ...

You want to use pip to install a module from PyPI. However your server distribution in a little old (at the time of writing this did not need to be all that old, so the problem has come up for me a number of times): example@localhost:~/piptest$ pip install mkdocs Downloading/unpacking mkdocs Cannot fetch index base URL http://pypi.python.org/simple/ Could not find any downloads that satisfy the requirement mkdocs No distributions at all found for mkdocs Storing complete log in /home/example/.pip/pip.log A first but not great fix is to simply override the install prefix using: ...

I wrote about using netcat to measure bandwidth between servers which works perfectly well in a minimal sort of way (and in particular between servers where the relatively common netcat is installed). For a slightly more user-friendly approach consider iperf. Once installed on both servers (let’s call them serverA and serverB): start iperf to listen on one server (add a port particularly if there are firewall restrictions in place which need to be adjusted/worked-with): <span id="line-58" class="anchor"></span>serverA$ iperf -s -p 12345 start iperf to send data from the other server: <span id="line-61" class="anchor"></span>serverB$ iperf -c serverA -p 12345 iperf displays results / status on both servers: serverA$ iperf -s -p 12345 ------------------------------------------------------------ Server listening on TCP port 12345 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.0.0.1 port 12345 connected with 10.0.0.2 port 48728 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 989 MBytes 829 Mbits/sec serverB$ iperf -c serverA -p 12345 ------------------------------------------------------------ Client connecting to ServerA, TCP port 12345 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.0.0.2 port 48728 connected with 10.0.0.1 port 12345 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 989 MBytes 830 Mbits/sec Run this in both directions a few times to get a good feeling for the bandwidth between the servers. There are other options (eg. parallel dual-direction testing) to consider, so the man page is worth a read.

It’s important to read the documentation, and then read it again once or twice. PHP’s date_parse function parses a datetime string and returns an “array with information about the parsed date on success or FALSE on failure.” This makes it tempting to use the function something like: $datetime = date_parse($datetime_string); if ( $datetime === FALSE ) { # Failure - deal with the bad $datetime_string } else { # Success - use the $datetime } However in this case failure means the the date_parse function has failed to work at a fairly fundamental level and reading the docs further you find “In case the date format has an error, the element ’errors’ will contains the error messages.” And indeed: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR apply to micro, small and medium-sized enterprises (SMEs)? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. ...

This article starts with a conversation about adjusting compiled perl regular expressions and a sudden realisation - which went a little along the lines of: OtherPerson> I can force a regular expression to change, for example: OtherPerson> DB<1> $a = qr{Abc} OtherPerson> DB<2> $a =~ s/A/a/ OtherPerson> DB<3> x $a OtherPerson> 0 '(?^:abc)' Me> No that doesn't work does it, oh wait <strong>it does</strong>, why? It took me a few moments of poking this myself in the debugger to notice the key change - which was that the compiled regular expression had become a string. See in the below example that $a starts as a RegExp and then becomes a scalar (at which point the ref function returns the empty string): ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Revisited: see Revisiting how the GDPR affects my setting of cookies (for security : a legitimate interest) Question: How does the GDPR affect my setting of cookies? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. There is a fundamental change for cookies between the previous directive (and it’s modernising follow-up directive Directive 2002/58/EC) and the GDPR: ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: Is my personal project subject to the GDPR? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. So you have a personal project which includes storage and processing of personal data, perhaps online (but perhaps it’s not even a computer based system - this regulation covers any data processing which has data in a filing system). First things first - your project is not covered by the GDPR unless the personal data in it includes EU residents - for more information please see http://blog.bjdean.id.au/2018/05/territorial-scope-of-the-gdpr/ . If your project is covered based on Territorial Scope then (just for personal projects) if your project is run by a “natural person” and if the nature of the work of the project is that it is “in the course of a purely personal or household activity” then it is explicitly not covered by the GDPR in Article 2. “Material Scope” Paragraph (2c). The nature of a “household activity” is not clearly defined - certainly it’s not in the Definitions of the regulation. Some vague examples are given in Recital (18) : “Personal or household activities could include correspondence and the holding of addresses, or social networking and online activity undertaken within the context of such activities.”. That the burden of proof is going to be on the controller (you, for your personal project). Exactly what qualifies as a household activity would seem to be something that would vary depending on the nature of the household, but you would be hard-pressed to claim that your particular household typically harvests, stores and processes EU resident personal data in bulk. In addition Recital (18) does specify that the activity should have “no connection to a professional or commercial activity” so even if your household is all about data processing, as it’s a professional and/or commercial activity it’s still going to be covered by the GDPR. ...

Disclaimer: I’m a Software Engineer who has spent a bunch of time working on issues relating to the GDPR from a systems design/security/process point of view. I’m definitely not a lawyer so this is my perspective and not legal advice. If you want legal advice, get a lawyer. Question: How does the GDPR cover my data processing of EU resident personal data if I’m not an EU organisation? The General Data Protection Regulation (GDPR) protects the personal data of EU residents who are in the EU. This regulation comes into effect 25th May 2018 (see Article 51/Paragraph 4) and has been designed to bring personal data protection up to date as the previous directive (Directive 95/46/EC) was adopted in 1995 and had fallen well behind in being able to cover modern day data protection issues. The GDPR unambiguously covers all processing of personal data of “data subjects” (defined as natural persons) who are EU residents by organisations and people who are in the EU. However the regulation also protects EU residents when using non-EU services in certain cases. For service providers outside the EU this means: ...

Description This is a simple chicken tagine featuring warm Moroccan spices - cumin, allspice, and cinnamon - along with sweet vegetables and dates. The long, slow cooking creates tender chicken and beautifully melded flavors. Stats ⏱️ Prep time: 15 minutes ✦ Cook time: 2+ hours ✦ Total time: 2 hours 15 minutes 🍽️ Serves: 4 people Ingredients 2-3 tablespoons olive oil 1 tablespoon ground cumin 0.5-0.7 tablespoon ground allspice 0.5-0.7 tablespoon ground cinnamon 1 brown onion 0.5 sweet potato 0.5 zucchini Chicken pieces (maryland recommended) 4-6 dates Instructions Heat base over very low heat and add olive oil and spices (cumin, allspice and cinnamon). Stir and cook spices for a minute or two. Finely dice the onion, increase to low-medium heat, add onion, and cook briefly. Roughly dice the sweet potato and zucchini and add to the pan. Place chicken pieces into pan. Mix all ingredients and cook five minutes to distribute heat. Halve and de-seed the dates, add around the top of the mixture in the pan. Move to very low heat and cover. Cook for at least two hours. Try to avoid lifting the lid to check as moisture will be lost. Notes Based on this recipe from my old wiki. This recipe uses stacked steel pots as a Dutch oven substitute rather than a traditional tagine vessel. The key to this dish is the long, slow cooking with minimal disturbance - this allows the flavors to develop and the chicken to become incredibly tender. Serve with couscous or rice.

It’s often the case when analysing system logs that you want to create a summary bringing together data from different lines of a log - for example capturing an initial and final state of a transaction, or (as in this example) capturing a date-time stamp and some data from a multi-line log entry. In this example I have a log file containing periodic extracts of ‘mysqladmin extended-status’ with a date-time line to record when the status was taken - for example (removing most of the lines with “…” for brevity): ...

I’m at a conference, I have my ChromeBook (vanilla, no local linux installs or any such thing), I have internet and I’ve set up a VM out there in the cloud, I’m regularly closing my laptop and wondering between talks, so it would be handy to be able to resume my ssh session without having to restart and without having to type “screen -r” every time. My ChromeBook ssh client doesn’t let me set up a “ssh here and run this command” so instead, given that I’m only logging into my VM with a single session at a time I can add this to the end of my ~/.bashrc file: ...

I was entertained by Numberphile’s dice-based generation of an approximate of pi so thought I’d throw together a quick script to try this out: random_vals_pi The results look pretty good: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 100000000 100000; done 3.14130098162 3.13774230925 3.14174020697 3.13769082316 3.14274853819 The theory is based on having large integers, so reducing this reduces the accuracy of the approximation: $ for x in 1 2 3 4 5 ; do python random_vals_pi.py 10 100000; done 3.07962120525 3.0938112197 3.08601801522 3.08910857173 3.08599352408 And similarly reducing the number of samples does the same: ...