Posts

This is a sentence*****. (I promise it’s safe to click up there - you’ll enjoy it.)

xargs makes it very easy to quickly run a set of similar processes in parallel - but did you know when you’re half-way through a long list of tasks it’s possible to change the number of parallel processes that are being used? It’s there in the man page under “P max-procs, –max-procs=max-procs” but it’s an easy feature to miss if you don’t read all the way through: -P max-procs, --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done. <strong>While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number.</strong> You cannot increase it above an implementation-defined limit (which is shown with --show-limits). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one existing command to terminate before starting another. Please note that it is up to the called processes to properly manage parallel access to shared resources. For example, if more than one of them tries to print to stdout, the output will be produced in an indeterminate order (and very likely mixed up) unless the processes collaborate in some way to prevent this. Using some kind of locking scheme is one way to prevent such problems. In general, using a locking scheme will help ensure correct output but reduce performance. If you don't want to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources). What does that look like? Spin up some slow processes and start with 3-way parallel execution: ...

Curious… While editing in vim you want to search and replace including a sub-string with an ampersand (&) - this doesn’t have an special regular expression meaning but given the input: &lt;foo&gt; And the search/replace (changing “foo” to “bar”): :s/&lt;foo&gt;/&lt;bar&gt; The result is: &lt;foo&gt;lt;bar&lt;foo&gt;gt; That looks… unexpected? Well, at least, undesired! What’s going on? Reading up on vim’s substitute command, we find a section on sub-replace-special where we find: <span class="h">magic nomagic action </span> <a class="d" href="https://vimhelp.org/change.txt.html#%26">&</a> \& replaced with the whole matched <a class="d" href="https://vimhelp.org/pattern.txt.html#pattern">pattern</a><span id="s%2F%5C%26" class="t"></span> \& <a class="d" href="https://vimhelp.org/change.txt.html#%26">&</a> replaced with <a class="d" href="https://vimhelp.org/change.txt.html#%26">&</a> Where magic is enabled by default - so what’s happening is this: On the right-hand side of the substitute (ie the output side) a non-escaped ampersand will be replace by the whole matched pattern, so the output suddenly makes sense (even though it’s still unwanted). Importantly note the ampersand on the left-hand-side is ok un-escaped as you would usually expect for a regular expression. In this particular case it looks like a complete mess because of having multiple ampersands on the right-hand-side, but it now makes sense (where I’ve shown the whole matched pattern in the output in bold for the two ampersands in the output): ...

While piping together commands that only output intermittently we run into the pipe buffers created by the pipe() system call (also see overview of pipes and FIFOs). This can particularly come into play when stringing together multiple pipes in a row (as there are multiple buffers to pass through). For example in the command below while “tail -f” flushes on activity and awk will flush on output but the grep in the middle ends up with a buffered pipe and so a quiet access.log will result in long delays before updates are shown: ...

To review disk usage recursively - a few different options exist (when scanning manually through with df and du are not enough). I have found ncdu to be fast and very easy to use. I’ve also used durep from time to time. For a desktop system (or a server with a X server handy) a few options exist. Some support remote scanning, though this can be slow and problematic as a network connection is required for the duration of the scan: ...

md Building a RAID array using mdadm - two primary steps: “mdadm –create” to build the array using available resources “mdadm –detail –scan” to build config string for /etc/mdadm/mdadm.conf Simple examples: RAID6 array Set up partitions to be used (in this case the whole disk): # for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done <span id="line-18" class="anchor"></span> Create the array (in this case, with one hot-spare): # mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1 <span id="line-21" class="anchor"></span> Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf): # mdadm --detail --scan <span id="line-24" class="anchor"></span>ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd <span id="line-25" class="anchor"></span> Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: “Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different.” (http://www.jerryweb.org/settings/raid/) ...

For old servers that need to stay that way (for whatever reason) updates are no longer available but you can access the packages that were available for that distro by pointing apt at the archive - for example see lenny: deb http://archive.debian.org/debian/ lenny contrib main non-free <span id="line-5" class="anchor"></span> And for ubuntu: deb http://old-releases.ubuntu.com/ubuntu/ natty main restricted universe multiverse <span id="line-11" class="anchor"></span>deb http://old-releases.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse <span id="line-12" class="anchor"></span>deb http://old-releases.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse

Synopsis #!/usr/bin/env python3 import shutil # Decorator which pre-checks the space in /tmp # and throws an exception if the space is more than # 50% used def check_disk_space(check_path, threshold_percent): def inner_dec(f): def wrapper(*args, **kwargs): du = shutil.disk_usage(check_path) used_pct = (du.used / du.total) * 100 if used_pct >= threshold_percent: raise Exception(f"Aborting call - {check_path} is >{threshold_percent} (={used_pct}) full") return f(*args, **kwargs) return wrapper return inner_dec # Build another pre-set decorator def check_tmp_over_50(f): return check_disk_space("/tmp", 50)(f) # Use the decorator on some function that # might need /tmp space @check_disk_space('/tmp', 50) def foo(a, b, c): print("Able to run foo - must have been disk space") @check_tmp_over_50 def bar(a, b, c): print("Able to run bar - must have been disk space") if __name__ == '__main__': try: foo(1,2,3) bar(1,2,3) except Exception as e: print(f'foo aborted with: {e}') Getting Started Decorator syntax and usage isn’t all that complicated - but at the moment you won’t find any help from the Python Tutorial (decorators aren’t mentioned in Defining Functions, nor in More on Defining Functions) and the Python Language Reference only really touches on the existence of decorators without much in the way of a detailed description in the Function definitions and Class definitions sections. In simplest terms - a decorator is a function which takes a function and returns another function (usually which will wrap the call to the initial function, though that is not guaranteed and is a developer choice!). The Synopsis above demonstrates the two main patterns: ...

This is for fun, for silliness and for do not use anywhere-ness - or to quote the instructions: If you’ve ever wondered just how egregious your use of Python could be, or how unnecessarily elaborately you could engineer a simple piece of code, we want your entries in our Rube Codeberg competition! Named for the famously complicated machines drawn by American cartoonist Rube Goldberg, the Codeberg competition is a chance for you to showcase your creativity. Everyone is welcome to participate in the competition - or just tune in as the results are announced. You may find you learn a thing or two about how Python ticks along the way. ...

When using and monitoring AWS for EC2 instances and their attached EBS volumes there are a couple of very important metrics to keep an eye on which can have enormous performance and availability implications. In particular I’m writing about General Purpose EC2 instance running in standard mode (eg. T2, T3 and T3a at the time of writing) and General Purpose EBS (gp2) because these are very common low-mid spec instances and the default disk storage type. If you’re already using one of the other EC2 or EBS types, chances are you’re already aware of some of the issues I’ll discuss below and those other products are designed to manage CPU and disk resources in a different load targeted way. These metrics are important because they report on the way these AWS resources (otherwise designed to mimic a real hardware) operate in a very different way. Note that while CPU credit reports are shown in the AWS web console for an EC2 instance under it’s Monitoring tab (and so people tend to see it), the EBS credit reports are not. To see these you need to find the EBS volume(s) attached to an EC2 instance (this is linked from the EC2 Description tab) and then look at the Monitoring tab for each EBS volume. ...

A bunch of processes have failed - and you’d like to restart them in a screen session in case you need to rerun them in an interactive shells (for instance to answer prompts from the processes) - lots of Ctrl-A-C … start command …. Ctrl-A-S … name the window … and repeat later there has to be an easier way! Step 1: Create a background screen session to hold the runs This will open a new screen session named “ScreenSessionName” into the background (so you don’t need to Ctrl-A-d): ...

This is really a quick reminder about a perl function which does a little more than you’d perhaps expect. Need to convert a binary (or hex or octal) string to an integer? The perl documentation for the oct(EXPR) function starts out with: Interprets EXPR as an octal string and returns the corresponding value. Then includes the comment: (If EXPR happens to start off with 0x, interprets it as a hex string. If EXPR starts off with 0b, it is interpreted as a binary string. Leading whitespace is ignored in all three cases.) ...

A very quick and simple comment on building single-quoted strings in shell scripts which include single quotes. Note that it’s not possible to include a single quote in a single-quoted string - for example the bash man page: Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash. And dash: Enclosing characters in single quotes preserves the literal meaning of all the characters (except single quotes, making it impossible to put single-quotes in a single-quoted string). ...

Ever wondered why you can read the $LINES and $COLUMNS environment variables from your text shell and have them seemingly aware (or indeed, actually aware) of the size of the graphical terminal in which that shell is running? Enter SIGWINCH - a signal sent to processes when a window size changes. This signal causes a process to retrieve it’s current window size. For example in linux it is done though an ioctl call in termios.h: ...

The Arctic Death Spiral describes events unfolding such that we have a feedback loop whereby the the increasing amount of ice melting over summer is reducing the “heat mirroring” effect of the arctic and allowing the planet to absorb more heat, which in turn melts more ice the next summer… For more information see: Arctic Death Spiral

Scenario For a certain code-base it’s decided it would be useful to be able to trigger an exception on-demand (mid-way through a process working). Why? Perhaps we have a process which loops over a repeated task which takes a reasonable amount of time, and we know that from time to time the input data to this process might need to be quickly updated and we’d like to abort whatever the current loop is doing to read in the latest data. Meanwhile for another process we’d simply like some diagnostic information (which might be slightly expensive to generate) to be available on demand. We might also have a standard application design whereby all otherwise unhandled exceptions will be captured at the last moment (before causing a process to fail) and signal sysadmins with emails or management systems with failure states (rather than simply aborting a process with a kill and having to communicate that this has been done). Signals are helpful here, and we even have SIGUSR1 and SIGUSR2 (see “Standard Signals” in your handy signal(7) man page) allocated for user-defined purposes. Rather than having to build some other signalling method with our application, we can use one of those to trigger an exception by setting up a signal handler. ...

When testing network connections to a server using the TLS SNI extension to allow a single IP address to respond with different certificates the openssl s_client program supports this with the -servername command-line option: -servername name Set the TLS SNI (Server Name Indication) extension in the ClientHello message to the given value. If -servername is not provided, the TLS SNI extension will be populated with the name given to -connect if it follows a DNS name format. If -connect is not provided either, the SNI is set to “localhost”. This is the default since OpenSSL 1.1.1. Even though SNI should normally be a DNS name and not an IP address, if -servername is provided then that name will be sent, regardless of whether it is a DNS name or not. This option cannot be used in conjunction with -noservername. ...

The Problem Working with copying text between applications which use the X11 PRIMARY (eg. the quick-copy of selected text and pasting by clicking the middle mouse button) and those which use the CLIPBOARD (eg. usually GUI applications using Ctrl-C to copy and Ctrl-V to paste). The CLIPBOARD is also the buffer used (for example) when web browser javascript automatically copies selected text (which is frequently a hassle because it is added to make difficult to select text easy to copy, but it is not then available to middle-click paste into a terminal). For more detailed information see the X11 documentation. ...

A list of commands / references I’ve found useful. Also see my old wiki page. stdbuf - Run COMMAND, with modified buffering operations for its standard streams See stdbuf - Run COMMAND, with modified buffering operations for its standard streams Tracing the DNS glue record for a domain To find the glue records (if any) for a domain use (for example): dig +trace +additional positive-internet.com NS This will give a full trace on how the NS records for the domain were found, and if they end up using a glue record it will be visible (only if +additional is given in the command) - for example in the lookup above we start with the global servers, then find the servers for .com. and then the next response contains the information from the .com. servers as to where to find positive-internet.com. data and this includes glue records: ...

You want to use pip to install a module from PyPI. However your server distribution in a little old (at the time of writing this did not need to be all that old, so the problem has come up for me a number of times): example@localhost:~/piptest$ pip install mkdocs Downloading/unpacking mkdocs Cannot fetch index base URL http://pypi.python.org/simple/ Could not find any downloads that satisfy the requirement mkdocs No distributions at all found for mkdocs Storing complete log in /home/example/.pip/pip.log A first but not great fix is to simply override the install prefix using: ...