SysAdmin on Brad's Blog

Understanding Linux Load Averages and Multi-Core Systems

Thu, 27 Nov 2025 11:34:44 +1100

How to interpret load averages on Linux systems and understand what happens when you add more CPU cores

Django Custom Management Commands

Wed, 12 Nov 2025 21:09:50 +1100

TL;DR

Django allows custom management commands to extend the core manage.py interface, allowing easy integration with the application when building backend processes, automation scripts, and scheduled jobs (while providing access into the Django application environment, data model and functions via the same structures used to build the website).

Where to create and/or find the code

Django discovers management commands through a specific directory layout in your apps:

your_app/
management/
__init__.py
commands/
__init__.py
send_notifications.py
update_reports.py

Every command file defines a Command class that extends BaseCommand:

Django Authentication and Permissions

Wed, 05 Nov 2025 15:24:37 +1100

TL;DR

Django provides a complete authentication and authorization sub-system out of the box. Use @login_required to restrict views to authenticated users and @permission_required to enforce granular access control based on custom permissions defined in your models.

Interesting!

Django automatically creates four default permissions for every model (add, change, delete, view) during migrations, but you can define custom permissions in your model’s Meta class to implement fine-grained authorization for any business logic you need.

Adding a Site to AWStats With Historical Logs

Mon, 13 Nov 2023 09:57:46 +1100

A quick walkthrough of adding a new site to AWStats and importing archived access logs without disrupting existing sites

Migrating git to svn (subversion)

Thu, 22 Jun 2023 15:36:50 +0000

I’ve found that most documentation / forum discussion around the web for this topic tends to be about migrating svn to git - so here’s a quick shell script (reconfigure the variables at the start) to help migrate from git to subversion.

This script is also available here: migrate-git-to-svn.sh.txt

#!/bin/bash
# Run this script from an empty working directory - it will:
# 1. git clone from the ORIGIN_GIT URL
# 2. run through a number of stages to construct a local svn repository
# 3. check out the svn repo for you to check
#
# NO error checking is done - you need to look at the output and
# look for any issues. This script DOES delete it's working directories
# on each run (so make sure to start in an empty directory to be safe!)
# Configuration
PROJECT_NAME=<strong>MyProjectName</strong>
ORIGIN_GIT="git@github.com:<strong>UserName</strong>/<strong>MyProjectName</strong>.git"
# Keep track of starting directory to make working in sub-directories easier
BASEDIR=$PWD
# Clone (bare) to main repo from main git repo
# Base because later stages want to talk only to a bare repo
echo "### Cloning git origin into local bare repo: ${PROJECT_NAME}.git"
if [ -d "${PROJECT_NAME}.git" ] ; then rm -rf "${PROJECT_NAME}.git" ; fi
git clone --bare "${ORIGIN_GIT}"
# Protect the real origin by removing it as a remote in the bare clone
echo "### Protect real origin by removing it as a remote in our clone"
( cd "${PROJECT_NAME}.git"; \
git remote remove origin; \
)
# Create an empty svn repository to migrate into
echo "### Create and initialise the target svn repository for the migration: ${PROJECT_NAME}.svnrepo"
if [ -d "${PROJECT_NAME}.svnrepo" ] ; then rm -rf "${PROJECT_NAME}.svnrepo" ; fi
mkdir "${PROJECT_NAME}.svnrepo"
svnadmin create "${PROJECT_NAME}.svnrepo"
svn mkdir --parents "file://${BASEDIR}/${PROJECT_NAME}.svnrepo/${PROJECT_NAME}/"{trunk,branches,tags} -m 'Inititalise empty svn repo'
# git svn (NOTE svn mode - needs the git-svn package installed on debian)
# Clone the new local svn repository into a git repo
# The --stdlayout option tells "git svn" that we are using the "standard" {trunk,branches,tags} directories
echo "### git-svn clone the target svn repo as a git directory (used to import from git and then export to svn): ${PROJECT_NAME}-git2svn"
if [ -d "${PROJECT_NAME}-git2svn" ] ; then rm -rf "${PROJECT_NAME}-git2svn" ; fi
git svn clone "file://${BASEDIR}/${PROJECT_NAME}.svnrepo/${PROJECT_NAME}" --stdlayout "${PROJECT_NAME}-git2svn"
# Set up the bare git clone as the origin for the "${PROJECT_NAME}-git2svn" clone
echo "### Add our git clone as the remote origin for ${PROJECT_NAME}-git2svn"
( cd "${PROJECT_NAME}-git2svn"; \
git remote add origin "file://${BASEDIR}/${PROJECT_NAME}.git"; \
)
# Import changes into an import branch in the "${PROJECT_NAME}-git2svn" clone and then export to svn
# Note:
# 1. git fetch first to get branch details
# 2. Then branch to an import branch tracking the remote origin/main
# 3. Rebase that onto master (rebase --root commits all reachable, allows to rebase the root commits)
# This builds the information needed to sync to svn via dcommit.
# 4. Then use svn dcommit - include author information (to help track who made changes)
echo "### Import full commit history into ${PROJECT_NAME}-git2svn and then send to subversion repo"
( cd "${PROJECT_NAME}-git2svn"; \
git fetch origin; \
git checkout -b import origin/main; \
git rebase --onto master --root; \
git svn dcommit --add-author-from ; \
)
# Checkout a svn working dir to check the export
echo "### Checking out a working svn directory to check the results: svn-check"
if [ -d svn-check ] ; then rm -rf svn-check ; fi
svn co "file://${BASEDIR}/${PROJECT_NAME}.svnrepo/${PROJECT_NAME}" svn-check
echo "Check the contents/log in svn-check/"

qemu - simplest command-line for a performant VM

Wed, 14 Jun 2023 15:19:59 +0000

TL;DR

You need to tell qemu to: use the local host CPU (rather than emulating one) and enable hypervisor mode, include a couple of CPU cores for performance, give it a hard disk and some memory and finally (after setting up a bridge network device on your host system) use a bridged network with a MAC address unique to your network. The final bit about networking requires root access - hence the sudo:

Increasing / decreasing number of xargs parallel processes (at run time!)

Fri, 18 Mar 2022 12:38:59 +0000

xargs makes it very easy to quickly run a set of similar processes in parallel - but did you know when you’re half-way through a long list of tasks it’s possible to change the number of parallel processes that are being used? It’s there in the man page under “P max-procs, –max-procs=max-procs” but it’s an easy feature to miss if you don’t read all the way through:

-P max-procs, --max-procs=max-procs
Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done. <strong>While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number.</strong> You cannot increase it above an implementation-defined limit (which is shown with --show-limits). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one existing command to terminate before starting another.
Please note that it is up to the called processes to properly manage parallel access to shared resources. For example, if more than one of them tries to print to stdout, the output will be produced in an indeterminate order (and very likely mixed up) unless the processes collaborate in some way to prevent this. Using some kind of locking scheme is one way to prevent such problems. In general, using a locking scheme will help ensure correct output but reduce performance. If you don't want to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources).

What does that look like? Spin up some slow processes and start with 3-way parallel execution:

stdbuf - Run COMMAND, with modified buffering operations for its standard streams

Wed, 25 Nov 2020 12:18:37 +0000

While piping together commands that only output intermittently we run into the pipe buffers created by the pipe() system call (also see overview of pipes and FIFOs). This can particularly come into play when stringing together multiple pipes in a row (as there are multiple buffers to pass through). For example in the command below while “tail -f” flushes on activity and awk will flush on output but the grep in the middle ends up with a buffered pipe and so a quiet access.log will result in long delays before updates are shown:

Disk Usage

Tue, 13 Oct 2020 15:00:05 +0000

To review disk usage recursively - a few different options exist (when scanning manually through with df and du are not enough).

I have found ncdu to be fast and very easy to use.

I’ve also used durep from time to time.

For a desktop system (or a server with a X server handy) a few options exist. Some support remote scanning, though this can be slow and problematic as a network connection is required for the duration of the scan:

md (software RAID) and lvm (logical volume management)

Tue, 13 Oct 2020 14:56:52 +0000

md

Building a RAID array using mdadm - two primary steps:

“mdadm –create” to build the array using available resources
“mdadm –detail –scan” to build config string for /etc/mdadm/mdadm.conf

Simple examples:

RAID6 array

Set up partitions to be used (in this case the whole disk):

# for x in /dev/sd{b,c,d,e,f}1 ; do fdisk $x ; done

Create the array (in this case, with one hot-spare):

# mdadm --create /dev/md0 --level=6 --raid-devices=4 --spare-devices=1 /dev/sd{b,c,d,e,f}1

Configure the array for reboot (append to the end of /etc/mdadm/mdadm.conf):

# mdadm --detail --scan
ARRAY /dev/md/0 metadata=1.2 spares=1 name=debian6-vm:0 UUID=9b42abcd:309fabcd:6bfbabcd:298dabcd

Considerations when setting up the partitions might be that any replacement disks will need to support that same size partition. Unconfirmed but it sounds like it might be a reasonable concern: “Enter a value smaller than the free space value minus 2% or the disk size to make sure that when you will later install a new disk in replacement of a failed one, you will have at least the same capacity even if the number of cylinders is different.” (http://www.jerryweb.org/settings/raid/)

Supporting old Debian distros

Tue, 13 Oct 2020 14:53:26 +0000

For old servers that need to stay that way (for whatever reason) updates are no longer available but you can access the packages that were available for that distro by pointing apt at the archive - for example see lenny:

Update /etc/apt/sources.list to use archive.debian.org:

deb http://archive.debian.org/debian/ lenny main contrib non-free
deb-src http://archive.debian.org/debian/ lenny main contrib non-free

And for ubuntu:

deb http://old-releases.ubuntu.com/ubuntu/ natty main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse

Also see:

A tale of two burst balances (AWS EC2 and EBS performance)

Fri, 04 Sep 2020 18:17:46 +0000

When using and monitoring AWS for EC2 instances and their attached EBS volumes there are a couple of very important metrics to keep an eye on which can have enormous performance and availability implications. In particular I’m writing about General Purpose EC2 instance running in standard mode (eg. T2, T3 and T3a at the time of writing) and General Purpose EBS (gp2) because these are very common low-mid spec instances and the default disk storage type. If you’re already using one of the other EC2 or EBS types, chances are you’re already aware of some of the issues I’ll discuss below and those other products are designed to manage CPU and disk resources in a different load targeted way. These metrics are important because they report on the way these AWS resources (otherwise designed to mimic a real hardware) operate in a very different way. Note that while CPU credit reports are shown in the AWS web console for an EC2 instance under it’s Monitoring tab (and so people tend to see it), the EBS credit reports are not. To see these you need to find the EBS volume(s) attached to an EC2 instance (this is linked from the EC2 Description tab) and then look at the Monitoring tab for each EBS volume.

Adding tasks to a background screen

Thu, 03 Sep 2020 20:13:41 +0000

A bunch of processes have failed - and you’d like to restart them in a screen session in case you need to rerun them in an interactive shells (for instance to answer prompts from the processes) - lots of Ctrl-A-C … start command …. Ctrl-A-S … name the window … and repeat later there has to be an easier way!

Step 1: Create a background screen session to hold the runs

This will open a new screen session named “ScreenSessionName” into the background (so you don’t need to Ctrl-A-d):

single quote characters in a single-quoted string in shells

Mon, 10 Feb 2020 10:03:40 +0000

A very quick and simple comment on building single-quoted strings in shell scripts which include single quotes. Note that it’s not possible to include a single quote in a single-quoted string - for example the bash man page:

Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

And dash:

Enclosing characters in single quotes preserves the literal meaning of all the characters (except single quotes, making it impossible to put single-quotes in a single-quoted string).

LINES and COLUMNS environment magic

Tue, 03 Dec 2019 22:17:07 +0000

Ever wondered why you can read the $LINES and $COLUMNS environment variables from your text shell and have them seemingly aware (or indeed, actually aware) of the size of the graphical terminal in which that shell is running? Enter SIGWINCH - a signal sent to processes when a window size changes. This signal causes a process to retrieve it’s current window size. For example in linux it is done though an ioctl call in termios.h:

XTermVT100selectToClipboard: true

Fri, 12 Apr 2019 23:08:45 +0000

The Problem

Working with copying text between applications which use the X11 PRIMARY (eg. the quick-copy of selected text and pasting by clicking the middle mouse button) and those which use the CLIPBOARD (eg. usually GUI applications using Ctrl-C to copy and Ctrl-V to paste). The CLIPBOARD is also the buffer used (for example) when web browser javascript automatically copies selected text (which is frequently a hassle because it is added to make difficult to select text easy to copy, but it is not then available to middle-click paste into a terminal). For more detailed information see the X11 documentation.

Useful Commands

Tue, 05 Feb 2019 23:30:40 +0000

A list of commands / references I’ve found useful. Also see my old wiki page.

stdbuf - Run COMMAND, with modified buffering operations for its standard streams

See stdbuf - Run COMMAND, with modified buffering operations for its standard streams

Tracing the DNS glue record for a domain

To find the glue records (if any) for a domain use (for example):

dig +trace +additional positive-internet.com NS

This will give a full trace on how the NS records for the domain were found, and if they end up using a glue record it will be visible (only if +additional is given in the command) - for example in the lookup above we start with the global servers, then find the servers for .com. and then the next response contains the information from the .com. servers as to where to find positive-internet.com. data and this includes glue records:

bandwidth measurement using iperf

Wed, 05 Sep 2018 04:45:08 +0000

I wrote about using netcat to measure bandwidth between servers which works perfectly well in a minimal sort of way (and in particular between servers where the relatively common netcat is installed). For a slightly more user-friendly approach consider iperf. Once installed on both servers (let’s call them serverA and serverB):

start iperf to listen on one server (add a port particularly if there are firewall restrictions in place which need to be adjusted/worked-with):

serverA$ iperf -s -p 12345

start iperf to send data from the other server:

serverB$ iperf -c serverA -p 12345

iperf displays results / status on both servers:

serverA$ iperf -s -p 12345
------------------------------------------------------------
Server listening on TCP port 12345
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.0.1 port 12345 connected with 10.0.0.2 port 48728
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 989 MBytes 829 Mbits/sec
serverB$ iperf -c serverA -p 12345
------------------------------------------------------------
Client connecting to ServerA, TCP port 12345
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.2 port 48728 connected with 10.0.0.1 port 12345
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 989 MBytes 830 Mbits/sec

Run this in both directions a few times to get a good feeling for the bandwidth between the servers. There are other options (eg. parallel dual-direction testing) to consider, so the man page is worth a read.

Joining log lines with sed

Tue, 12 Sep 2017 23:54:59 +0000

It’s often the case when analysing system logs that you want to create a summary bringing together data from different lines of a log - for example capturing an initial and final state of a transaction, or (as in this example) capturing a date-time stamp and some data from a multi-line log entry. In this example I have a log file containing periodic extracts of ‘mysqladmin extended-status’ with a date-time line to record when the status was taken - for example (removing most of the lines with “…” for brevity):

Conference, ChromeBook, A VM and Me

Mon, 07 Aug 2017 00:43:37 +0000

I’m at a conference, I have my ChromeBook (vanilla, no local linux installs or any such thing), I have internet and I’ve set up a VM out there in the cloud, I’m regularly closing my laptop and wondering between talks, so it would be handy to be able to resume my ssh session without having to restart and without having to type “screen -r” every time. My ChromeBook ssh client doesn’t let me set up a “ssh here and run this command” so instead, given that I’m only logging into my VM with a single session at a time I can add this to the end of my ~/.bashrc file:

Which ssh publickey was used to access an account

Mon, 10 Jul 2017 01:23:25 +0000

When you have more than one public key set up to be able to access a single account (ie more than one public key listed in the authorized_keys you may want to check which public key was used to make a login). Since openssh 6.3 (released 2013) the public key fingerprint is logged - for example the below shows a set of made up “Accepted publickey” entries from an ssh auth.log:

bandwidth measurement using netcat

Thu, 02 Mar 2017 01:19:55 +0000

For plain bytes/second bandwidth testing - ie without taking things like encryption overhead and compression improvements into account - the netcat command-line utility is pretty handy. Once installed on both servers (let’s call them serverA and serverB):

start netcat to listen on one server and pipe the output through wc -c to both count the bytes (for confirmation) but also so that the bytes are not written to a filesystem or the terminal (which would cause a bottleneck and likely reduce the apparent bandwidth). By default nc will quit when the first network connection it accepts is closed:

serverA$ nc -l -p 12345 | wc -c

start netcat to send data from the other server - using dd to send as quickly as possible (using /dev/zero is fast). Using -q 0 will cause netcat to quit as soon as it sees an end of file (EOF):

serverB$ dd if=/dev/zero bs=$((2**20)) count=$((2**10)) | nc -q 0 serverA 12345

On the sending server (serverB) the output will show the number of bytes transmitted and the time it took to do that:

1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 9.68678 s, 111 MB/s

And on the other server (serverA) the number of bytes will be printed (confirming the transmission was complete):

1073741824

Run this in both directions a few times to get a good feeling for the bandwidth between the servers.

/usr/bin/base64 - copying and pasting code / patches betweeen terminals

Tue, 09 Aug 2016 06:29:49 +0000

The Scenario

A couple of terminals open, connected to a mix of your own workstation, local development servers and remote servers running on a different network and frequently behind a variety of security barriers. You want to copy a smallish chunk of code from a file across the network, or the output of a diff to apply a patch - but characters like whitespace and newlines which should not change will frequently be modified through the copy. You end up with whitespace changes where you don’t want them (perhaps later causing source code control merges to fail, and patches will fail straight away).

Getting WordPress Up and Going

Sat, 18 Jul 2015 13:05:38 +0000

Setting up WordPress server there were a couple of minor wrinkles to sort out. I’ve run a blog before before and that fell by the wayside when I started using a personal wiki instead. But this seems like a good opportunity to see how one of the very popular blogging platforms works and what’s involved in keeping that running under the hood. I work primarily with Debian systems, so that was a natural place to start. The wordpress package makes it very easy to get the base dependencies going with a known supported version, so if you’re running a recent release of Debian that seems like a reasonable place to start as well. That said, this of course means that the package is reconfigured along Debian guidelines and I found that I needed to spend a little time working out how this was done before it made sense.

Protecting Joomla : User-Registration Spam Relay

Sat, 18 Jul 2015 10:35:05 +0000

The Problem: A Default Setting

By default user registration is enabled. It’s important to realise that even though links to the user registration page may not have been included in the design of a Joomla site the components are still present and they will be regularly targetted by automatic spiders searching for vulnerable sites. Check access logs for requests to paths like:

/index.php/shop-login
/index.php/shop-login?view=registration&layout=complete
/index.php/component/users/?view=registration
/index.php/component/user/?task=register
/index.php?option=com_user&view=register

With user registration enabled scripts can use a Joomla site as an open mail relay by registering users with target email addresses and inserting spam/attack payload into the user details. The Joomla site will send a confirmation email to the target email address, and any email tracing of the source of the email will lead directly to the weakened Joomla server.

SysAdmin on Brad's Blog

Understanding Linux Load Averages and Multi-Core Systems

Django Custom Management Commands

TL;DR

Where to create and/or find the code

Django Authentication and Permissions

TL;DR

Interesting!

Adding a Site to AWStats With Historical Logs

Migrating git to svn (subversion)

qemu - simplest command-line for a performant VM

TL;DR

Increasing / decreasing number of xargs parallel processes (at run time!)

stdbuf - Run COMMAND, with modified buffering operations for its standard streams

Disk Usage

md (software RAID) and lvm (logical volume management)

md

RAID6 array

Supporting old Debian distros

A tale of two burst balances (AWS EC2 and EBS performance)

Adding tasks to a background screen

Step 1: Create a background screen session to hold the runs

single quote characters in a single-quoted string in shells

LINES and COLUMNS environment magic

XTerm*VT100*selectToClipboard: true

The Problem

Useful Commands

stdbuf - Run COMMAND, with modified buffering operations for its standard streams

Tracing the DNS glue record for a domain

bandwidth measurement using iperf

Joining log lines with sed

Conference, ChromeBook, A VM and Me

Which ssh publickey was used to access an account

bandwidth measurement using netcat

/usr/bin/base64 - copying and pasting code / patches betweeen terminals

The Scenario

Getting WordPress Up and Going

Protecting Joomla : User-Registration Spam Relay

The Problem: A Default Setting

XTermVT100selectToClipboard: true