xargs makes it very easy to quickly run a set of similar processes in parallel – but did you know when you’re half-way through a long list of tasks it’s possible to change the number of parallel processes that are being used?
It’s there in the man page under “P max-procs, –max-procs=max-procs” but it’s an easy feature to miss if you don’t read all the way through:
-P max-procs, --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done. While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number. You cannot increase it above an implementation-defined limit (which is shown with --show-limits). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one existing command to terminate before starting another. Please note that it is up to the called processes to properly manage parallel access to shared resources. For example, if more than one of them tries to print to stdout, the output will be produced in an indeterminate order (and very likely mixed up) unless the processes collaborate in some way to prevent this. Using some kind of locking scheme is one way to prevent such problems. In general, using a locking scheme will help ensure correct output but reduce performance. If you don't want to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources).
What does that look like? Spin up some slow processes and start with 3-way parallel execution:
$ seq 10 30 | xargs -t -I{} -L1 -P3 sleep
sleep 10
sleep 11
sleep 12
At this stage we have three processes running:
$ ps auxw | grep -e [s]leep -e [x]args
username 613932 0.0 0.0 8384 2028 pts/3 S+ 12:27 0:00 xargs -t -I{} -L1 -P 3 sleep
username 613933 0.0 0.0 8084 564 pts/3 S+ 12:27 0:00 sleep 10
username 613934 0.0 0.0 8084 560 pts/3 S+ 12:27 0:00 sleep 11
username 613935 0.0 0.0 8084 560 pts/3 S+ 12:27 0:00 sleep 12
Before the first sleep completes, increase the number of parallel processes by three (sending SIGUSR1 to the xargs process three times); xargs immediately spawns the extra processes:
$ kill -USR1 613932 $ kill -USR1 613932 $ kill -USR1 613932
And we see the new processes being spawned:
$ seq 10 30 | xargs -t -I{} -L1 -P 3 sleep
...
sleep 13
sleep 14
sleep 15
$ ps auxw | grep -e [s]leep -e [x]args
username 613932 0.0 0.0 8384 2024 pts/3 S+ 12:27 0:00 xargs -t -I{} -L1 -P 3 sleep
username 613933 0.0 0.0 8084 496 pts/3 S+ 12:27 0:00 sleep 10
username 613934 0.0 0.0 8084 560 pts/3 S+ 12:27 0:00 sleep 11
username 613935 0.0 0.0 8084 564 pts/3 S+ 12:27 0:00 sleep 12
username 613963 0.0 0.0 8084 564 pts/3 S+ 12:27 0:00 sleep 13
username 613964 0.0 0.0 8084 496 pts/3 S+ 12:27 0:00 sleep 14
username 613965 0.0 0.0 8084 560 pts/3 S+ 12:27 0:00 sleep 15
If the number of processes needs to be reduced, send SIGUSR2 to the xargs process the number of times needed. Note that xargs does not kill of ‘excess’ processes, it lets them complete and does not spawn any more parallel processes until the number has dropped below the new limit. The minimum limit is one (so you won’t stop execution by sending SIGUSR2 more times than the current limit):
# Sending more than the limit - the new limit will stop at one: $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932 $ kill -USR2 613932
From here on in – only one process will be running at a time (until the work is complete, or the limit is increased again):
$ ps auxw | grep -e [s]leep -e [x]args
username 613932 0.0 0.0 8384 2028 pts/3 S+ 12:27 0:00 xargs -t -I{} -L1 -P 3 sleep
username 615299 0.0 0.0 8084 560 pts/3 S+ 12:27 0:00 sleep 22
... time passes ...
$ ps auxw | grep -e [s]leep -e [x]args
username 613932 0.0 0.0 8384 1984 pts/3 S+ 12:34 0:00 xargs -t -I{} -L1 -P 3 sleep
username 615311 0.0 0.0 8084 512 pts/3 S+ 12:37 0:00 sleep 24