Increasing / decreasing number of xargs parallel processes (at run time!)
xargs makes it very easy to quickly run a set of similar processes in parallel - but did you know when you’re half-way through a long list of tasks it’s possible to change the number of parallel processes that are being used? It’s there in the man page under “P max-procs, –max-procs=max-procs” but it’s an easy feature to miss if you don’t read all the way through: -P max-procs, --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time. Use the -n option or the -L option with -P; otherwise chances are that only one exec will be done. <strong>While xargs is running, you can send its process a SIGUSR1 signal to increase the number of commands to run simultaneously, or a SIGUSR2 to decrease the number.</strong> You cannot increase it above an implementation-defined limit (which is shown with --show-limits). You cannot decrease it below 1. xargs never terminates its commands; when asked to decrease, it merely waits for more than one existing command to terminate before starting another. Please note that it is up to the called processes to properly manage parallel access to shared resources. For example, if more than one of them tries to print to stdout, the output will be produced in an indeterminate order (and very likely mixed up) unless the processes collaborate in some way to prevent this. Using some kind of locking scheme is one way to prevent such problems. In general, using a locking scheme will help ensure correct output but reduce performance. If you don't want to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources). What does that look like? Spin up some slow processes and start with 3-way parallel execution: ...