During work on my machine learning project I needed to perform some quite computation-heavy calculations several times — each time with a bit different inputs. These calculations were CPU and memory bound, so just spawning them all at once would just slow down overall running time because of the increased amount of context switches. Yet running 4 (=number of cores in my CPU) of them at a time (actually, 3 since other applications need CPU, too) should speed it up.

Fortunately, I have an old laptop with 2 cores as well as access to a somewhat more modern machine with 4 cores. That results in 10 cores spread across 3 machines (all of 'em have some version of GNU Linux installed). The question was how to exploit such a treasury.

And the answer is GNU Parallel with some additional bells and whistles. GNU Parallel allows one to execute some commands in parallel and even in a distributed way.

The command was as follows:

parallel -u --wd ... -S :,host1,host2 --trc {}.emb "sh {}"

Here we have:

• **wd** stands for working directory. Three dots mean parallel's temporary folder
• **S** contains a list of hosts with : being localhost
• **trc** stands for "Transfer, Return, Cleanup" and means that we'd like to transfer an executable file to the target host, return the specified file and do a cleanup

parallel accepts a list of command arguments (file names) in standard input and executes a command (sh in my case) for each of them.

ls -1 jobs/* | parallel -u --wd ... -S :,host1,host2 --trc {}.emb "sh {}"

There's a problem: we usually need more than one file to do useful stuff. There are several solutions to that problem:

• **Bring all files manually**
  It's a solution, but a somewhat tedious one: setting up a computing environment on several machines is dull
• **tar it and do all the stuff in a command**
  Looks better, but some shell kung-fu is required
• **Use shar**
  Basically it's a tar archive with some shell commands for (self-)extracting. I chose this way and glued in some of my code.