Issue: How can you simultaneously use all the available processors on a set of parallel machines to run independent serial jobs? Your starting point is a serial job that already runs in the CTC batch system.

Solution: There are two different ways to do this. It will be apparent which method will suit you better. We will explain this in detail so that you can run your serial jobs without having to learn about parallel programming. Both methods use the mpirun command, normally used to invoke a parallel program. The first method we will refer to as Deterministic. The second is Farm Out Work. Sample files/templates are provided for each.

Important Consideration:

• Deterministic: This is the simpler of the two methods. It is especially well-suited to the situation where the computation for each task takes a comparable amount of time so that no processor will be spend much time idle. Should this not be the case, you may want to consider the Farm Out Work method, which does a better job of load balancing. We call this method deterministic because the number of serial jobs you intend to run is fixed and is equal to the number of processors. Each processor is therefore assigned one serial job. When you run a serial job on one machine, all commands are executed on that machine and there is no interaction with any other machine.

  To simultaneously use more than one machine requires coordination of the machines. The mpirun command takes care of this. There is a sequence of three mpirun commands. The first mpirun is used to execute a setup file to create local subdirectories on T: of each machine and copy files from H: to T:. The second executes the serial jobs. The third copies the output from T: on each machine back to H:. Each time an mpirun command completes, control is transferred to the next command in the .bat file you ccsubmitted.

  Inside the mpirun environment, this scheme takes advantage of the environment variable MSTI_RANK, an integer that is the same as the task identifier. If there are n parallel tasks, they are numbered 0,...,n-1.

  We will present 2 examples. The first, and more common, situation is where the same executable is used with different input data. The second example shows how to handle the case where a different executable is used by each task.

  First Example: Presuming that the name of your executable is same_executable.exe, you put the statement set CODE=same_executable in the .bat file that you submit to run in the batch system. Therefore, specifying %CODE%.exe ensures that each serial task uses the same executable. Suppose that the input files read by the executable, one per task, are named input_data0.txt, input_data1.txt, ... input_datan-1.txt. The statement set INFILE=input_data will then be used with MSTI_RANK to allow each task to use a different input file. Each task is executed from a unique local directory T:\%USERNAME%\%MSTI_RANK%.

  • same_executable_submit.xml/same_executable_submit.bat The .xml file is that one that you submit to batch. It runs the .bat file.
  • same_executable_setup.bat The setup file used by same_executable_submit.bat.
  • same_executable_execute.bat The file that is executed by each process. This is the file that uses MSTI_RANK.
  • same_executable_cleanup.bat Cleanup after the job completes.

  Second Example: This uses Compiled Matlab, where each executable has a different name. CODE is defined in the .bat file that you submit to run in the batch system. Therefore, specifying %CODE%%MSTI_RANK%.exe ensures that each serial task is uniquely assigned to one of the executables files matlab_executable0.exe, matlab_executable1.exe, ... matlab_executablen-1.exe. Each task is executed from a unique local directory T:\%USERNAME%\%MSTI_RANK%.

  • matlab_submit.xml/matlab_submit.bat The .xml file is that one that you submit to batch. It runs the .bat file.
  • matlab_setup.bat The setup file used by matlab_submit.bat.
  • matlab_execute.bat The file that is executed by each process. This is the file that uses MSTI_RANK.
  • matlab_cleanup.bat Cleanup after the job completes.

• Farm Out Work: This is the method of choice when there are more serial jobs than there are processors. The essence of the scheme is to use a "master" program that farms out work to the processors on the machines as they become free. Processor 0 controls/monitors things and does not take part in the calculations. The overall structure is general and can be used to run an arbitrary sequence of serial codes.

  There are 4 files that are required to run the program. These are listed below and will be explained. You should save the .bat and .txt files to your own directory and customize them for your application. Be sure to save the files with the file extensions indicated.

  • farm_out_work_batch.xml/farm_out_work_batch.bat The .xml file is that one that you submit to batch. It runs the .bat file. Items in bold are ones that you will need to change.
  • setup.bat The setup file used by farm_out_work_batch.bat
  • farm_out_work.exe This is the file that manages the tasks. It is located in H:\CTC Tools, in the default path. It takes a single argument, the file farm_out_work_commands.txt, which contains the list of commands, one per line.
  • farm_out_work_commands.txt The set of commands processed by farm_out_work. In this example, each command is a .bat file, but any Windows command or executable name is also valid. Each line in the file is executed by the command system(...).
  Typical scenario: You have a .bat file for a serial job. Now you would like to take advantage of several V2 machines to run a series of jobs without having to submit them individually to the queue. We'll explain how to convert a typical .bat file for a serial job so that it can be used with farm_out_work.
  • The setup for creating directories and T: is done by the mpirun ... setup.bat command. Remove the commands relating to directory creation and file deletion on T:. You don't want to delete files on T: because they might be needed by a different serial job.
  • Likewise, after the computation completes, do not remove files on T:.
  • Remove the vsched -c. Otherwise the job will end. The only vsched -c should be in the file that you vsched -s.
  Additional Elements
  • Periodically copying files from each task to H:

    It may be useful periodically to see the results of your calculations from the tasks running on each of the machines. To implement this, just after the mpirun ... setup, add the command
    start /b mpirun -np %NPROC% %ROOTDIR%\parallel_copyback.bat.
    
    parallel_copyback.bat In this example, there is a loop where the process sleeps for 15 minutes, executes the commands, and then sleeps for another 15 minutes. The memory usage for processes starting with myexecutable is logged for each task. All .out and .err files are copied.

  • Limiting duration of a job so that it is not killed by the system

It is to your advantage to implement a clean shutdown of your job. You gain control of what happens in the rest of your .bat file. With a clean shutdown the system takes much less time to free up the machines so that they can be used for another job. The list of tasks in the file farm_out_work_commands.txt may not complete before the time limit for the job is reached. Let's say that you have set a limit of 1440 minutes(24 hours) with the <minutes> xml tag. Furthermore, let's presume that all commands except for the mpirun used to execute the job take less that 900 seconds(15 minutes). The command to use to limit the execution time for the mpirun is H:\CTC Tools\job_limiter.exe, in your default path. It requires two arguments. The first is a number of seconds. The second is the command that you would normally issue. After the specified number of seconds, the command specifed by the second argument and all its child processes are killed.

The original .bat file contains

mpirun -np %NPROC% farm_out_work.exe farm_out_work_commands.txt >stdout.out 2>stderr.err

For a job where you have asked for 1440 minutes(86400 seconds), modify your .bat file to add job_limiter 85500 before the mpirun command.

job_limiter 85500 mpirun -np %NPROC% farm_out_work.exe farm_out_work_commands.txt >stdout.out 2>stderr.err



• farm_out_work_batch_additional.xml/farm_out_work_batch_additional.bat
  The .xml file is that one that you submit to batch. It runs the .bat file. This .bat file expands upon that shown earlier by adding the parallel_copyback and job_limiter commands. Items in bold are ones that you will need to change.

If you have any questions, please send email to consult@tc.cornell.edu or call (607) 254-8686 and press 2 to speak to a consultant.