Overview: Cornell Multitask Toolbox for MATLABŪ

MATLABŪ is an interactive "matrix laboratory" for tasks involving matrices, graphics and general numerical computation. The Cornell Multitask Toolbox for MATLABŪ enables multiple copies of MATLABŪ running simultaneously on a network of workstations to exchange matrices, thus facilitating parallel computations.

When a computation becomes so large that it cannot be completed in a reasonable amount of time on a single computer, one way to get answers more quickly is to utilize several computers operating in parallel. If the computation has been carried out with MATLABŪ, it is desirable to move from serial to parallel computation without leaving the MATLABŪ programming environment.

 

The multitask toolbox enables multiple copies of MATLABŪ running simultaneously on a network of workstations to exchange matrices, thus facilitating parallel computations. It also enables more complete utilization of multiprocessor machines by giving the programmer the ability to run several instances of MATLABŪ in concert on a single machine.

 

The multitask toolbox makes message-passing functions available to the programmer so that parallel programs can be written as M-files. Here is a "Hello, World" example for two copies of MATLABŪ using the multitask toolbox:

 

MM_Eval(1,'message=''Hello, World'';');
MM_Eval(1,'MMPI_Send(message,0,1);');
message=MMPI_Recv(1,1)
This program could be entered at the command prompt in the window for the master task, or it could be placed in a file named MM_run.m and it would be executed automatically when the multitask toolbox is initialized.

 

CMTM is installed on the machines in the CTC Collaboratory and runs under Microsoft Windows XP or Server 2003. It is not available for any other operating system at present.
CMTM requires MPI/Pro 1.7.0 or later and Matlab R2006a or later

 

General Notes
All multitask toolbox functions and variables begin with the prefix MM_ or MMPI_. No user functions or variables should use those prefixes. This convention is similar to that adopted by the MPI Forum.

 

Differences from MPI
Because MATLAB matrices contain information about their contents, it is unnecessary for the user to specify the number of elements and their type as one does in MPI. Matrices may be of any numeric or character datatype that is compatible with the operation to be performed. Object matrices are not currently supported. In some collective communication functions, such as MMPI_Reduce, the input matrices on all tasks must match in datatype, complexity, and number of elements (though their dimensions may differ).

 

The multitask toolbox has two classes of functions. Functions in the first class have the MM_ prefix and are generally higher level than the second class MMPI_ calls that mimic MPI functions quite closely.

 

Most of the first class functions require that their arguments be strings which name the matrices to be used. Because MATLAB requires that matrix input arguments appear on the right side of a function call and output matrices appear on the left, but MPI functions have all arguments on the right (except for the error return code in the C binding), arguments for second class functions have been redistributed according to their intent. In cases where an MPI argument has intent INOUT, it appears in both places in the corresponding MMPI call. Some arguments, like the source, tag, and communicator for MMPI_Recv, have been given default values, so they are optional. Some collective communication calls (MMPI_Bcast, MMPI_Scatter, etc.) have input arguments that are read only on the root process. If such arguments are given as matrices on other processes, the given matrices must exist, even though they aren't read. Alternatively, those arguments can be specified as strings which give the name of the matrix to be used. If the named matrix does not exist, no error occurs (except of course on a task where the matrix needs to be read - and isn't there). Similarly, an output argument that is written on only the root process (e.g. MMPI_Reduce, MMPI_Gather) will be set to an empty matrix on any other process where it is given in the function call.

 

Status objects returned by MPI calls such as MPI_Recv and MPI_Testany are returned in the "status" output argument. If the matrix transfer required multiple MPI calls (2 for complex matrices, 3 for real sparse matrices and 4 for complex sparse matrices), status is an array of structs. Their contents are as follows:

 

status(1): Real part
status(2): Complex part
status(3): Row indices for sparse matrix
status(4): Column indices for sparse matrix
Additional functions
Besides functions that mimic, as closely as possible, the behavior of the functions in the MPI-1 library, there are additional functions in the multitask toolbox:
MM_Eval - Causes a given string to be evaluated on selected processes
MM_Put - One-sided transfer of a matrix from the master to a worker
MM_Get - One-sided fetch of a matrix from a worker to the master
MM_Distribute - Spread a matrix from the master to the workers
MM_Collect - Gather a matrix from all workers to the master
MM_Shift - Shift the elements of a distributed matrix
MM_Plot - Visualize a distributed vector or matrix

 

Operation
The multitask toolbox can be used either interactively or in a standalone MPI/Pro job. The interactive mode is useful for development and debugging of parallel applications, but since few people would put a full, licensed, copy of an interactive application like MATLAB on each node of a cluster, an option is provided to use the MATLAB compiler to produce an application that can be run like any other MPI/Pro job in batch mode. Such applications require that only the free MATLAB runtime libraries be installed on each cluster node.

 

Starting multitasking in an interactive MATLAB session
The MM_Init command will start MATLAB on the nodes that are specified in the machines file in the current directory. If no machines file exists, one will be created with the name of the workstation in it.

All other MATLAB tasks started as part of the MPI job (including other MATLAB tasks on the same node as the master) will not accept input from their windows; all interactive input comes from the master via a private MPI communicator. Besides setting up MPI communication between the master and worker tasks, MM_Init runs the startup m-file on each worker and sets two global variables, MM_id and MM_procs, which contain the rank and size, respectively, of this task in MPI_COMM_WORLD. MM_Init can be called anytime because it checks to see whether MPI_Init has already been called. Specifically, MM_Init can be called to restore the global variables if they have been changed or deleted.

 

When multitasking starts up, it looks for a file named MM_run.m in the initial directory. It will be executed if found. The MM_run script will be executed before you get control. The variables it creates will not be in the interactive workspace on the master node (but the current workspace on each worker node will be populated).

 

Output
If MPI/Pro has been configured for interactive access on the machine where a task is running, a MATLAB window will open for each process that runs there. Only the window for the master process can be manipulated. All tasks create logs named CMTMlog.n ("n" is the rank of the process in MPI_COMM_WORLD) in the initial directory. MPI/Pro starts all other instances in the directory with the same name as the master. Output can be redirected to a different file by issuing the command diary filename. Subsequent output will go to filenameuntil the job ends or the command diary off is given.

 

There are two ways in which the multitask toolbox can be used for parallel programming. In the interactive mode, one or more machines that have full installations of MATLAB are used. In the batch mode, the compiled MATLAB program is run much as any other MPI parallel program. At present, no mechanism exists for mixing the two modes.

 

Since high performance computing is typically performed in a batch environment without GUI access to processes on the cluster nodes, it doesn't make a lot of sense to pay for a full installation of MATLAB on each cluster node. Instead, the CTC parallel clusters have just the MATLAB runtime libraries installed, enabling the execution of batch jobs that have been compiled with the multitask toolbox, but precluding interactive mode jobs. Thus, it is expected that parallel program development will be performed in interactive mode on machines where a full copy of MATLAB, including the multitask toolbox and MPI/Pro, have been installed, such as the the CTC Collaboratory. Then, when a properly running program has been produced, minor modifications to it can be made and it can be compiled there. Finally, production runs of the code can be done in batch mode on the clusters.

 

To get started then, one should use a workstation running Windows XP or 2003, MATLAB R2006a or later, the multitask toolbox, and MPI/Pro. Write the initial version of your program for executation in interactive mode.

 

Here is a "Hello, World" example for two copies of MATLABŪ using the multitask toolbox:

 

MM_Eval(1,'message=''Hello, World'';');
MM_Eval(1,'MMPI_Send(message,0,1);');
message=MMPI_Recv(1,1)
This program could be entered at the command prompt in the window for the master task after initializing the multitask toolbox with MM_Init(2), or it could be placed in a file named MM_run.m and it would be executed automatically when the multitask toolbox is initialized.

 

In general, to run a parallel program interactively, start with MM_Init() to start up the appropriate number of tasks, use one or more MM_Eval calls to run code on them (either using actual command strings or by referencing m-files) and call MM_Finalize to shut down the extra tasks and leave only the master MATLAB running. If your goal is ultimately to create a compiled code to run in batch mode, avoid any calls to other first class (MM_) functions because they will have to be removed before compilation.

 

 

If you have a parallel code that runs interactively, you will have to make a few changes to make it compilable. Whatever commands you issued at the console of the master MATLAB must now be coded into a main m-function that will be executed on all nodes. This main function (or a child) will also need to call MMPI_Init and MMPI_Finalize where MM_Init and MM_Finalize were before. To eliminate MM_Eval calls, put in if..end blocks based on the task's rank to cause functions to be evaluated on different tasks. See the Applications section for examples.

 

Before compiling the code, make sure that your path includes mpipro.lib. Then issue the command:
mcc -Glm mymain mpipro.lib

This will produce an executable named mymain.exe in the current directory. It should be runnable on any parallel cluster using a batch script of the kind described below.

 

Note to CTC users: If you compile in the Collaboratory, you must start Matlab from a command window, not from the Start menu or a desktop shortcut. Open a command window and type:

("" are necessary) to start MATLAB. The reason for this is that MATLAB opens a command shell to carry out its compilation and if a command shell is opened in a context that doesn't derive from a command shell, login.bat will be run, wiping out your path.

 

If you will not be running the code at CTC, you need to follow whatever procedures for running parallel jobs with MPI/Pro that your site has established.

 

The only way to run code that uses the multitask toolbox on the batch nodes at CTC is to compile it into a MPI/Pro application first. Once you have a compiled application, running it is not much different from running any other mpi application with one exception. Before calling mpirun for the compiled application, insert the following line in your batch script:

 

call setup_matlabruntime
this adds the directory containing the MATLAB runtime libraries to your path so your executable will be able to find them.

---

Installation
The multitask toolbox is installed by adding a new toolbox named "multitask" to the MATLAB toolbox directory. Additionally, a file is added to the path subdirectory of the local toolbox so that the multitask toolbox will be on the default path in MATLAB. The multitask toolbox contains a subdirectory "mpifun" with pairs of m-files and dll's containing the mexFunctions. This technique was chosen to provide online help in the MATLAB style.

 

---

Multitask functions
Class 1: Called only on master process
MM_Init
MM_Finalize
MM_Eval
MM_Put
MM_Get
MM_Bcast
MM_Distribute
MM_Collect
MM_Reduce
MM_Shift
MM_Plot

 

Class 2: Called by workers as well as the master (interactive) process. Only these functions can be used in a compiled program
MMPI_Abort
MMPI_Barrier
MMPI_Bcast
MMPI_Comm_dup
MMPI_Comm_free
MMPI_Comm_rank
MMPI_Comm_size
MMPI_Comm_split
MMPI_Finalize
MMPI_Gather
MMPI_Init
MMPI_Iprobe
MMPI_Irecv
MMPI_Recv
MMPI_Reduce
MMPI_Scatter
MMPI_Scatterv
MMPI_Send
MMPI_Testany
MMPI_Wtime
 
Miscellaneous Functions

 

CMTM_Timer
MM_Demo

 

This software was developed by Dr. John A. Zollweg and Dr. Arun Verma
Last updated on July 12, 2006

 

 

Interactive Applications
There are many interactive mode applications in the demo package. See MM_Demo for an overview and how to run them. The source code for them is in the toolbox\multitask\demo directory.

 

Compiled Applications
The compiler imposes a number of restrictions on what code is suitable for a compiled program. Thus, all first class (MM_) functions are not allowed. The runtime environment isn't available at compile time, so input, eval, and exist cannot be used to reference workspace variables. The alternative string format for the names of inputs to functions like MMPI_Bcast cannot be used in compiled code.

 

The usage scenario for compiled programs is also different. For interactive programs, we start with MATLAB open and say MM_Init() at the command prompt to start additional tasks. Compiled codes are started using the mpirun command. No copy of MATLAB is actually started during a compiled program.

 

Here are two skeletal versions of a master/worker application that uses a programming paradigm that works very well with the multitask toolbox. Many problems that involve massive amounts of computing consist of relatively small quanta of work that can be described fairly concisely, don't interact with each other, and take variable amounts of time. Distributing these pieces of work over a number of parallel tasks in an arbitrary way may result in some tasks finishing their work much sooner than others. This is wasteful of parallel resources. In this algorithm, the master is responsible for parcelling out the work to the worker tasks and for collecting and collating the results. Each worker task gets one piece of work at a time and receives a new one upon completion of the previous one. Those workers that need a longer-running quanta will get fewer and those that get easy ones will get many. As long as the pieces of work that require the longest time don't get sent out near the end, all workers will finish about the same time and little parallel resource will be wasted.

 

The interactive version shows how it would be written to use the multitask toolbox in interactive mode. The details of the computation have been removed for clarity, leaving behind the framework of multitask toolbox calls and organizational matrices.

 

The compiled version shows how the interactive code needs to be modified to make it suitable for compilation into a code that can be run in batch mode.

---

CMTM Related links

 

 

Workshop materials and Online Tutorials

---