Parallel programs collective communication

We now reexamine message passing as it pertains to software development and the interface of application software and library software. Compiler-managed parallelism is not yet ready for prime time. This means that efficient parallel programs are usually coded by hand, typically using point-to-point message passing and simple collective communications. There are many problems associated with this approach. 

Kiclmann, T. Hofman, R.H.F., Bal, H.E., Plaat, A., Bhoedjang, R.A.F. (1993) "Magpie MPFs collective communication operations via clustered wide area systems In Proc. Symposium on Principles and Practice of Parallel Programming, Sand Diego. 

The values for these machine-specific parameters can be somewhat dependent on the application. For example, the flop rate can vary significantly depending on the type of operations performed. The accuracy of a performance model may be improved by using values for the machine-specific parameters that are obtained for the type of application in question, and the use of such empirical data can also simplify performance modeling. Thus, if specific, well-defined types of operations are to be performed in a parallel program (for instance, certain collective communication operations or specific computational tasks), simple test programs using these types of operations can be written to provide the appropriate values for the pertinent performance parameters. We will show examples of the determination of application specific values for a, and y in section 5.3.2. 

Collective communication operations can reduce the scalability of a parallel program by introducing a communication bottleneck. Consider, for example, an algorithm requiring floating point operations and using... 

---