Skip to content

Introduction

The distributed memory paradigm

In a distributed memory architecture, each processing element has access only to its own local memory or address space. Unlike shared-memory systems, there is no globally shared address space that all processors can read from or write to. This means that every process operates independently, maintaining its own set of data in a private section of memory.

Distributed memory Distributed memory

The main consequence of this design is that processes cannot directly access one another’s data. If one process requires information stored in another’s memory, it must explicitly send a message requesting that data and then wait for a response. Communication and data exchange therefore occur explicitly through message passing over a communication network. This explicit communication model gives programmers precise control over data movement and synchronization but also requires careful design to manage communication overhead and ensure efficiency.

Distributed memory systems are commonly found in large-scale parallel computers, clusters, and supercomputers, where scalability and modularity are critical. The design allows the system to grow by simply adding more nodes, each with its own processor and memory, interconnected by a high-speed network.

Because distributed memory systems rely on explicit communication between processes, there must be a reliable and efficient way to send and receive data across the network. This need led to the development of message-passing libraries. These libraries provide a set of functions or routines that processes can use to exchange information, synchronize computation, and coordinate their actions.

An effective message-passing library must satisfy several important goals. It must be flexible, supporting a range of communication patterns, from simple point-to-point exchanges to complex collective operations involving many processes. It must also be efficient, minimizing the overhead associated with communication and ensuring that the cost of data transfer does not dominate the total runtime. Portability is another crucial feature: programs written using a standard message-passing library should be able to run on different types of distributed-memory hardware without requiring changes to the source code. Finally, a good library should hide low-level details of the underlying hardware and software communication mechanisms from the user, allowing programmers to focus on algorithm design rather than system-specific networking code.

In essence, a message-passing library acts as a high-level abstraction that simplifies the complex task of coordinating independent processes in a distributed environment.

The Message Passing Interface (MPI)

The Message Passing Interface, or MPI, was created as a standardized solution to the problem of portable and efficient parallel programming on distributed-memory systems. Rather than being a specific software package, MPI is a standard specification that defines how message passing should work; what functions should be available and how they behave. Multiple implementations of this standard exist, but all conform to the same rules, ensuring that an MPI program written on one system will run correctly on another.

MPI was designed with several guiding principles in mind:

  • Portability: an MPI program should be able to run across a wide range of platforms without modification.
  • Efficiency and scalability: MPI provides mechanisms that allow communication to overlap with computation and avoid unnecessary copying of data, enabling applications to scale effectively to thousands or even millions of processes.
  • Flexibility: MPI supports different communication models, including point-to-point communication between pairs of processes, collective communication involving groups of processes, and one-sided communication (also called remote memory access), which allows a process to directly read from or write to another process’s memory.
  • Standardization: defining clear syntax and semantics for all of its operations and providing bindings for multiple programming languages, most notably C and Fortran.

Through these design goals, MPI became the foundation for most large-scale scientific and engineering applications running on distributed-memory supercomputers.

History

Before MPI, the landscape of message-passing systems was fragmented. A number of libraries and frameworks were developed independently, such as Express (ParaSoft), P4 (Argonne National Laboratory), PARMACS (GMD), PVM (Oak Ridge National Laboratory), NX/2 (Intel), and Vertex (Cornell). Each had its own interface and communication model, which made software portability between systems difficult or impossible.

By the early 1990s, it was clear that a common, standardized approach was needed. In April 1992, a workshop on "Standards for Message Passing in a Distributed Memory Environment" launched the effort to create such a standard. Over the next year, researchers and developers from academia, industry, and national laboratories collaborated to draft and refine what would become the first MPI specification. The initial draft of MPI-1 was completed in November 1992, with further revisions continuing through early 1993. The draft standard was presented publicly at the Supercomputing 93 conference, and in June 1994, version 1.0 of the MPI standard was officially released.

Since its initial release, MPI has continued to evolve to meet the demands of modern high-performance computing. Today, it remains the most widely used programming paradigm for distributed-memory systems. Versions 3 and 4 introduced major improvements in collective communication, remote memory access, and tools for better interoperability with modern programming languages and hardware architectures. The current officially approved version is MPI-4.1, which was finalized in November 2023.

As of June 2025, MPI-5.0 has been approved but not yet implemented, marking another milestone in the standard’s development. One of the most notable additions in MPI-5.0 is the introduction of a standard Application Binary Interface (ABI), which simplifies interoperability between different MPI implementations.

Standard vs. Implementation

It is important to emphasize that MPI is a standard, not a single implementation. The MPI standard defines a specification, i.e., a detailed set of rules, functions, and behaviors that all compliant implementations must follow. This guarantees that programs written in MPI are portable and interoperable across systems.

Several independent implementations of the MPI standard exist, each developed and optimized for specific platforms or performance needs. Among the most popular are OpenMPI, MPICH, and Intel MPI, as well as vendor-specific versions such as those provided by Cray and IBM. These implementations may differ internally in how they handle communication and memory management, but they all present the same MPI interface to the user.

This separation between standard and implementation is what allows MPI to serve as a long-lived and stable foundation for parallel computing. A program written in MPI today can be recompiled and run on future systems with minimal or no modification, preserving both performance and portability across generations of computing architectures.