A parallel implementation of an implicit finite element formulation for
incompressible fluids on a distributed-memory massively parallel computer is
presented. The dominant issue that distinguishes the implementation of finite
element problems on distributed-memory computers from that on traditional
shared-memory scalar or vector computers is the distribution of data (and hence
workload) to the processors and the nonuniform memory hierarchy associated with
the processors, particularly the nonuniform costs associated with on-processor
and off-processor memory references. Accessing data stored in a remote processor
requires computing resources an order of magnitude greater than accessing data
locally in a processor.  This distribution of data motivates the development of
alternatives to traditional algorithms and data structures designed for
shared-memory computers, which must now account for distributed-memory
architectures. Data structures as well as data decomposition and data
communication algorithms designed for distributed-memory computers are presented
in the context of high level language constructs from High Performance Fortran.
The discussion relies primarily on abstract features of the hardware and software
environment and should be applicable, in principle, to a variety of
distributed-memory systems. The actual implementation is carried out on a
Connection Machine CM-5 system with high performance communication functions.