The control of quantum systems is of fundamental scientific interest and promises powerful applications and technologies. Impressive progress has been achieved in isolating quantum systems from the environment and coherently controlling their dynamics, as demonstrated by the creation and manipulation of entanglement in various physical systems. However, for open quantum systems, engineering the dynamics of many particles by a controlled coupling to an environment remains largely unexplored. Here we realize an experimental toolbox for simulating an open quantum system with up to five quantum bits (qubits). Using a quantum computing architecture with trapped ions, we combine multi-qubit gates with optical pumping to implement coherent operations and dissipative processes. We illustrate our ability to engineer the open-system dynamics through the dissipative preparation of entangled states, the simulation of coherent many-body spin interactions, and the quantum non-demolition measurement of multi-qubit observables. By adding controlled dissipation to coherent operations, this work offers novel prospects for open-system quantum simulation and computation.