Multiprocessing package - tensorplay.multiprocessing

Strategy management

CUDA in multiprocessing

Cross-process sharing is implemented for CPU tensors only: pickling a tensor on any other device raises a RuntimeError, so move tensors to the CPU before sending them through a queue, and move them back on the device in the receiving process.

Using CUDA inside subprocesses additionally requires the spawn or forkserver start method. The runtime cannot be re-initialized in a process that was forked after initialization; such a child fails with RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method.

Sharing strategies

This section provides a brief overview into how different sharing strategies work. They apply to CPU tensors only; tensors on other devices cannot be shared between processes.

Spawning subprocesses

Spawning a number of subprocesses to perform some function can be done by creating Process instances and calling join to wait for their completion. This approach works fine when dealing with a single subprocess but presents potential issues when dealing with multiple processes. Namely, joining processes sequentially implies they will terminate sequentially. If they don’t, and the first process does not terminate, the process termination will go unnoticed. Also, there are no native facilities for error propagation. The spawn function below addresses these concerns and takes care of error propagation, out of order termination, and will actively terminate processes upon detecting an error in one of them. A ProcessContext is returned by spawn when called with join=False.

tensorplay.multiprocessing.pool