distributed_data_parallel.py and run.sh show an example using Amp with
apex.parallel.DistributedDataParallel or
torch.nn.parallel.DistributedDataParallel
and the Pytorch multiprocess launcher script,
torch.distributed.launch.
The use of Amp with DistributedDataParallel does not need to change from ordinary
single-process use. The only gotcha is that wrapping your model with DistributedDataParallel must
come after the call to amp.initialize. Test via
bash run.sh
This is intended purely as an instructional example, not a performance showcase.