horovod tensorflow 分布式多gpu

概念 rank is your index within the entire ring, local_rank is your index within your node. For example, you have 4 nodes and 4 GPUs each node, so you spawn 16 workers. Every worker will have a rank [0,

>>阅读原文<<