Do you know of any google TPU models that use pipeline parallelism? From my understanding, you just FSDP/TP within a TPU pod with ICI then do replicated data parallel between pods with DCN. See the PALM paper for not info about their parallelism strategies
Not sure I agree on the last section. I believe Google has better pipeline parallelism libraries that run much more efficiently internally.
Do you know of any google TPU models that use pipeline parallelism? From my understanding, you just FSDP/TP within a TPU pod with ICI then do replicated data parallel between pods with DCN. See the PALM paper for not info about their parallelism strategies
They have jax and can implement pipeline. GTC2023 has a very interesting presentation on it https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s51800/
I don't think there is any publicly disclosed info tho.
Yes. Ppl use pipeline parallelism in Jax with GPUs