I've been supporting pytorch + cuda just using 'python3 -m venv foo' and pip (+ pip-compile) for the last year and it works - not super fast, but I prefer using the python.org blessed tooling.

Torch, nvidia, and triton site pkgs are quite a bit bigger on Ubuntu since I guess nvidia gpus aren't available on mac.