Mac Studio M2 Ultra max Compressor Instances?

Also, to add to what Luis said, not all codecs can use all CPUs fully to transcode. Not all apps can use all CPU cores at once, depending on the tasks being done. So you'll never see all of your CPU cores being fully used for transcoding video.

And from the user manual for Compressor:

"The number of available Compressor instances is determined by your computer’s cores and memory. After meeting the minimum system requirement (four cores and 2 GB of memory), you can add one additional instance for every additional four cores and 2 GB of memory."

Create additional instances of Compressor - Apple Support

Remember that a CPU core is the same thing be it Intel or M1, only the way it's assimilated into the chip architecture is different, but RAM and CPU cores are basically the same animal in every computer, Intel or Mx.

While I have not seen any specific reference to this in the documentation, I think that it is roughly on par with other examples of macs. My rule of thumb is one instance per about 5 cores.

On an M1 Max Mac Studio (10 cores) it allows for 1 additional instance (total of 2).

On my 2019 16” MBP (8 physical cores that appear to the OS as 16, due to hyperthreading) I get up to 3 additional instances (total 4).

I expect that the M2 could handle more load, but probably there is some algorithm that determines how many are allowed. You may want to use feedback on that.