I prefer to attach my Card reader to the Mac mini instead of the display. Not sure if it makes any difference it's only my feeling. I believe the important thing is you select a reader which is at least USB3 and not USB 2.
Many manufacturers claim to have readers with a faster read speed. This can be true or not: try to find a reputable internet page to read a comparison. Another point is the kind of CARD you have and its speed, to avoid the use of a reading device with speed which are not supported by the card or the contrary. One of the main difference is between the type UHS I and UHS II - the first has 1 single line of contact pins while the second has 2 lines of pins. This is also depends by the model of camera you use.
Hope this helps