Drag and drop while scrolling.

As an alternative, you can always open a second Finder window showing the same folder, and drag from one tho the other, avoiding that long scrolling.

This may be related to my problem, with a MacBook Pro Trackpad.

You can drag a file (or selection of multiple files) up or down within a column view, and when you hit the top (or bottom) it starts scrolling.

My problem is that if there are lots of folders, they immediately whip by too fast to select one near the middle to drop it into. Usually you can control the scroll speed by how far beyond the edge you drag the cursor, but not in the Finder. It goes at full speed or nothing. Even slowing down the scrolling speed to minimum in Accessibility>Pointer Control>Mouse and Trackpad>Trackpad Options doesn't help.

I usually have to resort to opening a separate tab or new Finder window, line up the target folder, and drag the file across to it from the other window/tab.

Another problem is often the cursor selects the filename instead of dragging the file. Timing is everything.

I don't have a Magic Mouse, but I often do similar things with my MBP trackpad.

Since the scrolling gesture in the Magic Mouse is similar (just drag with one finger vs drag with two fingers in the trackpad), I expect it will also work similarly in this case: click and start to drag; keep pressing the button and use another finger to do the scrolling.