Can I use iMac (Retina 5K, 27-inch, 2017) as display for Mac mini M1

Sorry, no. No modern Mac supports Target Display mode. See the link below.

Use your iMac as a display with target display mode - Apple Support

+1, sorry no that will not work. If the MBP needs an external display please buy one which you will find less expensive (no expensive cables needed), more reliable than Target Display Mode (which died in 2014) ever was and much easier to setup. In short, iMacs are computers and were never designed to be external displays for other devices.