how do you replace the battery in the Apple 4k TV remote?

You don't. The rechargeable battery inside is not replaceable. Simply charge with a lightning cable.

https://support.apple.com/en-us/HT201531

You don't: you recharge it using the lead provided from a standard charger. It takes some hours to do a full charge.