Control and Notification Center not working

It has not been even a month I got my new Apple Watch SE 44mm. It is updated to version 7.3.1 (latest). Several times during the day, I am neither able to swipe up the control center nor swipe down the notification center. I already unpaired the apple watch and added it as a new watch i.e everything was erased. The only TEMPORARY solution is to restart the watch SEVERAL times in a day.

Did not expect such a problem from Apple's side, my research tells me I am not the only one facing this issue. Please provide a permanent solution. Do not tell me to unpair, reset, etc because I have already tried that. Thank you.