Location Services can't find me

Almost every day I have to stop and start location services on my M1 Pro because macOS Sonoma doesn't know where I am. This has been extremely frustrating when I'm trying to get directions using maps and web sites don't know where to find me for ordering etc.

I typically have to turn off Location Services, wait 10-15 seconds, then turn it back on and them both me and my M1 know where I am.

Does anybody know why this might be happening and how I can get Location Services to remember my location?