Logic Pro X Midi Clock Start issue

Currently using Logic Pro 11.1 and Sequoia 15.1 but have had this same issue on my old computer running previous versions as well. It seems to be project and even save file specific because reverting to older versions of the same project will sometimes (temporarily at least) fix it.

Basically at a certain point in some bigger project I'm encountering a bug where the audio buffer is out of sync with the Midi clock so when I start recording or even just playback my external Midi is out of sync from the audio buffer. Basically as soon as I press play the Midi clock starts playing but due to the buffer lag the audio starts playing later so the Midi clock is substantially early and completely unusable if I want to keep adding to that project.

Is there a certain temporary file that is being created that I need to just delete and recreate or something? I've already tried resetting the MIDI drivers and that doesnt work. My MIDI interface is a Unitor 8 which is relatively older but has up to date drivers from 2022. Again this only happens on specific saves of specific projects in Logic (I also use Ableton sometimes and havent had this issue) so I assume it is something on the Logic side like a project specific file getting corrupted or something. Onlu workaround I have managed so far is exporting as one mixdown in the current state and creating a new project to add to then reimporting as needed to the original but it is incredibly frustrating as this keeps happening semi regularly and seemingly at random with only certain project files.