Ask Siri play songs only from downloaded music, not Apple Music

I do not subscribe to Apple Music, nor will I ever. All my music is stored on my Mac and synced to my iPhone when I connect to it.

When I'm driving, I'll ask Siri to play a song from my iPhone. Most of the time it at least finds the song and starts playing it, but it always wastes time trying to get it from Apple Music, responding with "there was a problem connecting to Apple Music" (duh), and then it finally starts the song. Sometimes this just doesn't work, even when it's the same song I've requested before. It just tries connecting to Apple Music, fails to because I don't subscribe to that thing, then just fails to play the song that's *right there on my device*.

Is there any way to make this work correctly? Why is it being so dumb about this?