How to exit Photos albums without saving to Memories?

Are you viewing the albums as a slideshow, while managing the album? As soon as you are playing an album as a slideshow, the Photos.app is creating a memory from it and you will have to deal with the memories. All slideshows are now automatically saved as memories: Make and play slideshows in Photos on iPad - Apple Support (CA)

To update the albums, just open the album and select the photos you want to remove or to add to a different album, but do not play the album as a slideshow. Add, delete, and find photos in albums on iPad - Apple Support (CA)

I mostly don't look at those Memories, and so, like you, I try not to make new Photos' Memories. But, even so, they don't take up any storage space. But you can delete those things using the More (•••) menu …

That way you don't have to be bothered by them if you make some you don't want.

"I watch the video/movie and usually change the song to one from my Playlist."

But when you are using this, you are playing a slideshow, and are creating a new memory. There is no longer any way in iOS/iPadOS to view the items in an album automatically as a film without creating a memory. As soon as you tap the "film" tab and change the music or modify the photos, you will be asked, if you want to save the new memory.

That should read "help or fix your problem?"