Why is my album artwork missing in Finder?

1. As originally suggested, I'm recommending you delete the entire Documents folder, which would include the three .sqlite files (probably one database and its supporting files) along with the artwork folder. I neglected to mention that you should either reboot the computer or restart the AMPArtworkAgent process. Details on how to do that are covered in this post Apple Music Big Sur Album Artwork Cache Rebuild? - Apple Support Communities if you want to go that route.
2. Yes, that is is what should happen. The purpose of a cache is to store data (or perhaps a low resolution impression of it) that takes some effort to fetch from elsewhere so that the cached data can be used more efficiently at a later date. If the item you want isn't in the cache then you go to the original source. If, for whatever reason, the cache returns bad information, and cannot self-repair, then resetting the cache should be a relatively pain-free process. People clear their browser caches all the time as a troubleshooting step.
3. I mentioned the show/hide hidden files and folders command because you, or anyone else reading this thread, might not know how to locate ~/Library/ otherwise.

tt2

You could try removing the artwork cache which should force Music to rebuild it. The location should be ~/Library/Containers/com.apple.AMPArtworkAgent/Data/Documents where ~ is your user's home folder.

Show hidden files and folders with Shift+Cmd+. to aid your navigation.

tt2

My understanding is that you delete the entire Documents folder along with all it contains. The cache may support more than just Music which might be why there are multiple sqlite files and images you don't recognize. The fact that you see images that are no longer relevant to your library would suggest a potential lack of garbage collection, another reason a clear out might help things.

tt2