If two jpegs have the same resolution in pixels but one is larger in file size, why doesn't the Photos Duplicate Merger Pick the larger one?

I think the Photos Duplicate Merger is picking the wrong files for me, but I'm not 100% sure because their UI is not very transparent as to what they are doing. Here's my reasoning, and maybe someone out there can explain.

I've been cleaning up my Photos library and have a lot of duplicates. I got to the Duplicates menu and see lots of photos where the app wants to merge them. Often there will be two copies of the same photo, with the file type on both being JPEG, the metadata is identical, the pixel resolution the same, everything, except one has a larger file size (ex: 4.4 MB vs 9.8 MB).

When I tell the app to merge these photos, it says it's going to "keep one version of the duplicates that combines the highest quality and relevant data". Yet when I double check which copy was kept, it's ALWAYS the one with the smaller file size.

Why would this be? Because they are both jpegs, I would assume that reason one has a smaller file size is that it was likely compressed more, and thus it would NOT be the one with the highest quality since jpeg has lossy compression. So why then is the photos checker picking that smaller file as the the photo that is kept?

Because this happens all the time for me, I'm wary of using the merge tool as I suspect it's keeping the lower quality images.