If I have my photo library backed up via with "optimize photos" to iCloud can I transfer to a smaller new device?

Theoretically, shouldn't the photos not count towards total device storage since optimize photos converts them to low res placeholders?  

Impossible to have a file of zero bytes. Optimized field are small, but they do exist and have to be accounted for in local storage.

How can I get this content transferred to my new phone? I don't want to lose all my app data or manually sort 10k images.

Not sure I understand your question to be honest.

How photos are stored on device A has nothing to do with how they'll be stored on device B. They are independent of each other.

Both devices will sync with iCloud, where the full size photos are stored. Device A does not sync with device B, They each sync to iCloud and do what they have to do based on their local storage.