Just to be clear, your Wallet does not save any of your account numbers on those cards. What is more likely is that the physical card was used on a terminal that had a skimmer installed by a scammer to read your account details. That is very common at gas stations and ATM's where the scammers can tamper with the terminals overnight with little interruption.
The bank of the issuing card will be able to tell you if the card numbers were used on the fraudulent purchase, which would clearly indicate it was not an issue with your Wallet, or if the bank approved your card to be added to another device, which would also indicate they knew your account number.
If there was a purchase made on your device, then a person would need to have the Passcode to authorize the charge, so I would also change that if it is known by someone else.