Walkie Talkie not connecting

When I try to make a walkie talkie call, I tap the contacts name and a message states "you invited XXXX to be walkie-talkie friends". I now have two options, 'invite again', or 'remove'. If I tap 'invite again' my watch will almost immediately show the contacts picture again with the text 'call ended'. I've rebooted both watches, removed and then re-entered the contact's info. Facetime seems to work fine on both of our iPhones. I'm stumped!