Howdy Esqueeks,
Thanks for posting in the Apple Support Communities.
We understand you're looking to limit communication to specific contacts only during the scheduled downtime, right?
If we have that correct, let's confirm the same steps are taken as shown below. This sounds like there may be a setting that could be conflicting with another, so we'll be able to rule that out by checking these steps. For more details on the steps and using Screen Time, take a look at the related resource: Set up Screen Time for a family member on iPhone - Apple Support
Set communication limits on a family member’s device
You can block incoming and outgoing communication on your family member’s device—including phone calls, FaceTime calls, and messages—from specific contacts, either at all times or during certain periods.
1. If you haven’t already turned on Contacts in iCloud on your family member’s device, go to Settings 
> [child’s name] > iCloud, then turn on Contacts.
Note: You can only manage your family member’s communication if they’re using Contacts in iCloud.
2. On your family member’s device, go to Settings > Screen Time.
3. If you haven’t already turned on Screen Time, tap Turn On Screen Time, tap Continue, then tap This is My Child’s iPhone.
4. Tap Communication Limits, then do any of the following:
• Limit communication at any time: Tap During Screen Time, then select Contacts Only, Contacts & Groups with at Least One Contact, or Everyone.
• Limit communication during downtime: Tap During Downtime. The option you selected for During Screen Time is already set here. You can change this setting to Specific Contacts.
If you select Specific Contacts, then tap either Choose From My Contacts or Add New Contact to select people you want to allow communication with during downtime.
• Manage a child’s contacts: If you’re using Family Sharing, you can view, edit, add, or delete your child’s contacts. Tap Manage [child’s name] Contacts.
If your child already has contacts in iCloud, they receive a notification on their device asking them to approve the request to manage them. If they don’t have contacts, they don’t get a notification and you can immediately add contacts.
When you manage your child’s contacts, a new row appears beneath Manage [child’s name] Contacts to show how many contacts your child has. Tap the row to view and edit the contacts.
• Allow contact editing: Tap Allow Contact Editing to turn off this option and prevent your child from editing their contacts.
Turning off contact editing and limiting communication at any time to Contacts Only is a good way to control who your child can communicate with and when they can be contacted.
If someone who’s currently blocked by the Communication Limit settings tries to call your family member (by phone or FaceTime), or send them a message, their communication won’t go through.
If your family member tries to call or send a message to someone who’s currently blocked by the Communication Limit settings, the recipient’s name or number appears in red with an hourglass icon, and the communication won’t go through. If the limit applies only to downtime, your family member receives a Time Limit message and can resume communication with the contact when downtime is over.
To allow your family member to communicate with contacts who are blocked by the Communication Limit settings, change the settings by following the steps above.
Be sure you see the correct contacts listed as being available to allow communication with during downtime, and confirm that the latest iOS version is installed on the devices in use. Let us know if you need additional assistance, and confirm which steps you've taken so far. We'll gain a better understanding of what could be occurring to provide the best next steps.
Have a great day!
