Communication limit at downtime

I have iphone with ios 16.3. Under Screen Time communication limits I have an option for limits during downtime and limit during screen time. I would like to limit who my child can communicate with late at night during downtime (our downtime is set 11pm-6am at the moment) so I set the downtime communication limit to specific contacts only. Now whenever my kid runs out of screen time during the day all contacts are locked as if it were downtime. Is this a program error or user error? Any fixes.