What is the camera model?
Have you tried reaching out to the camera manufacturer?
The modern way is to support HLS (HTTP Live Streaming) which has much better support in modern apps.
Like: https://192.168.1.77:554/hls/camera1/master.m3u8
HLS doesn’t do credentials in the URL afaik, so credentials should be handled through the player app, if needed.
rtsp://admin:password@192.168.1.77:554
I realize it’s just an example, but that should probably also have a stream identifier, like: rtsp://admin:password@192.168.1.77:554/media/video2 , depending on the make and model camera, as the details will be different for various brands and models and your camera settings.
The are various RTSP apps in the Apple TV App Store. VLC isn’t the only game in town. Do try some others. Simply search for “RTSP”.