I have this problem, and I think it's because my wrist is too boney, so the sensors can't get a clean reading because of all the external light leaking in. What I've found helps (if this is your problem) is using something that blocks the light, like the Action Band by Twelve South. It's basically a sweatband, but with a watch mount molded in. You can also just put a sweatband over your watch, or cut the top off of a thick sock and put that on your wrist over the watch, but then you can't see the face.
For other exercises it can help to push the watch up your arm a little (one to one-and-a-half watch width higher than you usually wear it), that's what I do for running. But for rowing your arm moves in a way that makes that not really work.
A bonus with the Action Band is that if you use your watch for sleep tracking, it's really comfortable for sleeping.