Loading Mail attachments takes (almost) forever

When I receive Mail with even very small attachments Apple Mail on both iPad and iPhone takes ages to download these attachments. Especially .pdf attachments take 20-25 minutes to download a 1 MB attachment.

Any settings I could change to speed that up?