I take it that they still have the Dam of the pup too?
So you are saying the pup is nervous of going outside in the garden in it's own home too? Very strange. Unfortunately the idea of you taking him for a while could be a mistake if he is afraid, taking him from the people he is learning to trust may well frighten him even more.

At a guess I would say that when the litter was produced the pup was just not introduced to the outdoors. I wonder where the litter was reared? Pups should be experiencing the outdoors with the dam and other pups from 5 weeks onwards.

Maybe due to the winter and it being their first litter the breeders did not know to take the pups into the garden for toilet training and exploration and confidence building, so now the pup has no confidence at all outside and does not count it as part of his territory. This you
would expect with some pups on walks
outside their territory of the house and garden; pups can be worried and may not have confidence when first leaving the home as everything smells different and they can pick up scents and sounds they are not used to, to be nervous here is understandable, but in his own garden where he was reared he should not feel like this at all.
Nevertheless, IMO the dam is the key here, they need to get the pup to go outside with it's mother, she needs to go on outdoor walks with him if not already too to give him confidence, it should be natural for him to follow her about and stay close to her, they also need to do the same in their own garden. The owners need to start spending more time in the garden involving the pup in play, taking his toys even his bed out there and just sitting out there ignoring the pup until he comes out to visit, he needs to feel it is safe.
Maybe with the dam with them as well they could all meet up with you and socialise with your dog, that would be good for the pup too he needs a lot of meet and greet, are they taking him to all sorts of places to hear noises cars, shops, people etc?
He's only 12 weeks, he will come along, maybe he just didn't get the start he should have...............