You are right; toweling a budgie is not a good way to gain their trust. It really isn't good for any bird but especially for budgies since they are, in my experience, more sensitive and flighty than most other common pet parrots. You really have to earn their trust. It is well worth it; once you do gain their trust, they are very sweet and friendly little guys. But they do take more patience at first.
One thing they usually love is to be talked to. They often also like toys with small parts they can move around, like bells, toys with sliding beads, or those small whiffle balls with bells inside to roll around. I would try to spend some time every day just talking to him. Say his name, tell him he's a pretty bird (there's a reason budgies are often taught to say pretty bird first; it's actually a bit like some of their natural vocalizations and one of the easiest things for them to learn to say), read a book out loud to him or tell him what you are doing. Try to spend time near him doing some sort of calm activity that he doesn't find threatening - even just watching TV, if it's not super loud.
Try reaching in and playing with his toys a little. Ring a bell on a toy, or slide a few beads around. Then back off and see if he'd interested in the toy. If he doesn't seem scared, stick your face near him when you talk to him sometimes - they seem to really enjoy being near your face when you talk.
Try offering your finger to step up - you can try bribing him with a treat, but don't resort to grabbing him. If he flutters out of his cage and needs you to pick him up, I would try just picking him up and talking to him a little like you would if he was in his cage, and see how he reacts. If he looks really scared and just wants to get back to his cage, then take him back. But after awhile, you will probably see that he looks less scared and starts to show signs of relaxing, like preening and shaking his feathers into place, ect. Once he's looking a little more relaxed, I would hold him for a little bit, away from his cage, and use the time to just sit quietly with him and talk to him for a few minutes before you take him back to his cage.
Mostly you just want to appear as non threatening as you can while spending time with him. It sounds like you have a good start in that he will step up to let you return him to his cage. Grabbing him will only harm trust. With budgies, it really is all about trust.
It can take a month or two, or even more to get a strong bond. Once you have that bond, then they do become very affectionate and trusting. I remember when I got my first budgie, I was used to my conure and was not expecting it to take a long to bond to my budgie as it did. I was rather discouraged a few times since he was tame but didn't seem to want to interact. But I've learned since then that they just take a little more patience and work at first, and there's something really special about a bond with a budgie, I can't describe it, but it's well worth the time it takes to bond.
