I would definitely talk to him and interact with him. He may not want to be handled yet but you sitting quietly with him and talking to him will likely be comforting to him. As was suggested, you need to try to follow his lead. Some birds want to be held and want to play right away. Others need a couple days in their cage just getting used to things. If your baby is super scared, you can try covering the cage on the 3 sides just leaving the front open, so he's not on alert for danger from all three sides. And silence can be scary too. Especially if he's used to other birds and activity. So some soft music or TV during the day may help too.