Is it possible that one of your family members isn't as diligent about voluntary, non forced, interactions with Crouton as you? Is your behaviour different with Crouton when your family is around? Is the 1:1 routine you have with Crouton changed (interfered with) when your family is around? Are they around more, or are you around less?
Those are all rhetorical, but are offered as examples of the dynamics to consider with the first part of what you're describing.
Try to think of things from Crouton's perspective.
Determining a potential trigger, would determine the right course of action (example - family, or family member, needs to be trained, or they need to not interact with Crouton).
... try to get him off, and he will gently bite me.
For this part, it may be related to the aggression, but it may simply be Crouton giving you gentle (appropriate) feedback that he doesn't really want to go back in his cage. I wouldn't consider that problematic by itself, unless it escalates, but you could try inserting another activity between your shoulder and the cage (so it's not such a big jump between where he wants to be - your shoulder - and where he doesn't want to be - the cage). Off your shoulder for a treat first, or neutral perch, for example, and then the cage.
Overall, I'd focus on positive interactions where Crouton isn't displaying aggression, to ensure you and he maintain your healthy relationship, and that may offer clues while troubleshooting his relationship with your family (and theirs with him).