In the U.S. at least Aspergillus contamination is relatively rare. The USDA and FDA monitor all cereals, grains, nuts, legumes, etc. that could potentially become contaminated with the fungus and these products aren't supposed to be processed or sold unless they are below the specified limit of contamination. It's also good to note that MANY different foods can harbor the fungus, and the reason why it surfaces is because of improper storage which encourages the fungi's growth. Peanuts aren't the only foods that you should be careful with, and they also probably shouldn't necessarily be vilified quite as much as they are.
This is a good article I found a while back, with info and advice from my avian vet and exotic avian husbandry professor at Cornell.
Overall I'd say definitely watch your birds, but you really don't have too much to worry about.