You may recall that there is something of a debate going on in America about gun control. Imagine that this couldn’t even be a debate. Everybody had laser eyes and a third arm with rocket-propelled knife fingers. You could, conceivably, rip those eyes out and amputate that arm, but a lot of people would complain about that and you’d have a much harder time justifying it. Especially since this wouldn’t be seen as an abnormal state of affairs.
Do wizards rely on wands? Then by taking those wands away you’re making them into non-wizards, and they probably won’t like that. Can they cast magic however they please? Then good luck trying to prevent them from doing so. Even if there’s way for it, don’t expect the populace to support you stealing their magic.
When magic is on the table there is nothing that you can do to keep one person from firing a Bad Spell at another person. If defensive magic is easier and more effective than offensive magic then you can make shields that make that a non-problem (most of the time). If somebody does get hurt, then you can bring down social consequences, whether that’s imprisonment, shunning, or something else.
But, short of taking magic away from a wizard, making zem no-longer-a-wizard-in-the-first-place, wizards will always have lethal power at their fingertips, and their society will have to live with this.
And the adults are younger than you might expect. This is because of the previous point. Remember that wizards, like non-magical folk, did not launch into existence with a fully-formed Twenty-First Century outlook on things. For most of history our ancestors considered peace to be a thing that happened between wars rather than the default condition. Wizards will go through the same thing, but because every wizard is a living weapon they will have to treat it in a slightly different way.
Here is the unfortunate truth of the matter: In pre-modern societies, the idea of a “noncombatant” was always a little fuzzy, but if you had a weapon and could use it then you were definitely a combatant and lawful prey, so to speak. And as we discussed a moment ago, every adult is armed. Moreover, if children start practicing magic at the age of eleven (for example) then every eleven-year-old is a potential threat.
The only way that you can get around this is if wizards don’t start learning magic until later in life. The idea that children should be spared in war only developed because children aren’t capable of slitting your throat with a couple of syllables and a wave of their hands. Change that, and you’re going to change what society thinks of young humans.
But children are still a valuable resource, and just because they know the bare minimum to be a threat doesn’t mean that they can handle themselves. Harry Potter tripped into this one by accident, but I’m still going to give a pass and say that Rowling got it right when she made Hogwarts a castle with strong protective wards.
Even if the situation has changed as of late, like it did with non-magical people, for most of their history wizards would have considered war to be the constant, and they would have structured their society accordingly in response. Wherever you have a large number of people who are fair targets in war, but can’t easily defend themselves, you’re going to want to defend that location.
Lastly, keep in mind just how difficult it is for anyone in our world to keep information locked down, or remove information from the internet. Even in the Eighties, in the Nineties, sure, wizards might be able to keep a lid on what’s going on. And maybe they could keep it going for a bit longer, staving off the inevitable.
But sooner or later, some idiot with a camera phone is going to see something and take a picture and upload it to the internet before damage control can roll in. Repeat that enough times, and the masquerade is going to fall.
If your world-as-presented seems to break down under examination
For example, the wizards of Harry Potter are collectively guilty for every muggle that dies of old age or sickness. Wizards routinely live into their hundreds, and more wizards than Flamel and his wife lived for multiple centuries. Given the preponderance of healing magic, their long lives absolutely have something to do with their ability to cast charms and take potions. There is no way that this couldn’t result in a longer life by at least a few years, so even if having a “magical core” or something also lengthens your lifespan that cannot be the only cause.
But we also have evidence that magic can work on muggles, even to the point of causing bodily transformations without future ill effect (see: what Hagrid and the Weasley Twins each did to Dudley), which means that the only thing preventing muggles from taking advantage of at least some magic is that the Wizarding World prefers its secrecy over all other concerns.
Or, for that matter, we wonder why, when wizards live into their hundreds, most of the influential wizards like Lucius Malfoy and Cornelius Fudge are surprisingly young. They’re the right age for muggles in the same position, but not wizards. Some fans theorize that most of the older wizards died in the war against Voldemort and one of the current problems in Britain is that wizards like Lucius Malfoy are just play-acting at what they think it means to be the head of an Ancient and Noble House, like if politics were suddenly taken over by teenagers. But as much as I like this theory, there’s not much in the text to support it.
Basically: It’s okay to leave clues in your story, only hinting at certain aspects of your world. But there is so much bad worldbuilding out there that you cannot expect the reader to simply assume that your world, when it also appears to break down like all the others, is nonetheless different from them.
Your turn: What else should worldbuilders keep in mind when designing a masquerade?