The feat is balanced on any class because multiclassing exists
The feat is restricted for thematic reasons, well, I can’t prove that, but it does make sense at the very least. Regardless, a Fighter, Monk, Barbarian, or anybody else can get this feat simply by multiclassing into a spellcasting class. I can’t personally think of any specific Invocations that are alarmingly unbalancing only when held by a level 20 non-spellcaster (at that point, balancing I’d enough of a nightmare already), and I doubt the developers looked that closely at such a combination.
Of course, taking 1 level out of your main progression is a meaningful and steep price, but I can’t see that as being a balance concern the developers would’ve thought deeply (or at all) about.
Of course, it’s possible the developers didn’t think about multiclassing at all and instead found a problematic combination of features and invocations precisely and only for the non-spellcasting classes. I find this an unlikely enough scenario to conclude that the feat has its prerequisites for purely thematic reasons.