dnd 5e – Does the Mage Slayer feat impose disadvantage on concentration saves from all damage you deal, or only if damaged by the feat’s reaction attack?

According to my character sheet on D&D Beyond, the Mage Slayer feat says:

When a creature within 5 ft. of you casts a spell, you can use your reaction to make a melee weapon attack against that creature, and when you damage a creature that is concentrating on a spell, that creature has disadvantage on the saving throw it makes to maintain its concentration. You also have advantage on saving throws against spells cast by creatures within 5 ft. of you.

Say a mage is concentrating on a spell. If this player character hits the mage with the Attack action on the player character’s turn, does the mage have disadvantage on that Con saving throw to maintain concentration?

Or does the caster only have disadvantage on the Con save to maintain concentration if damaged by the reaction attack that is triggered by a spell being cast by the mage within 5 feet of the player character?