Otherwise, we’re caught in a loop of cast → lethargy → didn’t cast → no lethargy, etc.
If the same character were to move and take an action, then drop concentration, the lethargy doesn’t undo actions they started/started. I’ll even give a concrete example:
Let us use a sufficiently high level Eldritch Knight to caste Haste (and also have multiple attacks), with Haste already active. They move up to an enemy and take the Attack Action. Through some means, after the first attack, they drop concentration (let’s say Hellish Rebuke and they fail their save). What happens to the Fighter’s remaining available attacks? They’re not taking an additional action, but simply finishing an action that was already begun.
What if you wanted to Ready a spell (any spell)? Would you begin to take the Ready action and prepare the spell, but the lethargy undoes your Ready and the next spell? Or perhaps, you CAN Ready spells, which would just bypass this entire problem.
There are too many logical loopholes, pit-falls, and catch-22s for the answer to be anything other than “you may finish casting after you start, even if it breaks concentration”.