RAW, Darkness should make the combat quicker
I think the problem is that you misunderstand how Unseen Attackers and Targets and Hide work.
Unseen Attackers and Targets
If you are unseen when you attack, you get advantage. If you can’t see your target, you get disadvantage. These cancel out. So, for everyone but the warlock and anyone else who can see in darkness, the darkness makes no difference to attacks. For those that can see, it makes them more effective so the combat should be shorter.
As for spells, any spells that require seeing the target can’t be used, but that is not most spells. Spellcasters can simply avoid those spells. Spells that use attack rolls will be at both advantage and disadvantage so their chance to hit is unchanged. Being unable to see does not affect saving throws.
You know where everybody is even if you can’t see them
Unless they have taken the Hide action and have rolled a Dexterity (Stealth) check better than your passive Wisdom (Perception) check which is at -5 because you have disadvantage to ability checks that require sight. And they haven’t “come out of hiding” – by attacking, for example.
See What advantages does hiding have?