Personally, I think you are edging towards making it overcomplicated. You are fortunate to have a simple use case, so make the feature simple too.
I would suggest you simply have one list which is a mix of groups and users. You can differentiate with a simple icon. You also have the benefit that in almost all cases, it will be obvious which ones are groups and which ones are users. Consider “Administrators” vs “John Smith”. Grouping them with headers is just overkill for this scenario.
I would also say that the scroll bar is very important. For example, you may want to search for “John” but you can’t remember the surname… and there are 100 Johns in the system. A scrollbar is critical in that instance so the user can scan the list for the one they want. I appreciate it’s an unlikely scenario, but you lost nothing by having the scroll bar supported, and you can make it conditional anyhow.
With that said, here is a quick example:
You will notice I didn’t include the “add” button. I think it is fine to include it, but I don’t think it is required either. So it’s up to you if you want to include it or not, shouldn’t matter much either way.