buttons – How should filtering toggles behave?

I have a page that display multiple notes that are assigned to each department.

At the top right corner of the page, there is a filtering section to allow the user to display notes that are only relevant to them/their department. The filters look like this:

enter image description here

It’s similar (although not entirely in terms of functionality) to iOS Scope buttons where clicking on a button and it’ll filter the results. I would like for the user to also be able to click on multiple buttons; this would filter the results based on those selections.

Deselecting all of the buttons will display all of the results, essentially reverting it back to its original state.

However, there are some kinks in this type of UI/functionality which I am unsure about.

  1. If we allow the user to select multiple buttons, that means it’ll display all results. If deselecting all also displays all of the results, isn’t that weird? I guess that’s no different than turning on all of the filters in an eCommerce site?

  2. I don’t think it’s clear to the user that the buttons are toggles – ie, you can select/deselect the buttons the same way, by clicking on them. I was thinking of adding an “x” next to it, but it looks like you’re removing the button instead of just deselecting.