gmail – Prevent label from being applied

How can I prevent GMail from applying certain labels to messages.

For example, I have a “Work” label and a “Home” label. Gmail is labeling some messages as both “Work” and “Home.” How do I prevent this?

I have a filter that specifies “Work” when it comes to my work email address.
I have a filter that specifies “Home” when it comes to my personal email address.

How do I tell Gmail not to apply both labels to an email coming to one but not the other?

My filters are as follows:

Matches: to:(myname@myworkaddress.com)
Do this: Apply label “Work”, Never send it to Spam

Matches: to:(mymickname@homesweethome.com)
Do this: Apply label “Home”, Never send it to Spam

Also, as a note, I never receive emails sent to both my home and work email addresses.