I assume a user can only have 1 department? or he/she is only a member of 1 department?
you could just filter the view by current user [me], and add the department to the view,
in that case, he can only see his task, he only have 1 department anyways. The list should be something like
This is a concept that I assume would work, let me know if it helps atleast.