Magento2 Product Collection Observer only changes one page at a time

In ListProduct block Magento is using initializeProductCollection() to initialize the collection.

While initializing it dispatches event catalog_block_product_list_collection as follows.

$this->_eventManager->dispatch(
    'catalog_block_product_list_collection',
    ('collection' => $collection)
);

Using the event catalog_block_product_list_collection I am trying to filter a list of products:

$_collection = $observer->getCollection();
$_collection->addAttributeToFilter('sku', ('nin' => $invalidSkus));

however instead of limiting the category from 50 products to 49 products, it is limiting the page results of 12 products to 11 products. If I have a list of 12 invalid skus that all coincide to be on page 1, then page 1 returns no products.

How do I filter the entire product list for a category, not just the product list for a single category page?