The page you are referring to is
This page is not customizable. The rendering and content displayed on the page is controlled by Microsoft.
The only way to turn off activities is to disable Graph from SP admin center. But once you disable it, you lose all functionality in Office 365 that is powered by the Graph, like Delve.
Reference – Office Delve for Office 365 admins
Having said that, activities in these cards are always security trimmed. It only shows the documents which you have access to.