I have an SQL database in Azure that is ~4.5 TB / 150 million rows worth of data. Users will need to query this database for large sets of data. Right now, I am having users submit their requests through a web application hosted on an Azure VM.
How can I serve the rows pulled from the Azure SQL database directly to the user? It seems slow for the server to have to stream the data from Azure SQL, and then for the client to have to stream that from the VM.
I wish the user could directly query the Azure SQL db from the client side, but the problem is that the user’s request has to be preprocessed and manipulated against a reference set of over a gigabyte of data; the request to the SQL db needs to happen server-side. Any help appreciated!