You may be able to enter the UK without a visa, with proof of your marriage, but it is not certain.
First, if you are a citizen of a country whose citizens normally do not require a visa for a short visit to the UK, then you do not need a visa.
Second, if you have a residence card issued by Belgium (or any other EU or EEA country or Switzerland), and that card says that it is a “residence card of a family member of an EU citizen,” then you can use that card instead of a visa when you are traveling with the EU family member.
Third, if you lack such a card, you are in general supposed to have an EEA family permit, which is essentially a visa. It must be issued free of charge and on the basis of an accelerated and simplified application procedure. To fly to the UK, the airline will require you to show the family permit before they will allow you to board the aircraft.
Fourth, if you lack the EEA family permit, you may be admitted under a provision in the Immigration (EEA) Regulations 2016 that allows you to “prove by other means” that you are entitled to the right of free movement. (This provision gives effect to a corresponding provision in the free movement directive 2004/38/EC.) This means that an immigration officer at the border should admit you under the circumstances you describe. We have reports on this site (although not recent ones) of people doing this successfully.
Fifth, some advice: if I were in your shoes, I would therefore apply for an EEA family permit if time allows. Having one gives you more certainty at the border. You probably won’t be turned back, but it’s possible. Why add uncertainty (and the attendant stress) to your trip?
Finally, the rules will change at 11 PM GMT on December 31, 2020 (midnight of 2020-12-31/2021-01-01 in Belgium and France), so if your anticipated visit is after that time, you should ignore this answer and apply for a normal visitor visa.