Firstly, welcome to the fifth issue of D & D, the learning curve is not too steep for this system and you should be able to learn it fairly easily. Before I come to the main part of the answer, I would like to clarify something else.
Unlike other systems that publish all the important rules in a single Grundregelwerk D & D has decided to divide the rules into the three main rules. The Player's Handbook (PHB), Dungeon Master's Guide (DMG) and the Monster Manual (MM). In an ideal world, a DM would have access to all three, but some are more important than others.
First, you need that Player's Handbook, This book contains all the major playable races, character classes, spells, talents, and pieces of equipment. Without this book, your players can not make characters or look up their abilities. It's at my table by far the most used book of the three.
Second, I suggest getting that Monsters manual, This may be controversial, but I (and my friends, the DM) find the MM more useful than the DMG. The MM contains the statistics of all the creatures and enemies your group will encounter throughout the campaign. With only this book, a good DM can perform whole campaigns with very little preparation.
Finally there is that Dungeon Master's Guide, The DMG is the book for all other things. Rules, guidelines and DM-specific advice that players do not need to see. It includes worldwide and campaign-based advice, plus a set of additional rules and advice that you do not really need to run the base game, but that you'll need later. Personally, I rarely use the DMG at my table and only occasionally use it between sessions. I am firmly convinced that you can get along without owning it at all.
There are also the basic rules that can contain a reduced version of the three ledgers and can also be a good starting point. It contains just enough to help you decide if D & D 5e is the right game for you, but not so much that you will not have to buy the books anyway.
What is the starter set?
The starter set contains:
- 64-page adventure book with everything the dungeon master needs to get started
- 32-page rulebook for playing Level 1-5 characters
- Pre-generated characters, with character sheet
- 6 dice
These are the basics and everything you need to play D & D 5th Edition. For more information, see Difference between DnD 5e Starter Set and Basic Rules.
So what should you get?
You seem to decide between the DMG and the Starter Set, but did not mention that you have access to the PHB. In my opinion, buying PHB and MM as DMG is more important, as the starter set includes basic versions of all three versions, so this is probably a good choice. Especially if you do not have the right dice.
If you combine the starter set with the basic rules, you should be able to play for a while without major problems. If you want more options, first read a copy of the PHB, then the MM, and finally the DMG and / or other extension books.