memory – What is the reason for computer going into a boot loop if it is switched off from the PSU switch?

My PC Specs:
Intel i5 4th Gen 4690 Processor
Asus B85M-E Motherboard
AITC DDR3 8GB 1600Mhz RAM
WD Blue 1TB HDD
ANTEC ATOM V450 450W Power Supply

Issue:
I shutdown windows 10 and after making sure everything is turned off (HDD, Cooling Fan, Casing Fan) I turn off power supply by flipping power supply switch. But when I turn on power supply back again and try to turn my pc on, it goes to a boot loop. HDD, CPU Fan and Casing fan start for few seconds and turn off and start again. This happens until i turn it off. Then I have to remove my RAM and reinsert it few times until pc turn on normally. But if I didn’t turn the power supply switch off after shutting down, then pc starts without any issues.

Does anyone have an idea what should i do? I bought a new graphic card and i am worried that my computer won’t turn on if I turn my power supply off to install it.