8 – View: Fields copy/reference each other when you add a separate display

So I have a normal page view with a handful of fields. Title, Price, ect.

I wanted to create a Entity Reference page so I click add entity reference and it copies my current fields from my page view. That is fine. I then start tweaking my entity reference fields, only to find out it started to tweak and delete my fields from the page view.

Is this normal? What is the point of having separate displays if the fields on both displays get altered at the same time? Is there a way to not have this happen or do I just need to add a 2nd duplicate view and go from there to have the results I want?