I think the problem here is one of level of detail. While vectorization works great for making an image that can scale infinitely, it relies on firm patterns that can be mathematically described.
Unfortunately, real life is full of imperfections and variations that make it impossible to describe in a pure vector format, at-least with any meaningful gain. We could theoretically make a vector format that maps every single pixel, but then we’d have a raster image that couldn’t scale any better than a normal raster image.
Image and video compression are already the applications of the kind of thinking you are talking about. They look for the patterns that can be identified to reduce the storage required and when using lossy compression, they bend the rules further to get a match so that they can reduce the amount of information necessary to represent the image.
Vectorizing the image is another level of extreme for this, but you will notice that the image quality always drops significantly when such vectorization is applied (due to the loss of the random information that makes a photo look like real life.)
As for whether a vector image of a scene is a photo or not. I think that’s a really hard question to answer. Personally, I’d say yes, if it is a rendering of real life based on sampling the light (regardless of how lifelike it may be), I think that it probably could be considered a photograph, but I could also see how someone might see it as more akin to a painting at that point as well. I don’t think there is a strong line answer there.