Building recognition

Hey guys,

I want to do something like this:

-1 big building

-you stand about 30m away from it (you will only be able to get part of the building - about 30% - on camera)

-I want to highlight certain building features, like the entrance, art or rooms (from the outside)

-Some features wont be good enough to track on their own - so I need something like anchor points in relation to the wanted building feature, to then highlight the right thing

-When I move around with the camera and scan other parts of the building it should still recognize the building and put certain overlays in the right place/position on the building

-So I bascially need to recognize smaller parts and keep them in the right place of the "bigger picture"

I uploaded an example picture (not the real building).

The "Flag" will probably get recognized, the rooms most likely not since they kinda look alike, so I would need the flag for instance as an anchor point. The green spray can thingy shows the part of the building which should be recognized and mildly highlighty with a green color or whatever, I dont know yet. Just to make it clear to the user, that THIS building part is actually recognized. I saw a similar youtube video, but without explanation and he didnt really move the camera around.

My experience with vuforia so far, was about smaller objects, which I held in my hand, so I dont really know how I should start a project like that.

Can you push me in the right direction, or give me a good idea on how to start?

I know that, I will need to code  a fair amount, since I have some logic behind the application anyway

Thanks in advance