I have to tell you I've had people express concerns in my riding as well, especially around Canada Day. Their condo board or the people who own their apartment building said they couldn't display the flag on their balconies or in their windows, so I think you are really on the right track here.
On February 2nd, 2012. See this statement in context.