Put yourself in the 9/11 victims shoes.
Well I can't, since they're dead. If you mean their relatives, then just because I'm sorry for their loss and understand a few may be upset at the idea doesn't mean I have to agree with them. If they can't distinguish between a place of worship for a religion and some people who killed many because of a twisted view of said religion, then that is their problem.
And it's not ON ground zero, it's near ground zero.