While the Muslims have the legal right to build the mosque there, the question of whether it is moral or not is the big debate right now between the two sides. I believe they are doing a massive demonstrations right now from both sides.
It will take a long time before the American public will change their opinion of Muslims. It is similar to what happened to Catholics, Jews, and etc. in US history. There were a lot of resistance against those groups at one time or another.