Am I the only one bothered by the term "people of color"?

I see this term pop up more and more and it's always struck me as an odd term. But it seems to have become widely accepted, and the troubling connotations that come with it aren't going away.

I mean, it seems to essentially divide Americans into two factions, white, and non-white. As if white people and non-white people are monolithic identities. In the case of white people, it seems to imply a people devoid of any culture of ethnic identity. We're not even a shade. We're just the norm. Even if it's meant well (I assume it is) it's basically the term colored all over again. And yet I see it liberally used by progressives.

Is anyone else bothered by the term?