I'm not going that far back in history, although I would disagree with your stance for that time frame. A lot of slaves, especially ones from the south during the time, were considered to be nothing more than disposable property. Even when some slaves were allowed to vote, they would only be considered 3/5's of a person. Native Americans were forced into smaller and smaller living areas. I would not be surprised if Asians that worked in the mines ended up with diseases and ended up dead. The services various minorities could provide back then was valued a lot more universally in comparison to their actual life imo. You are free to go more in depth with others on that area, but I am personally going to be more focused on the modern times.
I think that's sort of Shace's point, is that even just the term 'matter' is up for debate. We really couldn't have built this country without the work of the African slaves, or the Chinese that helped build the railways, etc. Objectively, they mattered. But morally, they didn't (to the people at the time). There's a distinction there that I think Shace is trying to call attention to. What context we're talking about, whether it be time, race of the person being asked, their living conditions, where in the country we're talking about, their political stance, whether we're talking moral or objective... everything has the potential to change the answer to your initial question, and those answers can change simply by pairing one of those previous pieces of context with another.
To me personally though, your question has always been a moral one, and has been about the current times, big picture United States. And again, in that sense I'd still say yes, minorities matter to most Americans, as human beings that deserve equality and happy lives. But basically every piece of context -including election related stuff- can get in the way of that, making it seem like far fewer people care than that actually do.