You obviously haven't lived down south. The phrase "a spade is a spade" has been co-opted down there and is entirely racist. I still remember the time when I lived in North Carolina when a very large White guy used it in the derogatory, negative sense to a very short, wiry Black guy during a pickup basketball game. The White was on his back, out, an instant later. The rest of us playing ball just stood there going "holy shit". But we didn't have much sympathy for the White guy, as everyone playing ball, Black and White, knew that was a pretty bad racial slur. Just like the swastika, just like the battle flags of the Army of Tennessee and of Northern Virginia. Just because you chose not to see it doesn't mean it's not extremely offensive to a large group of people. You don't get to define what is offensive to people.