Virtually anyone alive today has ancestors who owned slaves--he also has ancestors who were slaves. The "virtually" could probably be stricken from the sentence, but there might be someone, somewhere, whose ancestors all came from some obscure place that never heard of slavery.
President Reagan summed it up nicely, "Some people go through their entire lives wondering if they made a difference, Marines don't have that problem!!!"