Ancestors are people you a related to by blood who lived before you were born. Are you implying that Black and Brown children today do not have ancestors that were slaves just like you have not ancestors that owned slaves? Black immigrants did not come here in significant numbers until after the Immigration and Nationality Act of 1965. So it stands to reason that probably more than 90% of the Black people in the United States right now can trace their roots back to slavery in this country through at least one of their parents. And any Black person here who has ancestors from at least two generations ago has someone in their family that lived through the Jim Crow era - like my parents did.
But as I said, no one should feel shame for something they have not done. You should not feel shame for anyone doing bad things today just because you are not in a position to stop them.