No ... Whites who grow up in poor neighborhoods are just as "unprivileged" as minorities. And minorities who grow up in affluent neighborhoods are just as "privileged" as whites.
Note the word "privilege", though.
Nobody is entitled to success ... There is no Right to it.
Even people who grow up surrounded by affluence can be unsuccessful, even if they take advantage of all opportunities.