We can say more than that. Difficulty of circumstances are defined by the conditions of the individual. In this respect, there's not much required for someone to have greater difficulty than another person.
All it takes is failing to overcome a circumstance in question. The reasons for this could be many: drug addiction, lack of family support, poverty, sexism, racism, personal interests, personal enemies to name but a few. Any one might make a situation unilaterally too difficult to overcome. Belonging to a social group who is favoured or one which is oppressed doesn't prevent this.
White privilege was never a reason to expect any given when person will do better or have less difficulty than someone of another race. It's just a description of a cultural state, that people who are white often have certain social resources.
Being rich doesn't necessarily mean you'll have less difficulty than someone who is poor. A rich person might act in a way which brings them more difficulty than a homeless person, they might be addicted to drugs, roam the street and refuse to use money to help mitigate hardships. Most of the time though, this doesn't happen. The rich person usually used wealth to mitigate hardship or help overcome their circumstance. Despite some rich people having difficulty, there is still a relation to describe.
White privilege is no different. It does not mean any white person will have less difficulty. Rather, it identifies certain distributions of resources, social events and significances, such that white people are less likely to face or do not face certain difficulties. It's never been a reason an individual overcomes their difficulties or not.
Privilege is not a causal reason.
The mistake a lot of people make (and one you are making here) is to think privilege is a causal reason. A lack of privilege is not a reason someone has difficulties. Nor is belonging race, sex, gender, etc., a reason someone has difficulty.
Causal reasons for difficulty are far more material. It's illness, lack of community, poverty,
actions of other people to exclude people of a race, sex or gender, etc., lack of services, an environment in which people harm each others and a host of other events we could name. Privilege is just
description of certain social relations and states formed out of those causes.