An earlier report by Public Health England said racism may contribute to the unequal death toll.
Dr Raghib Ali suggested it was time to stop using ethnicity when deciding who needed help.
He said focusing on factors like jobs and housing would help more people.
The newly appointed expert adviser said this approach would help prevent everyone at risk - including poorer white groups, including those living in crowded housing - from missing out on help.
Dr Ali made his comments during a briefing on the first quarterly report on Covid disparities, led by the government's Race Disparity Unit and the Minister for Equalities, Kemi Badenoch.
The report is a stocktake of the actions taken and evidence gathered since previous analyses by Public Health England, which set out the greater risks faced by black, Asian and other ethnic minority groups.
The doctors' union, the British Medical Association, responded to the quarterly eport by calling for "tangible action right now to protect BAME [black and minority ethnic] people".
"As we sit amid a second wave of infections, we know that about a third of those admitted to intensive care are not white - showing no change since the first peak," said Dr Chaand Nagpaul, BMA council chair.
"Meanwhile, black and Asian people have been found twice as likely to be infected compared to white people."
To find out more about why these risks exist, the authors reviewed recent studies, including figures from the Office for National Statistics, and Public Health England.
While no single study was able to examine all the possible reasons, the report agrees with previous analyses that a range of factors are involved.
* Where people live - with a higher risk in urban areas with greater population density
* Occupational exposure - for example data shows Black people are more likely to work in healthcare than other groups
* Household composition, with larger households at greater risk
* Pre-existing health conditions - such as obesity, which can make it harder to recover from the virus
Dr Ali said that while studies often differed in their conclusions, the older people are, and where they lived, were two of the biggest factors behind the increased risks.
He added that much of the increased risk faced by ethnic minorities could be explained by factors such as the ones mentioned above.
But he agreed with the report authors that a "small part" of the excess risk remained unexplained for some groups.
* It will become mandatory for ethnicity to be recorded on death certificates as "this is the only way of establishing a complete picture of the impact of the virus on ethnic minorities". This will involve patients being asked about ethnicity during consultations with their doctors.
* A community champions scheme will help increase tailored communication and advice for communities most at-risk and fund grassroots groups.
Recommendations in the report include:
* Risk assessments at work for everyone, not just people from ethnic minority groups, to avoid stigma and help protect all people in need of support
* A risk calculator tool is being developed by the University of Oxford to help doctors and individuals understand more about their risks from Covid.
Minister for Equalities, Kemi Badenoch, told the BBC: "Quite often people expect there will be announcement that will be specifically just for ethnic minorities but actually the risk profile for vulnerability goes across many different groups.
She added: "We have made sure all of the work we have done, all of the action taken to safeguard people's lives have been done across the population as a whole, with a particular focus on the vulnerable."
In the briefing, Dr Ali said: "The problem with focusing on ethnicity as a risk factor is that it misses the very large number of non-ethnic minority groups, so whites basically, who also live in deprived areas and overcrowded housing and with high risk occupations."
He added the whole population should have a "personalised risk assessment" rather than just targeting ethnic groups.
"It doesn't make sense to put all ethnic minorities in the same basket as it doesn't make sense to put all whites in the same basket," he said.
On structural racism, he said he was not convinced by the narrative that racism played a part in coronavirus deaths.
He pointed out that generally people from BAME backgrounds have better health outcomes than other groups.
Prof Naveed Sattar, at the University of Glasgow, said: "This report reminds us that there has been 'uneven spread' of the virus dependent on where and in what kind of environments people live and the jobs they have and that these factors explain a substantial part of the ethnic differences in risk of severe Covid, but not all."
He suggested some health factors also appeared to be relevant and said more work was needed to better understand these factors.
Meanwhile, GP, Dr Hajira Dambha-Miller, said she welcomed the comprehensive report.
But she added: "Further detail is still needed in explaining why BAME groups are more susceptible to worse outcomes. I don't think the report goes far enough in exploring the wider social factors that may contribute to viral transmission and death."
The ones who are crazy enough to think they can change the world, are the ones that do.