NASA expects humans to live on the Moon very soon.

[nzoomed]

a lot of technological advances have come from the space race though, so putting the brakes on it could stifle that

Well, yes we have had advantages from the space race. 

But when we still have people dying of hunger in some places while in other places some dine on caviar & gold leaf (yep, really) then I'd say that we'd do better to clean up our planet & fix the inequality, & then we can see what other benefits may come from the space race - preferably, things which can benefit us all rather than just the fortunate few.

Maybe so, but we have had huge advances in medicine due to research that is happening on the ISS and studies to the human body.
Its unfortunate that poorer countries dont see much of the benefits, but things are slowly improving.
We have had technology developed for recycling wastewater on the ISS that can give benefits back on earth in developing countries where they need water sanitization, etc.

The most important thing is that the days of governments wasting money on expendable rockets and other schemes designed to put money in the pockets of a few who run these rocket companies is going to be short lived once private enterprise manages to dominate the market.
SpaceX is already saving the US govt money every time they are used to put a payload into space.
The billion dollar SLS rocket will be retired once starship proves itself useful. Its reusable and its launch costs is a tiny fraction of an SLS launch.