Not true. Rare earth metal mining is just as environmentally intensive as uranium. As a matter of fact, uranium and rare earth metals are sometimes mined together.
https://www.theguardian.com/sustainable-business/rare-earth-mining-china-social-environmental-costs
But then, due to very low surface power density of PV and wind, you need whole lot of rare earth metals. Just scroll to the table...
https://www.sciencedirect.com/science/article/pii/S0301421518302726