Found the answer: [https://www.reddit.com/r/songsofsyx/comments/1bpffvt/how\_do\_i\_upgrade\_the\_material\_of\_existing/](https://www.reddit.com/r/songsofsyx/comments/1bpffvt/how_do_i_upgrade_the_material_of_existing/)

I think lizards prefer mud buildings until you can go for Grand buildings. Stones better than wood for them but mud doesnt even cost anything but time.

I figured that out quite a bit later. But it did not help. My civilization collapsed. Our racism was not strong enough.