Wildfires raise fear of a ‘mass exodus’ by residents unable to afford housing