Applicants: yes, but not in my house!

Every time asylum seekers' centers open in Switzerland, the population is worried, even very hostile. So how do we explain this mistrust? Do asylum seekers really generate crime and insecurity?