Because of their size, they have many enemies of bigger size. Haru tries to chase them off. But not all humans are their enemy. Sho protects her from crows and Shawn's grandfather builded a Dollhouse for them.
They can live everywhere. The prefer to live under human's house, because they can "borrowing" food en masse.
The population of Borrowers is widely dispersed. Arrietty's parents didn't knowother Borrowers existoutside of their home. Pod Clock get know about other of his kind, when he meets Spiller in the woods. In the woods many Borrowers live together to protect themself, because the life in the woods is more dangerous than under human's houses.
In Borrowers' perspective the borrowing is not stealing, because it is necessary for survival. They never bring the borrowed things back, of course.