It's up to you to do your own assessment but I don't see any reason to be fearful.
These are regulars at that library who never caused enough disruption to be banned, and aren't dangerous enough to be in jail. They also have more to lose by getting banned than housed patrons.
We're not talking about people being fine or not, we're talking about assessing the risk associated with homeless people patronizing libraries (which is effectively zero).
You are more likely to be attacked by a shark than a homeless person at a library and that should be obvious to anybody.
These are regulars at that library who never caused enough disruption to be banned, and aren't dangerous enough to be in jail. They also have more to lose by getting banned than housed patrons.