I think the best argument for this is developing the social skill of being able to interact with people who are different from you. When you give money to a panhandler, it's an opportunity to have an interaction you wouldn't have had otherwise.

My impression is that many of these beggars are earning enough to survive, albeit in poverty, so your marginal dollar is probably more effective elsewhere given most people are not making the choice to give to them or not based on EA principles and others will continue to support them. If you consider local homelessness a top priority, my guess is that other interventions than small direct giving would be more effective, though I have not looked into it.

A few days ago I gave money to a homeless veteran who was in a wheelchair, on the streets of my hometown.  He had a card that proved he was a veteran.  He said he wanted to stay in a homeless shelter in Nashville and needed some cash for that purpose.  Maybe so.  I gave him $7.  Why?  Because (1) it was my town; and (2) he was a veteran.  I didn't do a calculus about whether the $7 would have been better spent on somebody far away.  I think we should try to do good locally as well.  If you ignore the suffering people in your immediate vicinity because it's more expensive and possibly more distressing to notice the suffering in your immediate vicinity, because perhaps you are implicated in it, then that's a problem.

I like this answer from theunitofcaring on tumblr:

Q: What do you do when (presumably) homeless people ask you for money? This happens fairly regularly where I live and I really don't think "I'm sorry sir, you are not the most efficient from of charity" would be received well. So I tend to ignore them, or say "no" and keep walking. Which feels cruel. It doesn't help that the ask is something like "any spare change?" so if I don't want to give them money I usually have to lie or get into a longer conversation.

A: I used to give homeless people money a lot because of basically this problem! I eventually realized that was unsustainable and stopped carrying cash so I did not have the option, and could truthfully say “I’m sorry but I don’t carry any cash”. This is inconvenient if you want to carry cash for emergencies or that one stupid Berkeley sushi place that doesn’t take cards, but “I don’t have any cash to give you, sorry” is always true. I still occasionally buy someone ice cream but this feels less pressure-y and more “it’s hot out, it will make me happy to buy myself and this person some ice cream”.

It also is totally okay to ignore people on the street. It feels bad to me too, and I try to avoid it, but streets only work because people mostly agree not to make random costly requests of one another except when desperately needed, and it is morally fine to avoid interacting with people while out in public.