I had a similar experience a few summers ago.

There's a homeless man that I used to see on my commute to work. He was sort of like a skinny Santa in a wheelchair. I always saw him at a short distance and he always looked disgruntled. One day I was leaving work and saw 2 or 3 people on the ground in front of him picking up change. He had a big bag of change that had broken and people were helping him collect it all. Since my heart is not two sizes too small I stopped to help and as soon as I started picking up change he said to me "No no, Mr. Black Man, that won't do. Put the money down." I stared at him in disbelief for a few seconds and said "Really?" So I put the change back on the ground in front of him and walked away. As I did I noticed the other people there left as well. One of them caught up to me at the corner and apologized for what happened. I told him he had nothing to apologize for. But the entire train ride home I couldn't stop thinking about this homeless man refusing help from me.