If you are doing some good work, negotiate with him that you want to contribute some amount for the welfare of the society and do it within limits.
If he does not listen then don’t worry.
When you are living together in one house then you do not have to say, this is his money or my money. The money belongs to everyone.
You have built the house in which both of you are living and you do not take rent from him for living in the house, right?
You are doing a noble act by helping poor children. See how much joy you are bringing to the helpless children. So you do not have to feel sorry about it.
If one is robbing to consume alcohol or indulge in such acts then that is not a correct thing to do.