Federal disaster relief denies people the opportunity to live cheaply in exchange for living dangerously. That opportunity is particularly valuable to the poor.
To put this another way, federal disaster relief essentially forces people — most of them poor people --- to buy insurance they'd rather not have. The premiums are hidden in their housing prices, but they are none the less real...
Poor people, more than most, value cheap housing. A policy of disaster relief makes cheap housing hard to find. Therefore a policy of disaster relief is likely to impose a particular burden on the poor. If you want to help poor people, eliminating federal disaster relief is a good place to start.