No arguments from me that fishing is not a no-impact past-time. However, trying to equate hooking and playing a fish, along with all the possible chances for injury to the fish, with stream "maintenance" is comparing apples to oranges. One has to do with the fish and the other has to do with the habitat of the fish (on a micro-scale, where that single branch is and on a macro scale, when those multiple branches accumulate in a log jam).
It's a given that a chance for injury to a fish exists by our very presence on the stream (regardless of what kind of tackle you use, or how delicately you play or release a fish). However, just because a fish might be altered in some way by our fishing for it does not also mean we have to alter their habitat and I interpolated that the removal was for the convenience of your fishing, not the convenience of the fish. If they were small limbs, streams have a pretty good means of cleansing themselves over a fairly short amount of time. And small limbs stacked up in front of a log jam make for some additional cover for fish to hide in.
I've accepted that my presence, for a time, is unnatural in a trout's world. However, I don't feel the need to alter their natural world by removing stuff from it. About the only thing I feel the need to remove from trout streams is man-made litter (although there will probably always be that entrepreneurial trout that has found a way to make that litter their home, like the two big browns in a shopping cart).
If everyone who fished removed four small branches from a stream, we'd have quite the piles of sticks streamside. If everyone who fished piled four small rocks into the tail of a pool, to raise the pool level, we'd have some unfortunate temporary fish barriers (but remedied by the next flood event).