I've had some middle back soreness for a while. It would always get bad while fishing. The thing that helped me was getting rid of the vest. I use a Richardson for flies and a waste pouch for tippet, shot, keys, etc. Been much better on my back. As others have said, anything to get the weight off your shoulders, get rid of the vest. Use a chest box, pack, anything that makes you feel more comfortable. While suffering is one way to get through it, making yourself more comfortable makes fishing more enjoyable, and you'll be more willing to spend a full day on the water without getting exhausted.
Now, I just gotta stop tent camping, sleeping on the ground made my back sore at the jam!