This is the output cluster. Left is 1st the middle gear is 5th and right is 4th. You can see the dog ring that drives 4th (there are dogs on the 5th gear side to - these engage directly into windows in 1st gear)
It could be that you have rounded or worn dogs and/or windows on 4th. This would drive the 2 apart under load. It could also be a bent selector fork that moves the dog ring.
Also, there are circlips that retain 4th and 3rd and if one of these has jumped its race the gears can wander up and down the shaft. Sadly you won't know until you've split the cases.