Yeah that is what I did, but if you drill new holes in your leather panniers and the inner backing/support plate you can do without the stock frames using Easy Brackets, probably best with alloy backing plates for strength. Unfortunately I like my pannier chrome cage rails and the only way they fit is onto the stock frames. Hence my panniers stick out further than the rear dresser bars.
I have some hard bags being painted to which I will fit
@Mittzy 's stainless quick release brackets.