I'm far from an 'expert' but have some anecdotal advice I'd like to give you:
There's a fine balance between stability and instability in a glide and I think you did a great job showing this in your experiments. An 'unstable' lure will have a tendency to roll. This roll is also what causes the lure to have an 'S' swimming pattern when straight retrieved. Instability is increased the higher the center of gravity is. A ballast close to the center line will cause it to roll and the further away from the center line the more stable it will be.
A stable lure will have no roll, and therefore no swimming action on a straight retrieve BUT will often have a really good gliding action when given a hard twitch. It's because this hard twitch/jerk forces a point of momentary instability, and when the lure stabilizes itself it then glides like a torpedo in a straight line either left or right.
You need something that is stable enough to glide, but unstable enough that it has a small amount of belly roll. From your experiments, I think you found both "extremes." My advice would be to find something in the middle.
Two things I would try:
1. Start with the most stable gliding bait, drill out your weight holes and add the weights. Test it. If it's a torpedo on the straight retrieve, remove the weights, drill the holes deeper (closer to the center line), put them back and test again. Keep doing this to see what happens. My thought is the closer the weights are to the center line, the more it will roll, but the less it will glide.
2. Start with the better swimming lure. Remove a small amount of weight from the largest ballast and take that little bit you removed and add it back to the bait so that it remains level when sinking. Test it. Keep doing this until you achieve a glide that's far enough with a bit of belly roll and you should be able to also have it swim on the straight retrieve.
The last thing you could do would be to change the shape of your lure... This sounds like it would be the most difficult but it would be interesting to see what your lure looks like to get a better idea of what's happening.