And it's not just a matter of losing for example +3 for a few levels because of missing a classing. When you turn them into torpers, as noted it's a minus 5 acc/rld which is a permanent loss. That would mean that even if you had elite gunners, you've turned them into 8/6 gunners. Since they are torpers, I assume it was a torpedo high stat... meaning those gunners would be the lowest of lows in terms of ability.
Just sell or trade the torpers for something fair(likely a little lower level) and work from there.