Is it possible to switch debug mode in software?
No, only using a pin.
But the combination soft / hardware is working just fine
Use a GPIO output to flip a switch on the mode pin and reboot from software, as long if your PCB is powered you stay in the same mode.