As long as you fix the "hole" then even if they continue to target you, they shouldn't get in. Of course the challenge is figuring out what that hole is. If they're targeting from the outside, yes getting your IP to change can help, but if it is malware on the inside or related to VPN or DDNS, that won't do anything.
Personally I'd be using a completely new and very secure username and password on the router, after factory resetting, flashing with fresh known good firmware from a known clean PC, then factory resetting again. Of course all devices are OFF the network until each one is checked and cleaned in an isolated environment.