In TLS Host name you should write devicename-yourID.dns.nextdns.io
For example:
ASUS-12a345.dns.nextdns.io
Then go to test.nextdns.io to see if it’s working.
Anyway, if you use Merlin, imho is better to use the NextDNS CLI client, that uses DoH instead of DoT...