_acme-challenge.FQDNis a CNAME pointing toFQDN._acme.netdef.org- FQDN is, as implied, the FULL host name in both cases.
_acme-challenge.pkg.netdef.org. IN CNAME pkg.netdef.org._acme.netdef.org.
 - it is always 
_acme.netdef.org- it does not matter if FQDN is undernetdef.orgor not. 
- FQDN is, as implied, the FULL host name in both cases.
 _acme.netdef.orgis served byns-ch.netdef.org(ONLY that server, there is no secondary, it makes no sense to have a secondary)- DDNS updates with a TSIG key 
certbot-key.are enabled on that zone - certbot is configured to use the 
python3-certbot-dns-rfc2136module to put the challenges into DNS using that TSIG key certbot certonly --deploy-hook /etc/letsencrypt/renewal-hooks/deploy/01_deploy_pkg_servers.sh --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/tsig.conf --agree-tos --manual-public-ip-logging-ok -d deb-us.netdef.org -d pkg-us.netdef.org -d rpm-us.netdef.org- the deploy-hook uses SSH to copy the keys to the target system
- there is a special key for this on 
ns-chin/etc/letsencrypt/ssh_push_id - this key is in 
authorized_keyson the package servers withcommand="/etc/letsencrypt/ssh_receive.sh"restriction /etc/letsencrypt/ssh_receive.shsaves the key and reloads nginx
 - there is a special key for this on 
 
NOTE: the version of python3-certbot-dns-rfc2136  on ns-ch did not support CNAMEs and was manually patched and marked with apt-mark hold .