Yes, I know, It’s been a while since I’ve last posted here, yeah, getting used to a new country is not an easy thing. Well, the idea is to keep up with the blog and try to pay more attention to it.
Now, let’s get to business, shall we?
I am a firm believer in properly planning, thinking, and designing, thinking every possible outcome, but when you are deploying new technology there is always a showstopper. Sometimes it can be easily solved and sometimes it takes a bit more time.
Lately, I’ve been working on vSphere with Tanzu and I came across an interesting issue with a Client.
My client which for confidentiality reasons I’ll name Contoso had an Active Directory domain that also served as a primary DNS zone contoso.local. as part of a deployment that has been going around for quite some time and a scenario that as consultants we can face often (A legacy environment that has hundreds or even thousands of servers coming from many years ago).
We designed a simple yet functional solution of vSphere with Tanzu leveraging vDS and using NSX ALB.
The initial deployment worked like a charm, we were able to set up our NSX ALB cluster, the service engine, set up the supervisor cluster, the TKG Cluster, and the Harbor repo all in less than a day.
The problem came when performing the first pull of an image. The worker nodes were not resolving internal DNS.
We logged in to the TKG Cluster nodes (for reference you can see here )
and we tested that the proper servers were configured.
additionally, the communication was working correctly.
Seems easy, doesn’t it? Hang in there.
Well, after some googling we came across this KB article that states:
Symptoms:
Dig and nslookup fails from the supervisor control plane nodes to hostnames ending in .local
Cause
This is not unique to vSphere with Tanzu. This is expected behavior from the systemd-resolved service.
https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html
- Multi-label names with the domain suffix “
.local” are resolved using MulticastDNS on all local interfaces where MulticastDNS is enabled. As with LLMNR, IPv4 address lookups are sent via IPv4 and IPv6 address lookups are sent via IPv6. - Queries for multi-label names are routed via unicast DNS on local interfaces that have a DNS server configured, plus the globally configured DNS servers if there are any. Which interfaces are used is determined by the routing logic based on search and route-only domains, described below. Note that by default, lookups for domains with the “
.local” suffix are not routed to DNS servers unless the domain is specified explicitly as routing or search domain for the DNS server and interface. This means that on networks where the “.local” domain is defined in a site-specific DNS server, explicit search or routing domains need to be configured to make lookups work within this DNS domain. Note that these days, it’s generally recommended to avoid defining “.local” in a DNS server, as RFC6762 reserves this domain for exclusive MulticastDNS use.
The KB also lists a workaround intended only to use on POC environments and NOT on production.
Basically, the problem is that PhotonOS (the OS for TKG nodes) uses the systemd-resolver service.
Systemd-resolver it’s a widely used MulticastDNS resolver, and RFC6762 defines that any domain with the .local suffix should be used only for internal use. (In other words, it’s the equivalent of naming your DNS domain localhost).
On top of that, we found another problem, as per the KB article:
The .local hostname is reserved for use in mDNS per RFC6762 therefore trying to resolve it against a DNS server violates RFC6762. As such VMware does not recommend any deployment which uses .local for any components. (this includes vCenter, ESXi, nsx manager, nsx edge nodes, and any endpoint TKGS uses like harbor).
The .local DNS suffix might be reserved in other products which we want to deploy in this environment, and funny enough. It does not only apply to VMware, Microsoft (who was the main promoter for the .local domains in the early days, is now requesting people to change it).
Lastly, since 2014 you are not able to get a public certificate with the .local suffix for the SAN. Cool.
So we can conclude, this is not a Tanzu issue, it’s a tech debt issue and should be addressed
What shall we do next?
We discussed a couple of options in the table.
The first option was to proceed with the workaround specified in the KB article, even though it’s production.
VMware said they would support us, but the problem is that each time you need to perform any lifecycle activity (such as an Upgrade, cluster provisioning or redeploy) you should re-apply manually the workaround.
Pros: It would work, other than that I don’t really see any.
Cons: Requires a LOT of manual effort. The KB said it was not supported for production but GSS said it was.
The second option is the rebuild the contoso.local domain and rename it to something supported and future-proof (contoso.tech for example). Particularly, I preferred this approach since it’s the only way to fix this issue and avoid any further issues.
Pros: It’s a permanent solution
Cons: Takes a LOT of effort, design, planning, and execution would take hundreds of man hours and it has quite a risk associated.
Third option: VMware to the rescue. Thanks to the GSS team for their support and their ideas, we were able to sort out this issue, at least for the short term.
The solution, only applied to Tanzu, was to re-design the Tanzu deployment but using Tanzu Kubernetes Grid Multicloud (TKGm) instead of using vSphere With Tanzu.
Once deployed at the moment of provisioning the TKGm you can run the steps described in KB 83623 and it quotes:
“To workaround this issue you will need to deploy your management and workload clusters with a modified plan that will update the name resolution parameters on the TKG nodes as they are deployed.
Nameservers on vSphere provides the instructions needed for using custom DNS servers on your TKG nodes. These instructions only need one modification to allow for domain names ending in “.local” to be resolved. A “searchDomains” line needs to be added to the end of the vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml files. Once modified, the end of these files should look like the following:
nameservers: [“8.8.8.8”]
searchDomains: [“corp.local”] “
Final thoughts:
You know I’m a big fan of VMware, and most of this blog speaks about my experience with VMware products. I have contact with some people inside VMware and I’ve escalated this particular topic, but after months of reading documentation, taking training, and assisting webinars I never saw a single mention of this limitation of using the .local DNS suffix. It would be nice to have that included in the documentation.
Additionally following on the domain migration topic, I like to explain the situation with this analogy.
My house in Argentina was built 60 years ago, it’s a nice house but a couple of years back we had to upgrade the pipes because it had copper pipes and they were getting rusty, the electrical cabling was designed 60 years ago, where a normal household didn’t have the same amount of appliances or devices that it has today, not even bringing Electrical Vehicles in the picture.
When this contoso.local domain it was built for a purpose, over the course of time the purpose shifted and it expanded, and now would be a good time to revisit the house’s electrical system to make it ready to withstand whatever the future brings our way.