NSX-T ESXi host preparation fails - "[Errno 1] Operation not permitted It is not safe to continue."

As today was a lab day for the first time in a long while, i decided to finally get around to finishing my NSX-T nested lab.

So after deploying the manager and controller cluster, the time comes to prepare the ESXi hosts. And as we are preparing we run into the following error:

Which in itself is not very helpful unfortunately. So off to the ESX cli we go.

First, we try to manually install the software itself:

esxcli software vib install -d /vmfs/volumes/T2\ -\ SATA/nsx-lcp-1.1.0.0.0.4788198-esx65.zip  

Which fails with the error

Could not install image profile: ([], "Error running command '['/etc/init.d/nsx-da', 'start', 'install']': [Errno 1] Operation not permitted\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update.")

Every time the service seems to be a different one, but the general error seems to be the same. When we search for "nsx" in the logs, we end up with a variety of messages, but a significant amount of warnings appear related to secure boot. An example is

VisorFSTar: 1484: File (usr/lib/vmware/nsx-da/setDaUser.py) has both the sticky bit and exec bit set. Incompatible with secure boot  

As i've recently rebuilt my lab, i've done this with 6.5. My nested lab used for NSX-T is a 6.5 lab as well, and NSX 6.5 supports secure boot, the nested ESXi instance will be configured and running with secure boot if your VM was configured with EFI (which seems to be the default if you select VMware ESXi6.5 as a guest operating system for your VM.

After we power off the nested ESXi hosts and disable Secure Boot, the installation works just fine. So if you're running into this issue, disabling secure boot should allow you to install the NSX-T kernel modules just fine. If required you should be able to reenable it after preparing the hosts.

Secure boot can be disabled on a per-VM basis in VM->Edit Settings->VM Options->Boot Options->Secure Boot. Note that your VM must be powered off to change this.