Migration Time
The planning is done, the tools are ready, the step plan is ready, it’s migration time!
Recently I was working with a team who were migrating from a legacy VMSphere 5.0 environment to Azure. Azure migrate had been setup and was replicating (curiously intermittently) and we were ready for the go/no-go call for that final replication.
GO
Open Azure migrate, click Migrate and then, an error…
After all of that, a final replication error, no errors seen on the VMWare hosts, but a failure and error in Azure migrate. Digging deeper we suspected that this and the intermittent migration errors were a symptom of the same issue. Knowing the source was a particularly legacy version of VMWare, we suspected it might be related to the VDDK version. Having already downgraded to version 7.0 we then looked to download to version 6.7.0, not easy to find but once this had been completed it was back to Go!
Note: I know a lot of this stress would have been forseen by a test migration, unfortunately we weren’t in a position to perform a test to an isolated network.
In to the Cloud
With the source VM now turned off and the Azure hosted VM powered up and passed initial smoke testing, the team then proceeded to work through the migration check list uninstalling legacy VMWare agents and installing Azure agents. We seemed to have a couple of issues with Proxy configuration where the winhhtp proxy had been configured which blocked a few of the agents from communicating correctly.
Having surmounted these issues, we just had to install the Azure Monitoring Agent and once complete, we were able to hand over to Hyper-Care…
AMA
The Azure Monitoring Agent is the replacement for the legacy log analytics agent and provides a greater level of detail than the old agent. It does require a little more configuration with Data Collection Rules (DCR’s) being required. The install is a little more fun as well, it’s not a case of installing the agent extension through the portal, you need to install with a command either through Azure CLI or Powershell
az vm extension set --name AzureMonitorWindowsAgent --publisher Microsoft.Azure.Monitor --ids <vm-resource-id> --enable-auto-upgrade true
Once the command has completed you should see the Agent in the Azure Extensions page, and you should be able to run the following command to see the agent reporting in to Log Analytics.
Heartbeat | where Category == "Azure Monitor Agent" and Computer == "<computer-name>" | take 10
what you shouldn’t see is the status “Transitioning”
Troubleshooter
Microsoft have a good AMA troubleshooting page on Microsoft learn.
Call the Cavalry
We ran the troubleshooter and also reviewed the logs in C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\1.30.0.0, we saw some issues with the Agent being unable to validate the hostname, which was a bit weird, but we couldn’t figure it out so we gave Microsoft a call…
ARC So…
The Microsoft engineer pointed out to us that the VM was registered with Azure ARC! This meant that whilst the VM appeared to be communicating correctly through the agent the AMA wasn’t able to gather the correct metadata, hence the hostname error.
It was a bit of a miss and a lesson learned for next time.
Now we had migrated the VM though, we needed to ensure that the agent was removed, first we check the status of the agent.
azcmagent show
We then ran a command to ensure the agent could communicate with the management plane and specified the location.
azcmagent check --location uksouth
Unfortunately we were stymied by the Proxy configuration of the agent, we wanted the agent to go direct and not through the proxy server that remained on-premsies.
azcmagent config clear proxy.url
Once we removed the proxy configuration we were able to disconnect the agent. We use the device code option as we needed to authenticate the agent with someone who had access to the ARC resource group. The migration team at this time did not.
azcmagent disconnect --use-device-code
With the agent disconnected, we then uninstalled the Connected Machine Agent from the device gave it a reboot, re-ran the agent install and the agent finally installed and appeared healthy.
Not so fast
Before we were able to congratulate ourselves too much, we found that we were unable to run remote commands on the VM. The issue pointed to the Azure VM Agent, so we ran a repair the Windows Azure VM Agent (which took a long time) but finally the agent repaired and we were able to run remote commands on the VM and manage and monitor it correctly. I suspect that the connected Machine Agent uninstall corrupted some of the Azure VM Agent components which was fixed by the repair.
Update
As was pointed out on my LinkedIn post, Microsoft have released a document providing details of the steps that Should be carried out for an Azure-Arc Connected device that is being migrated to Azure. Migrate your on-premises or other cloud Azure Arc-enabled server to Azure