wsp not getting deployed in all servers

Last week, got into an issue that wsp was not deployed in all servers and hang. Please find below more information on that:

Issue:

Unable to deploy wsp solution in SP2016 On-Premise server

Resolution:

Cleared SharePoint config cache and rebooted all SharePoint servers and the SQL server, ran the command Add-SPSolution and  Install-SPSolution and the .wsp solution got deployed

Troubleshooting Steps Followed:

-   ​In Central Admin checked in Farm Solution Management the status was stuck on deploying​

-   Stopped the SharePoint Admin and SharePoint Timer services

-   Checked and found multiple One-time jobs​, deleted those using the scripts below ​ 

-   Find onetimer running job from timer job definition and delete with below query

Get-SPTimerJob | where{$_.schedule.description -eq "One-time"} | ForEach-Object{$_.Delete()}​

Get-SPTimerJob | where{$_.DisplayName -eq $null} | foreach-object {$_.Delete()}Get-SPTimerJob | where{$_.schedule.description -eq "One-time"} | ForEach-Object{$_.Delete()}​

-   Check if the timer service is ON using the script

$farm  = Get-SPFarm

$disabledTimers = $farm.TimerService.Instances | where {$_.Status -ne "Online"} ​

if ($disabledTimers -ne $null) ​

{ ​

   foreach ($timer in $disabledTimers) ​

   { ​

       Write-Host "Timer service instance on server " $timer.Server.Name " is not Online. Current status:" $timer.Status ​

       Write-Host "Attempting to set the status of the service instance to online" ​

       $timer.Status = [Microsoft.SharePoint.Administration.SPObjectStatus]::Online ​

       $timer.Update() ​

   } ​

} 

-  Clear the timer Cache on all the servers in the below path

C:\programdata\Microsoft\SharePoint\Config\<<Folder with GUID with hyphen(-)>>

- Run the command Add-SPSolution and  Install-SPSolution, deploy failed

- Check and update the status of  Allow Service Jobs and Allow Content DB Jobs  using the script below

$farm = Get-SPFarm

$FarmTimers = $farm.TimerService.Instances

foreach ($FT in $FarmTimers)

{

write-host "Server: " $FT.Server.Name.ToString();

write-host "Status: " $FT.status;

write-host "Allow Service Jobs: " $FT.AllowServiceJobs;

write-host "Allow Content DB Jobs: " $FT.AllowContentDatabaseJobs;"`n"

}

- As the health analyzer detected errors related to SPAdminv4 ran the command 

net stop SPadminv4

net start SPadminv4

- Rebooted the SharePoint servers and ran IISreset

- Tried deploying the solution using below command but it failed

stsadm.exe -o deploysolution -name SharePointProject1.wsp -allcontenturls -immediate -allowgacdeployment -allowcaspolicies

- So ran the below commands and solution was deployed

Add-SPSolution -LiteralPath E:\Ravi\SharePointProject1\SharePointProject1\bin\Debug\SharePointProject1.wsp

Install-SPSolution -Identity SharePointProject1.wsp -GACDeployment -ForceAdd-SPSolution -LiteralPath E:\Ravi\SharePointProject1\SharePointProject1\bin\Debug\SharePointProject1.wsp

x