Cisco DNA Upgrade Issues – Application Update Stuck

After initiating Cisco DNA Appliance version 2.1.2.4 and starting an upgrade towards 2.2.2.8 in order to get to 2.2.3.5 I got a strange issue where the appliance system update went fine but the switch to 2.2.2.8 was disabled until Application Updates did not finish.

The real issue here was that Application Updates of Cloud Connectivity – Data Hub got stuck on 12% for 4 days without timing out or finishing. Tried several appliance reboots from CIMC which didn’t help.

Below are the steps that helped sort out Application Updates issues with container pods being stuck at the point of pooling images from the cloud and creation on the Cisco DNA appliance. Of course, this was a long troubleshoot and here I listed only maglev CLI commands that were useful.

I SSH into the appliance and started digging the maglev CLI tool.

#ssh -l maglev 10.10.10.10 -p 2222

I listed all Application Updates and it was visible that there are multiple applications that waited to be downloaded and installed and there was also my Cloud Connectivity – Data Hub app stuck on its 12% of completion

$ maglev catalog package display

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO      
--------------------------------------------------------------------------------------------------------
access-control-application Access Control Application               2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
ai-network-analytics      AI Network Analytics                     2.6.9.455       PARTIAL    Package needs to be pulled/downloaded
app-hosting               Application Hosting                      1.6.6.2112161504 ERROR      Unable to update ManagedService=app-hosting/database/postgres:1.6.1. LibraryServiceBundle=postgres:1.6.1 does not exist in repository 'None'
application-policy        Application Policy                       2.1.368.170003  READY                
application-registry      Application Registry                     2.1.368.170003  READY                
application-visibility-service Application Visibility Service           2.1.368.170003  PARTIAL    Package needs to be pulled/downloaded
assurance                 Assurance - Base                         2.2.2.485       PARTIAL    Package needs to be pulled/downloaded
automation-core           NCP - Services                           2.1.368.60015   READY                
base-provision-core       Automation - Base                        2.1.368.60015   READY                
cloud-connectivity-contextual-content Cloud Connectivity - Contextual Content  1.3.1.364       READY                
cloud-connectivity-data-hub Cloud Connectivity - Data Hub            1.6.0.380       PARTIAL    ==> [ 12%]
cloud-connectivity-tethering Cloud Connectivity - Tethering           2.12.1.2        READY                
cloud-provision-core      Cloud Device Provisioning Application    2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
command-runner            Command Runner                           2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
device-onboarding         Device Onboarding                        2.1.368.60015   READY                
disaster-recovery         Disaster Recovery                        2.1.367.360196  PARTIAL    Package needs to be pulled/downloaded
dna-core-apps             Network Experience Platform - Core       2.1.368.60015   READY                
dnac-platform             Cisco DNA Center Platform                1.5.1.180       READY                
dnac-search               Cisco DNA Center Global Search           1.5.0.466       PARTIAL    Package needs to be pulled/downloaded
endpoint-analytics        AI Endpoint Analytics                    1.4.375         PARTIAL    Package needs to be pulled/downloaded
group-based-policy-analytics Group-Based Policy Analytics             2.2.1.401       PARTIAL    Package needs to be pulled/downloaded
icap-automation           Automation - Intelligent Capture         2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
image-management          Image Management                         2.1.368.60015   READY                
machine-reasoning         Machine Reasoning                        2.1.368.210017  PARTIAL    Package needs to be pulled/downloaded
ncp-system                NCP - Base                               2.1.368.60015   ERROR      Unable to update ManagedService=fusion/database/postgres:1.6.1. LibraryServiceBundle=postgres:1.6.1 does not exist in repository 'None'
ndp-base-analytics        Network Data Platform - Base Analytics   1.6.1028        PARTIAL    Package needs to be pulled/downloaded
ndp-platform              Network Data Platform - Core             1.6.596         ERROR      Unable to update ManagedService=ndp/database/elasticsearch:1.7.4. LibraryServiceBundle=elasticsearch:1.7.4 does not exist in repository 'None'
ndp-ui                    Network Data Platform - Manager          1.6.543         PARTIAL    Package needs to be pulled/downloaded
network-visibility        Network Controller Platform              2.1.368.60015   READY                
path-trace                Path Trace                               2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
platform-ui               Cisco DNA Center UI                      1.6.2.446       READY                
rbac-extensions           RBAC Extensions                          2.1.368.1910001 PARTIAL    Package needs to be pulled/downloaded
rogue-management          Rogue and aWIPS                          2.2.0.51        PARTIAL    Package needs to be pulled/downloaded
sd-access                 SD Access                                2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
sensor-assurance          Assurance - Sensor                       2.2.2.484       PARTIAL    Package needs to be pulled/downloaded
sensor-automation         Automation - Sensor                      2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
ssa                       Stealthwatch Security Analytics          2.1.368.1091226 PARTIAL    Package needs to be pulled/downloaded
system                    System                                   1.6.594         READY                
system-commons            System Commons                           2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
umbrella                  Cisco Umbrella                           2.1.368.592015  PARTIAL    Package needs to be pulled/downloaded
wide-area-bonjour         Wide Area Bonjour                        2.4.368.75006   READY  

After going deeper in detail I listed the status of service containers of Cloud Connectivity – Data Hub and there it was visible that not all pods were running. It was as if some dependencies were missing from other services (the list above) or something just got stuck and didn’t try again to recreate those container pods.

$ maglev catalog package status cloud-connectivity-data-hub

KIND                 RESOURCE                                      STATE      MESSAGE                                 
----------------------------------------------------------------------------------------------------------------------
AppStack             dxhub:1.6.0.380                               PARTIAL    One or more child resources are not available yet
Package              cloud-connectivity-data-hub:1.6.0.380         PARTIAL    One or more child resources are not available yet
ServiceBundle        dxhub/dxhub-api-proxy:1.6.0.380               PARTIAL    Pulling artifacts for service dxhub-api-proxy (pulled=0, remaining=1)
ServiceBundle        dxhub/dxhub-registry:1.6.0.380                PARTIAL    Pulling artifacts for service dxhub-registry (pulled=0, remaining=1)

The next command that I found useful here was “maglev catalog package edit cloud-connectivity-data-hub” which ended up being a way to edit K8S manifest for those PODs running Cloud Connectivity – Data Hub service.

I searched the manifest listed below and found at the end the setting which stated: “requiresPull: false”. Didn’t have anything to lose here so I just edited that part: “requiresPull: true”.

$ maglev catalog package edit cloud-connectivity-data-hub
capabilityStatus:
  modified: 1653544689.9558897
_dependsOn:
  capabilities: []
_labels:
  release:
  - dnac:2.1.2.7
  - dnac:2.1.2.8
_manifestCount: 4
_provides:
  capabilities:
  - level: 2
    minLevel: 0
    name: dxhub:registry
  - level: 2
    minLevel: 0
    name: dxhub:api-proxy
_pullStatus:
  artifactPullCompleted: false
  bytesToDownload: 20606255
  manifestPullCompleted: true
  manifestsDownloaded: 2
  memberId: catalogserver-77dc8dcdbc-t4mmk
  modified: 1653544610.1903415
  progress: 12
  timestamps:
    pullStart: 1653544505.7862234
_size: 20606255
appstacks:
- dxhub:1.6.0.380
description: DxHub provides APIs to create streams, subscribe to streams, publish
  messages to streams and consume messages from the streams.
displayName: Cloud Connectivity - Data Hub
kind: Package
labels:
  category: Cisco DNA Center Core
  team: dxhub
manifestVersion: v4
name: cloud-connectivity-data-hub
requiresPull: false
status:
  message: '[dxhub/dxhub-registry:1.6.0.380] Pulling artifacts for service dxhub-registry
    (pulled=0, remaining=1)'
  state: PARTIAL
version: 1.6.0.380
  modified: 1653544689.9558897
_dependsOn:
  capabilities: []
_labels:
  release:
  - dnac:2.1.2.7
  - dnac:2.1.2.8
_manifestCount: 4
_provides:
  capabilities:
  - level: 2
    minLevel: 0
    name: dxhub:registry
  - level: 2
    minLevel: 0
    name: dxhub:api-proxy
_pullStatus:
  artifactPullCompleted: false
  bytesToDownload: 20606255
  manifestPullCompleted: true
  manifestsDownloaded: 2
  memberId: catalogserver-77dc8dcdbc-t4mmk
  modified: 1653544610.1903415
  progress: 12
  timestamps:
    pullStart: 1653544505.7862234
_size: 20606255
appstacks:
- dxhub:1.6.0.380
description: DxHub provides APIs to create streams, subscribe to streams, publish
  messages to streams and consume messages from the streams.
displayName: Cloud Connectivity - Data Hub
kind: Package
labels:
  category: Cisco DNA Center Core
  team: dxhub
manifestVersion: v4
name: cloud-connectivity-data-hub
requiresPull: false
status:
  message: '[dxhub/dxhub-registry:1.6.0.380] Pulling artifacts for service dxhub-registry
    (pulled=0, remaining=1)'
  state: PARTIAL
version: 1.6.0.380

After that everything started to work again. There were multiple other applications that started downloading themselves which was visible in maglev CLI and also on WEB UI.

$ maglev catalog package delete cloud-connectivity-data-hub

NAME                      DISPLAY_NAME                             VERSION         STATE      INFO      
--------------------------------------------------------------------------------------------------------
access-control-application Access Control Application               2.1.368.60015   READY                
ai-network-analytics      AI Network Analytics                     2.6.9.455       READY                
app-hosting               Application Hosting                      1.6.6.2112161504 READY                
application-policy        Application Policy                       2.1.368.170003  READY                
application-registry      Application Registry                     2.1.368.170003  READY                
application-visibility-service Application Visibility Service           2.1.368.170003  READY                
assurance                 Assurance - Base                         2.2.2.485       READY                
automation-core           NCP - Services                           2.1.368.60015   READY                
base-provision-core       Automation - Base                        2.1.368.60015   READY                
cloud-connectivity-contextual-content Cloud Connectivity - Contextual Content  1.3.1.364       READY                
cloud-connectivity-data-hub Cloud Connectivity - Data Hub            1.6.0.380       PARTIAL    Package needs to be pulled/downloaded
cloud-connectivity-tethering Cloud Connectivity - Tethering           2.12.1.2        READY                
cloud-provision-core      Cloud Device Provisioning Application    2.1.368.60015   READY                
command-runner            Command Runner                           2.1.368.60015   READY                
device-onboarding         Device Onboarding                        2.1.368.60015   READY                
disaster-recovery         Disaster Recovery                        2.1.367.360196  READY                
dna-core-apps             Network Experience Platform - Core       2.1.368.60015   READY                
dnac-platform             Cisco DNA Center Platform                1.5.1.180       READY                
dnac-search               Cisco DNA Center Global Search           1.5.0.466       PARTIAL    Package needs to be pulled/downloaded
endpoint-analytics        AI Endpoint Analytics                    1.4.375         READY                
group-based-policy-analytics Group-Based Policy Analytics             2.2.1.401       READY                
icap-automation           Automation - Intelligent Capture         2.1.368.60015   READY                
image-management          Image Management                         2.1.368.60015   READY                
machine-reasoning         Machine Reasoning                        2.1.368.210017  READY                
ncp-system                NCP - Base                               2.1.368.60015   READY                
ndp-base-analytics        Network Data Platform - Base Analytics   1.6.1028        PARTIAL    ======>                        [ 25%]
ndp-platform              Network Data Platform - Core             1.6.596         PARTIAL    ==========>                    [ 38%]
ndp-ui                    Network Data Platform - Manager          1.6.543         PARTIAL    Package needs to be pulled/downloaded
network-visibility        Network Controller Platform              2.1.368.60015   READY                
path-trace                Path Trace                               2.1.368.60015   PARTIAL    =====================>         [ 74%]
platform-ui               Cisco DNA Center UI                      1.6.2.446       READY                
rbac-extensions           RBAC Extensions                          2.1.368.1910001 PARTIAL    Package needs to be pulled/downloaded
rogue-management          Rogue and aWIPS                          2.2.0.51        PARTIAL    Package needs to be pulled/downloaded
sd-access                 SD Access                                2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
sensor-assurance          Assurance - Sensor                       2.2.2.484       PARTIAL    Package needs to be pulled/downloaded
sensor-automation         Automation - Sensor                      2.1.368.60015   PARTIAL    Package needs to be pulled/downloaded
ssa                       Stealthwatch Security Analytics          2.1.368.1091226 PARTIAL    Package needs to be pulled/downloaded
system                    System                                   1.6.594         READY                
system-commons            System Commons                           2.1.368.60015   READY                
umbrella                  Cisco Umbrella                           2.1.368.592015  PARTIAL    Package needs to be pulled/downloaded
wide-area-bonjour         Wide Area Bonjour                        2.4.368.75006   READY   

For the problematic cloud-connectivity-data-hub status now was as if it wasn’t downloaded yet and waiting for the pool from the cloud to begin. It was just needed to initiate it:

 
$ maglev catalog package pull cloud-connectivity-data-hub

NAME                      DISPLAY_NAME                             VERSION         STATE     
---------------------------------------------------------------------------------------------
cloud-connectivity-data-hub Cloud Connectivity - Data Hub            1.6.0.380       PARTIAL   

Use maglev catalog package status '[:]'" to monitor the progress of the operation

The output after this command showed that the service started to pull container images and deploy the pods successfully after a few moments:

 
$ maglev catalog package display status cloud-connectivity-data-hub

KIND                 RESOURCE                                      STATE      MESSAGE                                 
----------------------------------------------------------------------------------------------------------------------
AppStack             dxhub:1.6.0.380                               PARTIAL    One or more child resources are not available yet
Package              cloud-connectivity-data-hub:1.6.0.380         PARTIAL    One or more child resources are not available yet
ServiceBundle        dxhub/dxhub-api-proxy:1.6.0.380               PARTIAL    Building container image for 'dxhub-api-proxy'
ServiceBundle        dxhub/dxhub-registry:1.6.0.380                PARTIAL    Building container image for 'dxhub-registry'
 
$ maglev catalog package status cloud-connectivity-data-hub

KIND                 RESOURCE                                      STATE      MESSAGE                                 
----------------------------------------------------------------------------------------------------------------------
AppStack             dxhub:1.6.0.380                               READY                                              
Package              cloud-connectivity-data-hub:1.6.0.380         READY                                              
ServiceBundle        dxhub/dxhub-api-proxy:1.6.0.380               READY                                              
ServiceBundle        dxhub/dxhub-registry:1.6.0.380                READY  

After all this, Cisco DNA appliance UI also showed that all components were downloaded successfully and the Upgrade process could be finished with the Update All option

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: