Cisco APIC and ACI Installation and Downgrade Guide - Pre-Upgrade Checklists [Cisco Application Policy Infrastructure Controller (APIC)] (2023)

All your APICs are in a fully adequate state

check status onSystem>Panel>Controllerto make sure that the cluster state on all your APICs is set to onetotally fitExpress. If one or more of the APICs are in another state, such asPartially divergent data layer, you must first resolve the state of your APIC cluster.

If your APICs are currently at version 4.2(1) or later, the commandbunch acidificationin each APIC CLI will verify the basics related to APIC clustering for you. Otherwise, follow the initial frame setup in the ACI Second Edition Troubleshooting Guide (http://cs.co/9003ybZ1d)

All your ACI switches are in an active state

CheckKnitting>Inventory>fabric membershipin the APIC GUI to make sure all your ACI switches are on oneActiveExpress. If one or more ACI switches are in another state, such asIdle, Maintenanceand so on, you need to solve these problems first.

• Idle: This means that the switch has fabric detection issues, such as IP reachability from APIC over the ACI infrastructure network. If your switches are currently on version 14.2(1) or later, the commandshow detection problemson the switch, the CLI will verify the basics of discovering the switch fabric for you.

• Maintenance: This means the switch is onmaintenance modethrough the GIR (Graceful Insertion and Removal) operation. This implies that the switch is isolated from the fabric and does not handle most APIC communications, including communications related to updates. You need to bring the switch back to theActivebefore performing an update. If you want to successfully upgrade the switch by first isolating the switches from the network, considerelegant updateinstead of. To seeSmart ACI Switches Upgradefor details.

Compatibility (target ACI version)

Mark aAPIC Upgrade/Rollback Support Matrixfor the supported upgrade paths of your current version.

Compatibility (CIMC Version)

Mark aAPIC Upgrade/Rollback Support Matrixto the supported UCS HUU version for your target APIC version to ensure that all server components are running the supported HUU package version.

Compatibilidade (APIC, Switch Hardware)

See the release notes for bothAPICmiACI switchesto the target version to make sure your hardware is compatible.

Compatibility (remote blade switch)

It is essential to enabledirect traffic forwardingfor remote leaf switches before upgrading to APIC version 5.0(1), as the option becomes mandatory starting with this version.

direct traffic forwardingit can be enabled since APIC version 4.1(2). Note that additional configurations for TEP IP addresses, such as routable subnets or external TEP, may be required to enable this option. This means that if you are running a version earlier than 4.1(2) and have remote leaf switch configured, you will not be able to upgrade directly to version 5.0. In this case, we recommend that you upgrade to a version 4.2, enable direct traffic forwarding, and then upgrade to the desired version 5.0.

See "Upgrading Remote Leaf Switches and Enabling Direct Traffic Forwarding" onCisco APIC Layer 3 Network Configuration GuideFor more information.

Addresses a related issue in "Remote Leaf Switch with Direct Traffic Forwarding Enabled" (CSCvs16767). If you are upgrading to 14.2(2) whiledirect traffic forwardingis enabled for remote leaf nodes, a malfunction (CSCvs16767) which can cause remote leaf nodes to hang due to the Multicast FIB Distribution Manager (MFDM) process. This issue only occurs when column nodes are first upgraded to version 14.2(2), while remote leaf nodes withdirect traffic forwardingthey are still on version 14.1(2). Realisedirect traffic forwardingit was introduced in version 14.1(2).

To avoid this issue, it is critical that you upgrade to version 14.2(3) or later instead of version 14.2(2) whendirect traffic forwardingis enabled.

If you need to upgrade to version 14.2(2) for any reason, you must upgrade your remote secondary nodes first to avoid this issue.

NTP (clocks are synchronized throughout the fabric)

Ensure that NTP is configured on the APICs and switches, along with the required IP range capacity for each individual node's out-of-band (OOB) or in-band (INB) NTP servers.

See the following sections onCisco ACI Troubleshooting - Second Edition:

• In-band and out-of-band management

• Pod Policies: BGP RR/Timestamp/SNMP

Implementation Change for Firmware Update Groups in APIC Version 4.0(1)

As of APIC version 4.0(1), there is only one type of switch upgrade group instead of the two used in previous versions (firmware groups and maintenance groups). By consolidating two pools into one, upgrade setup is simplified. However, when upgrading Cisco APICs from a pre-4.0 release to 4.0(1) or later, you must remove all firmware group and maintenance group policies prior to the upgrade.

• To remove a firmware group policy, go toAdministrator>firmware>De malla node firmware>Grupos de firmware, right-click the firmware group name and chooseDelete firmware group.

• To delete a maintenance group policy, go toAdministrator>firmware>De malla node firmware>maintenance groups, right-click the maintenance group name and chooseDelete maintenance group.

Once the APICs are upgraded to 4.0(1) or later, you can create new switch upgrade groups and upgrade switches from pre-14.0 to 14.0(1) or later.

This only applies when you are upgrading your APICs from a version prior to 4.0 to 4.0(1) or later. Once your APICs are at 4.0(1) or later, you don't need to worry about this for future updates.

Observation

Internally, APICs running version 4.0(1) or later handle switch upgrade groups with the same objects as the old maintenance group policies (such asmaintenance), but with additional attributes. If you are using an API to configure update policies, you should use only APIC maintenance group policies version 4.0(1) or later, without manually creating any firmware group policies, unlike earlier versions.

Settings that should be disabled before updates

The following features must be disabled before updates:

• app center apps

• maintenance mode viaKnitting>Inventory>fabric membership>Maintenance (GIR)

• configuration zone

• Rogue Endpoint (only when running version 14.1(x) or when upgrading to 14.1(x))

Deprecated Administered Objects

The Pre_Upgrade checker script checks for the existence of the following deprecated managed objects in the running version of the software and blocks the upgrade if they exist in the configuration. You must update your script or code to use the new managed object.

• Clase: config:RsExportDestination

• Clase: config:RsImportSource

• Class: fabric:RsResMonFabricPol

• Class: infra:RsResMonInfraPol

• Class: fabric:RsResMonCommonPol

• Class: Trigonometry: On

• Clase: trig:TriggeredWindow

• Clase: fv:CCg

• Clase: fv:RsToCtrct

• Class: mgmt:RsOobEpg

• Clase: mgmt:RsInbEpg

• Clase: vns:RsCIfAtt

• Class: fails: RsHealthCtrlrRetP

• Clase: fv:PndgCtrctCont

• Clase: vz:RsAnyToCtrct

  (Video) ACI Firmware Upgrades

• Clase: fv:PndgCtrctEpgCont

• Class: fv:AREpPUpd

• Clase: vns:Chkr

• Class : aaa : RsFabricSetup

• Class: ap:PluginPol

• Clase: tag:ExtMngdInst

• Class: Telemetry:Server

• Class: Telemetry:FltPolGrp

• Class: Telemetry:FilterPolicy

• Class: Telemetry: FlowServerP

• Class: pol:RsFabricSelfCAEp

• Class: Fabric:PodDhcpServer

• Class: Fabric:SetupAllocP

• Class: Fabric:AssociatedSetupP

• Class: cloud:AEPgSelector

• Clase: fv:VmmSelCont

All switch nodes in vPC

High Availability (HA) is always key in network design. There are several ways to achieve this, such as with server configurations such as NIC teaming, virtualization technology such as VMware vMotion, or network device technology such as link aggregation across chassis. ACI provides high availability by using Virtual Port Channel (vPC) as link aggregation in the chassis.

It is important to keep traffic flowing even during upgrades by upgrading one switch in the same HA pair at the same time. In ACI, it will be a vPC peer unless you have other HA technologies on the server or virtualization side.

The pre-upgrade validator verifies that all switch nodes are in a vPC pair. This check is done when you upgrade APICs instead of switches because in ACI, APICs are upgraded first before switches, and setting up a new vPC pair potentially requires a change in network design and this must be done before any update. If you have other HA technologies installed, you can skip this validation. The vPC is not a requirement for the update to complete, but the built-in tools to prevent leaf switches in a vPC domain from updating at the same time will not work if they are not in a vPC. If you are not using vPC, you must ensure that the switches that are updated do not cause an outage if they are updated at the same time.

APIC Disk Space Usage (F1527, F1528, F1529)

If an APIC is low on disk space for some reason, the APIC update may fail. The APIC will generate three different failures depending on the amount of disk space remaining. If any of these faults occur in the system, the problem must be resolved before performing the update.

• F1527: A warning level failure for APIC disk space usage. This increases when the utilization is between 80% and 85%.

• F1528: A major level failure for APIC disk space usage. This increases when the utilization is between 85% and 90%.

• F1529: A critical level failure for APIC disk space usage. This increases when utilization is between 90% and above.

You can run the followingmockingin the CLI of any APIC to check if these failures exist in the system. The locks are also visible in the GUI. In the following example, with the fouls against/firmware, you can simply remove unnecessary firmware images inAdministrator>firmwarein the APIC GUI. You should not run the linux commandrmto remove an image directly from/firmware, since firmware images are synced between APICs. If the fault is in an unknown disk space, contact Cisco TAC to resolve the issue before upgrading.

Failure Example (F1528 - APIC Disk Space Usage Major Fault)

Below is an example situation where disk space on the/firmwareyou are running out of APIC 1 (node ​​1).

admin@apic1:~>moquery -c failInst -f 'fail.Inst.code=="F1528"'Total objects displayed: 1 # Fault.Instcode : F1528ack : noannotation :cause : equipment-fullchangeSet : available (Old: 5646352, New: 6036744),capUtilized(Old: 86, New: 85),used (old: 33393968, new: 33003576) childAction: created: 2021-05-27T11:58:19.061-04:00 delegate: nodescription: storage drive/firmware on node 1 with hostname apic1 mounted on/firmware is 85% fulldn: topología/pod-1/nodo-1/sys/ch/p-[/firmware]-f-[/dev/mapper/vg_ifc0-firmware]/fault-F1528domain: infraextMngdBy: undefinedhighestSeverity: majorlastTransition: 2021-05- 27T12:01:37.128-04:00lc:raisedmodTs:nuncaocorrer:1origSeverity:maiorprevSeverity:maiorrn:falha-F1528regra:eqpt-storage-full-majorseverity:majorstatus:subject: equipment-fulltype:operacionaluid:

Notice that all three failures look the same except for the percentage of utilization and the severity of the failure.

Using boot flash to change ACI

ACI switches mainly have two different faults about the file system usage of each partition:

• F1820: A minor level failure to use the switch partition. This is generated when the partition utilization exceeds the minor threshold.

• F1821: A fatal level failure to use the switch partition. This is generated when the partition utilization exceeds the main threshold.

The limit for minor and major depends on the partitions. Critical for updates is /bootflash. Bootflash limit is 80% for minors and 90% for majors.

Additionally, a built-in behavior is added to each switch node where it will take steps to ensure that the /bootflash directory maintains 50% capacity. This is specifically to ensure that switch updates can successfully push and pull the switch image during an update.

To do this, there is a built-in script that monitors the usage of /bootflash and if the usage exceeds 50%, it will start deleting files to free up the file system. Given its aggressiveness, there are some edge-case scenarios where this cleanup script could trigger against the switch image it intends to use, which could result in a switch update launching a switch at the loader prompt, since the boot image has been removed. from /bootflash.

To avoid this, check /bootflash before an update and take the necessary steps to understand what is written there and why. Once understood, take steps to clean up unnecessary /bootflash files to ensure there is enough space to avoid the self-cleaning corner case scenario.

The pre-upgrade validator (APIC and application) monitors the F1821 fault, which can detect high utilization of any partition. When this fault is present, we recommend resolving it before upgrading, even if the fault is not bootflash.

The ACI pre-upgrade validation script described earlier in this chapter targets bootflash utilization on each switch specifically to see if there are any issues with bootflash where usage is greater than 50%, which can trigger the ACI pre-upgrade validation script. built-in cleanup script.

You can run the pre-upgrade validator or script to check for this issue. The following is detailed information on bootflash cleanup with a threshold of 50%.

Validation

Once you are logged into a CLI leaf switch, the use of /bootflash can be verified usingdf-h:

hoja1#df-hFilesystem Size Used % Usage Available Mounted on rootfs 2.5G 935M 1.6G 38% /bin/dev/sda4 12G 5.7G 4.9G 54% /bootback/dev/sda2 4,7G 9,6M 4,4G 1% /recovery/dev/mapper/map-sda9 11G 5,7G 4,2G 58% /isan/libnone 3,0G 602M 2,5G 20% /dev/ shmnone 50M 3,4M 47M 7% /etc/dev/sda6 56M 1,3M 50M 3% /mnt/cfg/1/dev/sda5 56M 1,3M 50M 3% /mnt/cfg/0/dev/sda8 15G 140M 15G 1% /mnt/ifc/log/ dev/sda3 115M 52M 54M 50% /mnt/pssnone 1.5G 2.3M 1.5G 1% /tmpnone 50M 240K 50M 1% /var/log/dev/sda7 12G 1.4G 9.3G 13% /logflashnone 350M 54M 297M 16% / var/log/dme/log/dme_logsnone 512M 24M 489M 5% /var/sysmgr/mem_logsnone 40M 4.0K 40M 1% /var/sysmgr/startup-cfgnone 500M 0 500M 0% / volátiles

Verifying automatic removal of /bootflash

If you suspect that the automatic cleanup removed some files inside /bootflash, you can check a log to validate it:

hoja1#egrep "top|removed" /mnt/pss/core_control.log[2020-07-22 16:52:08.928318] Bootflash usage exceeds 50%![2020-07-22 16:52:08.931990] File: MemoryLog.65%_usage removed !![2020-07-22 16:52:08.943914] File: mem_log.txt.old.gz removed !![2020-07 -22 16:52:08.955376] File: libmon.logs removed !![2020-07-22 16:52:08.966686] File: urib_api_log.txt removed !![2020-07-22 16:52:08.977832] File: disk_log.txt removed!![2020-07-22 16:52:08.989102] File: mem_log.txt removed!![2020-07-22 16:52:09.414572] File: aci-n9000-dk9.13.2.1m. garbage removed!!

You can run the following snot from the CLI of any APIC to check the boot flash usage of each switch node.

f2-apic1#moquery -c eqptcapacityFSPartition -f 'eqptcapacity.FSPartition.path=="/bootflash"'Total objects shown: 6# eqptcapacity.FSPartitionname: bootflashavailable: 7214920childAction :dn : topologia/pod-1/es-101/sys/eqptcapacity/fspartition-bootflashmemAlert: normalmodTs: nevermonPolDn: uni/fabric/monfab-defaultpath: /bootflashhrn: fspartition-bootflashstatus:Used: 4320184

MD5sum verification for APIC and firmware change

When performing an upgrade on an ACI fabric, multiple image transfers must be performed to prepare all nodes for the upgrades. Most of these transfers perform first-rate image validation. However, in case of failure, it is worth re-verifying the image on each respective node.

Update Image Transfer Contact Points:

1. Upload the image to your desktop/file server from cisco.com.

   Manually run MD5 on this image. You can validate the expected MD5 of the image at cisco.com.

   

2. Upload the image from your desktop or ftp server to one of the APICs.

   • Watch theDownloading APIC and changing images in APICin the appropriate chapter for instructions on how to perform this operation on APICs:

     • Updating with APIC Versions Before 4.x Using the GUI

     • Upgrading to APIC Versions 4.x or 5.0 Using the GUI

     • Updating with APIC version 5.1 or later using the GUI

   • APIC will automatically perform an image validation and generate an F0058 fault if the image appears to be corrupted or incomplete after the transfer is complete.

3. Once the image is added to the firmware repository, the initially loaded APIC will copy that image to the remaining APICs in the cluster.

   You can manually check the MD5 on each APIC copy of the update image by running thei am md5command against the copy of each APIC in the image.

   For example:

   APIC1#md5sum /firmware/fwrepos/fwrepo/aci-apic-dk9.5.2.1g.binf4c79ac1bb3070b4555e507c3d310826/firmware/fwrepos/fwrepo/aci-apic-dk9.5.2.1g.bin

4. Switches will eventually get a copy of the switch .bin image when preparing to upgrade.

   You can run MD5 on the individual switch image at /bootflash.

   For example:

   hoja1#md5sum /bootflash/aci-n9000-dk9.15.2.1g.bin02e3b3fb45a51e36db28e7ff917a0c96/bootflash/aci-n9000-dk9.15.2.1g.bin

APIC firmware synchronization between APICs

Once the image is downloaded to one of the APICs, the image is synced to all the APICs in the cluster. This is especially critical for APIC images because each APIC needs the image locally to update itself.

For that, you can access each APIC and check/firma/fwrepos/fwrepofor the target image.

If the image is missing one or more APICs, please wait about 5 minutes if right after download. If the image is still missing, make sure the APIC collation state is correct on all APICs and remove the image from the GUI or API (but not with the linux commandrm) and then download the image again to turn file sync back on. If the image is still missing, contact the Cisco TAC.

File system on standby APIC

Because a standby APIC is a cold standby and not part of the cluster, it is not actively monitored for error conditions. Since full file system checks fall under this category, it means that any waiting APICs that show these conditions will not fail and should instead be checked manually.

To do this, you can log into a standby APIC asrescue userso rundf-hto manually check file system usage.

If any file system is at 75% or higher, contact the TAC to identify and remove the condition.

EPG configuration on ports connected to APIC (F0467: port configured for apic)

In a healthy ACI implementation, no EPG policy or implementation should be sent to any interface where an APIC controller is attached. When an APIC is connected to a leaf switch, LLDP validation is performed between the APIC and the leaf switch to allow it to enter the fabric without any user configuration. When a policy is sent to a leaf switch interface that is connected to an APIC, this configuration will be denied and a fault will be generated. However, if the link to the APIC fails for any reason, particularly during an update when the APIC is restarted, the policy can be deployed to that leaf switch interface. This causes the APIC to be unable to re-attach to the structure after reloading.

It is critical that you resolve these issues before upgrading to avoid problems. You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Failure example (F0467: port configured for apic):

The following error shows an example of node 101 eth1/1 that is connected to an APIC that has some EPG configurations.

admin@apic1:~>moquery -c failInst -x 'query-target-filter=wcard(faultInst.descr,"puerto-configurado-para-apic")'Total de objetos mostrados: 1 # Fault.Instcode: F0467ack: noannotation: cause: configuration-failedchangeset:configQual: configured port for apic, configSt: failed to apply, debugMessage:port set to apic: the port is connected to the APIC;,temporaryError:nochildAction :created: 2021-06-03T07:51:42.263-04:00delegated: yesdescr: Installation error foruni/tn-jr/ap-ap1/epg-epg1 o 101 eth1/1due toPort connected to controller, mensaje de limpieza: port-configured-for-apic: Port is connected to the APIC;dn : topology/pod-1/node-101/local/svc-policyelem-id-0/uni/epp/fv-[uni /tn-jr/ap-ap1/epg-epg1]/node-101/stpathat-[eth1/1]/nwissues/fault-F0467dominio: inquilinoextMngdBy: undefinedhighestSeverity: minorlastTransition: 2021-06-03T07:53:52.021-04: 00lc: raisemodTs: neveroccur: 1origSeverity: minorprevSeverity: minorrn: failure-F0467rule: fv-nw-issues-config-failedseverity: minorstatus: subject: managementtype: configuid:

Conflicting L2/L3 interface mode (F0467: port configured as l2, port configured as l3)

This is another type of F0467 family of trouble codes that you should check before an update. This failure alerts that an interface configured to Layer3 Out (L3Out) has failed because the port on which the policy is implemented is operating in the opposite mode. For example, you might have configured a routed subinterface on an L3Out, making the port an L3 port. However, the L2 policy already exists on that port. A port on ACI can be either L2 or L3, but not both, just as a port on any Layer 3 switch can be "switch port” (L2) or “sem switch port(L3), then this policy fails in this situation. The same rule applies if a port is already an L3 port, but you implement the L2 configuration on it. After an upgrade, it is possible that the previous working configuration will be broken if this faulty policy is deployed first after reloading the switch.

It is critical that you resolve these issues before upgrading to avoid problems. The interface on which the fault is generated must be corrected or removed to clear the fault. You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Fault example (F0467: port configured as l2):

The following error shows an example of the L3Out configurationOSPFunder tenant jr it failed on node 101 eth1/7 because the same port is already configured as L2 by other components like EPG or other L3 outlets that use the same port as SVI. This implies that, in this case, L3OutOSPFyou are trying to use node 101 eth1/7 as a routed port or routed subinterface (L3) instead of SVI (L2).

admin@apic1:~>moquery -c FaultDelegate -x 'query-target-filter=wcard(faultInst.changeSet,"port-configured-as-l2")'Total de objetos mostrados: 1 # Fault.Delegateaffected : resPolCont/rtdOutCont/rtdOutDef-[uni/tn-jr/out-OSPF]/node-101/stpathat-[eth1/7]/nwissuescode : F0467ack : nocause : configuration-failedchangeSet : configQual:port configured as l2, configSt: failed to apply, temporaryError: nochildAction: created: 2021-06-23T12:17:54.775-04:00descr: failed delegate:uni/tn-jr/out-OSPF 101 eth1/7 node configuration failed due to interface configured as L2,mensaje de depuración:dn: uni/tn-jr/out-OSPF/fd-[resPolCont/rtdOutCont/rtdOutDef-[uni/tn-jr/out-OSPF]/node-101/stpathat-[eth1/7]/nwissues ] -fault-F0467domain : inquilinohighestSeverity : minorlastTransition :2021-06-23T12:20:09.780-04:00lc : raisemodTs : neveroccur : 1origSeverity : minorprevSeverity : minorrn : fd-[resPolCont/rtdOutCont/rtdOutDef-[uni/tn-jr/ out -OSPF]/node-101/stpathat-[eth1/7]/nwissues]-fault-F0467rule: fv-nw-issues-config-failedseverity: minorstatus: subject: managementtype: config

Fault example (F0467: port configured as l3):

The following technical problem shows an example of the opposite of the above situation. In this example, L3OutputIPv6attempts to use node 101 eth1/7 as an L2 port and fails because other L3 outlets are already using the same port as an L3 port.

admin@apic1:~>moquery -c FaultDelegate -x 'query-target-filter=wcard(faultInst.changeSet,"port-configured-as-l3")'Total de objetos mostrados: 1# Fault.Delegateaffected : resPolCont/rtdOutCont/rtdOutDef-[uni/tn-jr/out-IPV6]/node-101/stpathat-[eth1/7]/nwissuescode : F0467ack : nocause : configuration-failedchangeSet : configQual:port configured as l3, configSt:failed to apply, debugMessage:port-configured-as-l3: Port has one or more Layer 3 subinterfaces;,temporaryError:nochildAction :created : 2021-06-23T12:31:41.949-04:00descr : Delegate de falha:uni/tn-jr/out-IPV6 101 eth1/7 node configuration failed because interface was configured as L3, debug message: port configured as l3: the port has one or more Layer 3 subinterfaces; dn: uni/tn-jr/out-IPV6/fd-[resPolCont/rtdOutCont/rtdOutDef-[uni/tn-jr/out-IPV6]/node-101/stpathat-[eth1/7]/nwissues]-fault- F0467domain : tenanthighestSeverity : minorLoadTransition : 2021-06-23T12:31:41.949-04:00lc : immersionmodTs : neveroccur : 1origSeverity : minorprevSeverity : minorrn: fd-[resPolCont/rtdOutCont/rtdOutDef-[uni/tn-IPVjr]/out; /node-101/stpathat-[eth1/7]/nwissues]-fault-F0467rule: fv-nw-issues-config - failed severity: minor status: issue: management type: configuration

L3Out subnet conflict for contracts (F0467: entry-prefix-already-in-use)

There is another type of F0467 family of trouble codes that you should check before an update. This fault warns that an external EPG defined on a Layer3 Out (L3Out) has a subnet with the "External subnet for external EPG” configured scope that overlaps with another L3Out external EPG on the same VRF. After an upgrade, it is possible that the previous working configuration will be broken if this faulty policy is deployed first after reloading the switch.

It is critical that you resolve these issues before upgrading to avoid unexpected outages when the switches are upgraded. The subnet on which the fault was generated must be corrected or removed to clear the fault. You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Fault example (F0467: prefix entry already in use):

Below is an example of L3OutOSPFwith an external EPG calledall. In this external EPG a subnet L3Out 112.112.112.112/32 is configured with “External subnet for external EPG” in an attempt to classify the source or destination IP address of packets to this external EPG for contract performance. However, it failed because the same subnet is already used by another external EPG on the same VRF.

admin@apic1:~>moquery -c failInst -x'query-target-filter=wcard(faultInst.descr,"prefix-entry-ya-in-use")'Total de objetos mostrados: 1 # Fault.Instcode : F0467ack : noannotation :cause : configuration-failedchangeSet : configQual:entry-prefix-already-in-use, configSt:failed to apply, debugMessage:prefix-entry-already-in-use: Prefix entry sys/ctx-[vxlan-2621440]/pfx-[112.112.112.112/32] is in use;,temporaryError:nochildAction : bred: 2021-06-22T09:02:36.630-04:00delegate: simdescription: configuration failed for uni/tn-jr/out-OSPF/instP-all due to a prefix entry already used in another EPG, debug message: prefix entry already in use: prefix entry sys/ctx-[ vxlan-2621440 ]/pfx-[112.112.112.112/32] is in use;dn: topología/pod-1/nodo-101/local/svc-policyelem-id-0/uni/epp/rtd-[uni/tn-jr/out-OSPF/instP-all]/nwissues/fault-F0467domain: inquilinoextMngdBy: undefinedhighestSeverity: minorlastTransition: 2021-06-22T09:04:51.985-04:00lc: raisemodTs: neveroccur: 1origSeverity: minorprevSeverity: minorrn: failure-F0467rule: fv-nw-issues-config-failedseverity: minorstatus:subject: managementtype: confiado:

Overlapping DB subnets on the same VRF (F0469: Duplicate, F1425: Overlapping subnet)

If, at any point, an overlapping subnet or IP address is implemented in a VRF, this policy will fail and a node-level failure will be generated. However, on upgrade, it is possible that this previous failed configuration is sent to the leaf switch before the previous working configuration. This results in a situation where the known working state before the upgrade is broken after the upgrade and can cause connectivity issues to the previously working subnet.

There are two fails for this situation:

• F0469 (duplicate subnets within ctx) generated when multiple BD subnets with the exact same subnet are configured on the same VRF

• F1425 (subnet overlap) is generated when BD subnets are not equal but overlap

It is critical that you resolve these issues before upgrading to avoid problems. The subnet on which the fault was generated must be corrected or removed to clear the fault. You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Failure example (F0469: duplicate subnets within ctx):

admin@f1-apic1:~>mockery -c falloInst -f 'fallo.Inst.código=="F0469"'Total de objetos mostrados: 4 # Fault.Instcode : F0469ack : noannotation :cause : configuration-failedchangeSet : configQual (Nuevo:duplicate-subnets-within-ctx), configSt(New: application failed), debugMessage(New: uni/tn-TK/BD-BD2,uni/tn-TK/BD-BD1)childAction :created : 2021-07-08T17:40:37.630 -07 : 00 delegate: yesdescription: database configuration failed for uni/tn-TK/BD-BD2 due to duplicate subnets within ctx: uni/tn-TK/BD-BD2, uni/tn-TK/BD-BD1dn: topologia/pod-1/node-101/local/svc-policyelem-id-0/uni/bd-[uni/tn-TK/BD-BD2]-isSvc-no/bdcfgissues/fault-F0469dominio: inquilinoextMngdBy: undefinedhighestSeverity: minorlastTransition: 2021-07-08T17:40:37.630-07:00lc: imersãomodTs: neveroccur: 1origSeverity: minorprevSeverity: minorrn: failure-F0469rule: fv-bdconfig-issues-config-failedseverity: minorstatus:subject: managementtype: configuid:

Fault Example (F1425: Subnet Overlap):

admin@apic1:~>moquery -c failInst -f 'fail.Inst.code=="F1425"'Total Objects shown: 1 # fault.Instcode : F1425ack : noannotation : cause : ip-provisioning-failedchangeSet : ipv4CfgFailedBmp (New: ipv4:Addraddr_failed_flag,ipv4:Addrctrl_failed_flag,ipv4:AddrlcOwn_failed_flag,ipv4:AddrmodTs_failed_flag,ipv4:AddrmonPolDn_failed_flag,ipv4:Addrpref_failed_flag, ipv4: Addrtag_failed_flag, ipv4: Addrtype_failed_flag, ipv4: AddrvpcPeer_failed_flag), ipv4CfgState (Novo: 1), operStQual (Novo:subnet overlap)childAction: created: 2020-02-27T01:50:45.656+01:00 delegate: nodescription: IPv4 address (10.10.10.1/24) is operationally down, reason: subnet overlap on node 101 fabric hostname leaf-101dn: topología/pod-1/node-101/sys/ipv4/inst/dom-jr:v1/if-[vlan10]/addr-[10.10.10.1/24]/fault-F1425domain: accessextMngdBy: undefinedhighestSeverity: majorlastTransition: 2020-02-27T01:52:49.812+01:00lc: raisemodTs: neveroccur: 1origSeverity: majorprevSeverity: majorrn: failure-F1425rule: ipv4-addr-oper-st-downseverity: majorstatus: subject: oper-state-errtype: operatinguid:

APIC SSD Integrity Status (F0101, F2730, F2731, F2732)

Starting with APIC version 2.3(1), failures occur when the SSD media wear indicator (life remaining) is less than a certain percentage on APIC nodes. A short-lived SSD can cause any operation that requires updates to the internal database, such as upgrade or downgrade operations, to fail. The APIC will fail three different times depending on how much life is left on the SSD. If the most critical failure (F2732) occurs in the system, you must replace the SSD by contacting Cisco TAC before upgrading.

• F2730: A warning level failure for the remaining lifetime of the APIC SSD. This is increased when the remaining health is below 10%.

• F2731: A severe level failure for the remaining lifetime of the APIC SSD. This is increased when the remaining health is below 5%.

• F2732: A critical level failure for the remaining life of the APIC SSD. This is increased when the remaining health is below 1%.

Also, on rare occasions, the SSD may have operational issues in addition to its lifespan. In this case, look for the F0101 fault.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

If your APICs are still running on a version prior to version 2.3(1), contact your Cisco TAC to verify the remaining useful life of the SSD.

Watch theAPIC SSD Replacement Tech Notefor more details.

Failure Example (F2731 - APIC SSD Remaining Lifetime Major Failure):

Below is an example of APIC 3 (node ​​3) with 1% remaining SSD life (the major fault F2731). In this case, although the F2732 critical fault for less than 1% life remaining does not increase, it is close enough to the F2732 threshold to recommend replacing the SSD.

APIC1# moquery -c FaultInfo -f 'fault.Inst.code=="F2731"'Total Items Shown: 1# error. Facility Code: F2731ack: no annotation: cause: team wear change set: media wear (old: 2, new: 1) childAction: created: 2019-10-22T11 :47:40.791+01:00delegate:nodescription: Storage unit /dev/sdb on Node 3 mounted on /dev/sdb has 1% life remainingdn: topología/pod-2/nodo-3/sys/ch/p-[/dev/sdb]-f-[/dev/sdb]/fault-F2731dominio: infraextMngdBy: undefinedhighestSeverity: majorlastTransition: 2019-10-22T11: 49: 48.788+01:00lc: raisemodTs: neveroccur: 1origSeverity: majorprevSeverity: majorrn: failure-F2731rule: eqpt-storage-wearout-majorseverity: majorstatus:subject: equipment-wearouttype: operatinguid:

SSD Health Status on ACI Switches (F3074, F3073)

As of version 2.1(4), 2.2(4), 2.3(1o) and 3.1(2m), crashes will occur if SSD flash life usage reaches a certain endurance threshold on leaf switches or column. A short-lived SSD flash can cause any operation that requires internal database updates, such as APIC communication, to fail or the switch to fail to initialize. The ACI switch will generate two different faults based on the amount of SSD life consumed. If the most critical failure (F3073) occurs in the system, you must replace the SSD by contacting Cisco TAC before upgrading.

• F3074: A warning level failure for the lifetime of the SSD switch. This increases when health reaches 80% of its limit.

• F3073: A severe level failure for the life of the switch SSD. This increases when health reaches 90% of its limit.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

If your APICs are still running an older version, contact the Cisco TAC to verify the SSD life status.

Watch theACI Switch Node SSD Lifespan Explained Tech Notefor more details.

Ejemplo de falla (F3074: Switch SSD Lifetime Warning Failure):

Below is an example of node 101 that has reached 85% of its SSD life.

APIC1# moquery -c failInst -f 'fail.Inst.code=="F3074"'Total de objetos mostrados: 4 # Fault.Instcode : F3074ack : noannotation : cause : equipment-flash-warningchangeSet : acc:read-write, cap:61057, deltape:23, descr:flash, gbb:0, id:1, libras: 0,life: 85, majorAlarm:no, mfgTm:2020-09-22T02:21:45.675+00:00, minorAlarm:yes, model:Micron_M600_MTFDDAT064MBF, operSt:ok, peCycles:4290, readErr:0, rev:MC04, ser:MSA20400892, tbw :21.279228, type:flash, vendor:Micron, notice:yes, wlc:0childAction: created: 2020-09-21T21:21:45.721-05:00delegate: noDescription: The SSD has reached 80% life and is approaching its endurance limit. Plan for switch/supervisor replacement soondn: topología/pod-1/node-101/sys/ch/supslot-1/sup/flash/fault-F3074domain: infraextMngdBy: undefinedhighestSeverity: minorlastTransition: 2020-09-21T21:24:03.132-05:00lc: raisemodTs: neveroccur : 1origSeverity : minorprevSeverity : minorrn : falha-F3074rule : eqpt-flash-flash-minor-alarmseverity : minorstatus : subject : flash-minor-alarmtype : operacional

VMM Controller Connectivity (F0130)

If there is a communication problem between the APIC and the VMM controller, the VMM controller status is marked offline and the F0130 fault is generated. Make sure that connectivity between them is re-established before upgrades so that the functions currently implemented on the switches based on communication with the VMM controller are not changed or lost because the APICs cannot retrieve the required information after an upgrade.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Fault example (F0130: VMM controller connection failure):

The following is an example APIC failing to communicate with the VMM controllerMyVMMControllerwith IP 192.168.100.100 in VMM domainLAB_VMM.

apic1# moquery -c failInst -f 'fail.Inst.code=="F0130"'Total de objetos mostrados: 1 # Fault.Instcode: F0130ack: knockout: connect-failedchangeSet: operSt (Antiguo: desconocido,New: offline)childAction:created: 2016-05-23T16:07:50.205-05:00delegate: yesdescription: Connection to VMM controller: 192.168.100.100 with name MyVMMController in data center LAB1 in domain: LAB_VMM fails repeatedlywith error: [Error retrieving ServiceContent from vCenter server 192.168.100.100]. Verify the network connectivity of the VMM controller 192.168.100.100 and verify that the VMM controller user credentials are valid. :04.219-05:00lc : raisemodTs : neveroccur : 1origSeverity : majorprevSeverity : majorrn :failed-F0130rule : comp-ctrlr-connect-failedseverity : majorstatus :subject : controllertype : communicationuid :

Missing LLDP/CDP adjacency between leaf nodes and VMM hypervisors (F606391)

ComOn demandoRigh nowimmediacy of resolution againstadvance provisionin the VMM domain by attaching it to an EPG, for some VMM integrations, such as the VMware DVS integration, the APIC checks the LLDP or CDP information of the leaf switches attached to the hypervisors and also of the VMM controller that manages the hypervisors. This information is required by the leaf switches and hypervisors to dynamically discover the leaf interface connecting to the hypervisor, even when there is an intermediate switch between them, such as Cisco UCS Fabric Interconnect. Once the interface is discovered, APIC dynamically implements VLANs only on the required interfaces of the leaf switch to which the hypervisor is connected.

Prior to APIC version 3.0(1), VLANs used to be removed from leaf interfaces if APIC lost connectivity to the VMM controller because APIC could no longer compare the LLDP or CDP information from the hypervisor's point of view. Starting with APIC version 3.0(1), VLANs are not removed from leaf interfaces even if APIC loses connectivity to the VMM controller to prevent management plane transients from affecting data plane traffic. However, it can cause some rotations in the APIC process by repeatedly trying to get the LLDP/CDP information. When the LLDP/CDP information is missing, the error F606391 is generated.

For these reasons, regardless of APIC version, it is important to resolve this error before upgrading. If a VMM domain configured for Cisco Application Virtual Edge (AVE) fails, LLDP and CDP can be disabled entirely since the control plane created to program the switches is based on the opflex protocol and not LLDP/CDP. When LLDP and CDP are disabled, the faults should be cleared. The settings for changing the LLDP/CDP state for a VMM domain are defined in the vSwitch policy for the VMM domain.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Failure example (F606391: Missing LLDP/CDP adjacency for hypervisors):

apic1# moquery -c failInst -f 'fail.Inst.code=="F606391"'Total objects shown: 5# error. Facility Code: F606391ack: no annotation: cause: fsm-failedchangeSet: childAction: created: 2019-07-18T01:17:39.435+08:00delegate: yesdescr:[FSM: FAIL]: Obtain LLDP/CDP adjacency information for physical adapters on host: hypervisor1.cisco.com(TASK:ifc:vmmmgr:CompHvGetHpNicAdj)dn: comp/prov-VMware/ctrlr-[LAB_VMM]-MyVMMController/hv-host-29039/fault-F606391domain: infraextMngdBy: undefinedhighestSeverity: majorlastTransition: 2019-07-18T01:17:39.435+08:00lc: raisemodTs: neveroccur: 1origSeverity: majorprevSeverity: majorrn: falha-F606391reget-hpsm-fsm-hp -nic-adj-fsm-failseverity: majorstatus: assunto: tarefa-ifc-vmmmgr-comp-hv-get-hp-nic-adjtype: configuid:

Dissimilar Infrastructure VLAN Injection via LLDP (F0454 - Infrastructure VLAN Mismatch)

If you have parallel-connected interfaces between two different ACI fabrics, you must disable LLDP on those interfaces before upgrades. This is because when the switch comes back up after the upgrade, it can receive and process LLDP packets from the other fabric that may be using a different infrastructure VLAN. If this happens, the switch will incorrectly try to discover itself through the other fabric's infrastructure VLAN and will not discover itself in the correct fabric.

Could not detect if an ACI switch is receiving an LLDP packet with infrastructure VLAN mismatch from other fabrics.

You can run the below rung in the CLI of any APIC to check if there is a system failure.

Fault example (F0454: LLDP with incompatible parameters):

apic1#mockery -c falloInst -f 'fallo.Inst.código=="F0454"'Total de objetos mostrados: 2 # Fault.Instcode: F0454ack: noalert: noannotation:cause: wire-check-failedchangeSet: wireIssues (Novo: ctrlr-uuid-mismatch,fabric-domain-mismatch,infra-ip-mismatch,infrastructure vlan mismatch)childAction:created: 2021-06-30T10:44:25.576-07:00delegate: nodescription: port eth1/48 is down due to controller UUID mismatch, fabric domain name mismatch, infrastructure subnet mismatch, infrastructure vlan mismatchdn: topología/pod-1/node-104/sys/lldp/inst/if-[eth1/48]/fault-F0454--- recorte ---

Programming CAM policies for contracts (F3545)

The F3545 fault occurs when the switch is unable to activate a contract rule (zoning rule) due to a hardware or software programming fault. If you see this, then the CAM policy is full and no more contracts can be deployed to the switch, and different sets of contracts can be deployed after reboot or upgrade. This can cause services that were working before an update to start failing after an update. Note that the same failure could occur for other reasons, such as an unsupported filter type in the contracts instead of the CAM usage of the policy. For example, first generation ACI switches support IP EtherType, but not IPv4 or IPv6 in contract filters. When this fault is present, check theOperations>capacity panel>sheets capacityno APIC GUI for CAM policy use.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Fault Example (F3545: Zoning Rule Programming Fault):

The following is an example of a node 101 schedule failure for contract rules 266 (zoneRuleFailed). While it also shows the L3Out subnets scheduling failure (pfxRuleFailed) in the changeset, a separate F3544 fault is generated for this.

apic1# moquery -c failInst -f 'fail.Inst.code=="F3545"'Total objects shown: 1 # failed. Instcode: F3545ack: no annotation: cause: actrl-resource-unavailablechangeSet: pfxRuleFailed (New: 80),zoneRuleFailed (Novo: 266)childAction: created: 2020-02-26T01:01:49.256-05:00 delegate: nodescr: 266 number of rules failed on sheet1dn: topology/pod-1/node-101/sys/actrl/dbgStatsReport/fault-F3545domain: infraextMngdBy: undefinedhighestSeverity: majorlastTransition: 2020-02-26T01:03:59.849-05:00lc: raisemodTs: neveroccur: 1origSeverity: majorprevSeverity: majorrn: falha-F3545rule: actrl-stats-report-zone-regra-prog-failedseverity: majorstatus: assunto: hwprog-failedtype: operacionaluid:

L3Out Subnet Programming for Contracts (F3544)

The F3544 fault occurs when the switch is unable to enable an input to assign a prefix toPCLabeldue to a hardware or software programming failure. These entries are configured for L3Out subnets with the "External subnets for the external EPG” under an external EPG on an L3Out and is used to assign L3Out subnets to EPG L3Out. If you see this due to host routes or LPM capability on the switch, this switch may trigger different sets of entries after reboot or upgrade. This can cause services that were working before an update to start failing after an update. When this fault is present, check theOperations>capacity panel>sheets capacityin the APIC GUI to use LPM routes and /32 or /128.

You can run the below rung in the CLI of any APIC to check for these faults in the system. The locks are also visible in the GUI.

Fault example (F3544: L3Out subnet programming failure):

The following is an example of Node 101 failing to schedule 80 L3Out subnets with "External Subnets to External EPG" (pfxRuleFailed). While it also shows the scheduling failure of the contracts themselves (zoneRuleFailed) in the changeset, a separate F3545 fault is generated for this.

apic1# moquery -c failInst -f 'fail.Inst.code=="F3544"'Total de objetos mostrados: 1 # Fault.Instcode : F3544ack : noannotation : cause : actrl-resource-unavailablechangeSet :pfxRuleFailed (New: 80), zoneRuleFailed (New: 266) childAction: created: 2020-02-26T01:01:49.246-05:00 delegate: nodescr: 80 prefix numbers failed on sheet 1dn: topology/pod-1/node-101/sys/actrl/dbgStatsReport/fault-F3544domain: infraextMngdBy: undefinedhighestSeverity: majorlastTransition: 2020-02-26T01:03:59.849-05:00lc: raisemodTs: neveroccur: 1origSeverity: majorprevSeverity: majorrn: falha-F3544rule: actrl-stats-report-pre-fix-prog-failedseverity: majorstatus: assunto: hwprog-failedtype: operacionaluid:

General scalability limits

Check the capacity panel atOperations>capacity panelin the APIC GUI to ensure that no capacity exceeds its limit. Exceeding the limit can cause inconsistency in the resources that are deployed before and after an update, as notedProgramming CAM policies for contracts (F3545)miL3Out Subnet Programming for Contracts (F3544).

We recommend that you consult the Capacity Table of each switch throughOperations>capacity panel>sheets capacitybecause those are typically the hardware limit rather than a software qualified limit. For example, the number of endpoints such as MAC (learned), IPv4 (learned), CAM policy, LPM, host routes, etc.

Overlapping VLAN Set

Overlapping VLAN blocks in different VLAN groups can lead to some forwarding issues, such as:

• Packet loss due to terminal learning problems

• Spanning Tree Loop Due to BPDU Forwarding Domains

  (Video) Generating a TacOutput File on the ACI APIC

These problems can appear suddenly after updating your switchesbecause switches get policies from scratch after an upgrade and can apply the same VLAN ID from a different group than the one used before the upgrade. As a result, the VLAN ID is assigned to a different VXLAN VNID than other switch nodes. This causes the two problems mentioned above.

It is critical to ensure that there are no overlapping VLAN groups in your fabric, unless you have a purpose with proper understanding of VLAN ID and VXLAN ID mapping behind the scenes. If you are not sure, considerApply EPG VLAN validationlowSystem>system settings>Wide weave configurationin the APIC GUI [available starting with version 3.2(6)], which avoids the most common problematic configuration (two domains containing overlapping VLAN groups associated with the same EPG).

Please refer to the following documents to understand how overlapping VLAN groups become a problem and when this scenario can occur:

• VLAN group overlap Main burst packet drop to VPC endpoints and spanning tree loop

• ACI: Common Migration Issue/Overlapping VLAN Teaming

• Overlapping VLAN ValidationnoCisco APIC Layer 2 Network Configuration Guide, Release 4.2(x)

MTU L3Out discrepancy

It is critical to ensure that the MTU values ​​on the ACI L3Out interfaces and the routers connecting to them match. Otherwise, when the ACI switch boots after an upgrade, it can cause problems during routing protocol neighbor establishment or route information exchange between peers.

See below for examples of details about each protocol.

BGP is the protocol that would establish the session regardless of the MTU. BGP "Open and Set" messages are small, but the messages to exchange routes can be huge.

OSPF will not be able to form neighbors if the MTU of both ends of the link do not match. However, although it is not strongly recommended, if the side with a higher MTU is configured to ignore the MTU and enable the OSPF neighbor, the OSPF neighbor will be formed.

During an edge leaf switch upgrade, routing sessions will be interrupted. When an edge leaf switch comes online with a new version, it activates the routing peer. After that, when it starts exchanging routing prefix information, it will generate frames with possibly higher payload. Depending on the size of the table, the upgrade may require a larger frame size that won't reach the other side. The size of this payload will depend on the local MTU. If the MTU on the other side does not match (if it is less than the local MTU size), these exchanges will fail, resulting in routing problems.

You can check and set the MTU on the L3Out interfaces viaTenant>the net>L3Fuera>Logical No Profile>Logical interface profile>Select interface type.

egrep “dn|encap|mtu”

In this example, an L3Out interface with VLAN 2054 is configured with MTU 9000 on the tenant.traditional knowledge, L3FueraOSPF, Logical node profileOSPF_Node Profileand logical interface profileOSPF interface profile.

apic1#moquery -c l3extRsPathL3OutAttTotal objects shown: 1# l3ext.RsPathL3OutAttaddr: 20.54.0.1/24 --- snip ---dn: uni/tn-TK/out-OSPF/lnodep-OSPF_nodeProfile/lifp-OSPF_interfaceProfile/rspathL3OutAtt-[topology/pod-1/routes-101/pathep-[eth1/12]]encap: vlan-2054 --- snip ---persona: 9000--- clipping ---

Alternatively, you can runtecido <node_id> shows an interfaceon its edge leaf nodes as well.

If the MTU showsyou inherit, the value is inherited fromKnitting>mesh policies>Policies>Global>Tecido L2 MTU>Pattern.

The script provided in this chapter checks the MTU of all L3Out interfaces for you. However, you must run the script on the APIC, and the APIC does not have visibility of the MTU value configured on the connected devices. Therefore, you need to manually check the MTU on the connected devices.

BGP L3Out Peer Connectivity Profile in a Non-Loopback Node Profile

Before upgrading to version 4.1(2) or later, you must ensure that one of the following two requirements is met:

• A node profile with a BGP peer connectivity profile has a loopback configured for all switches in the profile, or

• BGP peer connectivity profiles are configured per interface.

You can configure the BGP peer connectivity profile per node profile or per interface. The first is to generate the BGP session from a loopback, while the second is to generate the BGP session from each interface.

Prior to version 4.1(2), when a BGP peer connectivity profile is configured on a node profile without configuring a loopback, the APIC uses another available IP address on the same Edge Leaf switch in the same VRF as the source BGP, like loopback. IP address of another L3Out or an IP address configured for each interface. This has the risk that the BGP source IP address is accidentally changed on reboots or upgrades. This behavior has changed based onCSCvm28482and ACI no longer establishes a BGP session through a BGP peer connectivity profile in a node profile when a loopback is not configured in the node profile. Instead, an F3488 fault is generated in these situations. This fault cannot be used as a pre-update check because it is only generated after an update.

Due to this change, when upgrading from a previous version to version 4.1(2) or later, a BGP session is no longer established if the session was generated through a BGP peer connectivity profile on a node profile and not a loopback is configured in the node profile.

When multiple interfaces on the same node profile need to establish a BGP peer with the same peer IP, you can try configuring a BGP peer connectivity profile on a non-loopback node profile so that the same BGP peer configuration is applied, such as an alternative due to the lack of loopback, for each interface in the same node profile. This is because a BGP peer connectivity profile with the same peer IP address cannot be configured on multiple interface profiles within the same node profile. This limitation has been loosened based onCSCvw88636in 4.2(7f). Until then, for this specific requirement, you must configure one node interface per interface profile, and configure the BGP peer connectivity profile on each interface profile on a different node profile.

L3Out Incorrect route map direction (CSCvm75395)

Before upgrading to version 4.1(1) or later, you must ensure that your roadmap (route profile) settings are correct.

defaultCSCvm75395, the following configuration may have worked before version 4.1(1) despite the incorrect configuration (a misconfiguration in the address):

• A road map withto exportaddress attached to an L3Out subnet withImport Route Control Subnet

• A road map withto importaddress attached to an L3Out subnet withExport Route Control Subnet

Where L3Out subnet means the subnet configured in an external EPG on an L3Out.

However, these misconfigurations will no longer work, which is the expected behavior, after upgrading the fabric to version 4.1(1) or later.

Although this method is not the most common or recommended among many ways to control the routes advertised or learned by ACI L3Outs, the correct configuration with this method should be as follows:

• A road map withto exportaddress attached to an L3Out subnet withExport Route Control Subnet

• A road map withto importaddress attached to an L3Out subnet withImport Route Control Subnet

Or, alternatively, you can follow the recommended settings below to control rerouting in L3Outs:

• standard exportroute map with IP prefix lists

• standard importroute map with IP prefix lists

With this configuration, you do not needExport Route Control SubnetoImport Route Control Subneton external EPGs, and you can have external EPGs dedicated to contracts or route leaks while having full control of the routing protocol via route maps like a normal router.

Also note that any import address roadmaps will only take effect whenRoute Control Executionis enabled to import inTenant>the net>L3Fuera>main profile. Otherwise, everything is imported (learned) by default.

EP announces version mismatch (CSCvi76161)

If the current version of the ACI switch is earlier than 12.2(4p) or 12.3(1) and you are upgrading to version 13.2(2) or later, it is susceptible to a defectCSCvi76161, where a version mismatch between Cisco ACI leaf switches can cause the EPM process to receive an unexpected EP advertisement message on the leaf switch, resulting in an EPM failure and switch reload.

To avoid this issue, it is critical that you upgrade to a fixed version of CSCvi76161 before upgrading to 13.2(2) or later.

• For fabrics running an ACI switch version earlier than 12.2(4p), first upgrade to version 12.2(4r) and then upgrade to the desired target version.

• For fabrics running an ACI switch version 12.3(1), first upgrade to version 13.1(2v) and then upgrade to the desired target version.

Intersight Device Connector is being updated

If you initiate an APIC upgrade while an Intersight Device Connector (DC) upgrade is in progress, the DC upgrade might fail.

You can check the status of Intersight DC atSystem>system settings>intervention. If the domain controller upgrade is in progress, wait a minute and try the APIC upgrade again. The Intersight Device Connector update typically takes less than a minute.