aci-sim/docs/DESIGN.md

aci-sim — DESIGN

A software simulator that faithfully impersonates the REST APIs of a 2-site Cisco ACI fabric (two per-site APICs + one ND/NDO), driven by an editable topology.yaml, so that pointing local autoACI at it makes Topology, NDO views, all query plugins, and Fabric Build render internally-consistent, believable control- + data-plane data.

The contract we emulate is in docs/CONTRACT.md (authoritative — trust it, don't re-explore autoACI).

Locked decisions (owner-approved)

1. Generic MIT engine — model each APIC as a real Managed Information Tree (DN-keyed objects, parent/child) with a query resolver implementing APIC's class/mo/node-scoped + rsp-subtree/target-subtree-class + eq/wcard/and/or + pagination semantics. One YAML → full object graph; every current AND future autoACI query works with no per-class code.
2. Accept-reflect writes + reset/snapshot — POST /api/mo and NDO schema/deploy upsert into the live MIT so read-back + Topology show the change (enables push→verify playbook testing). In-memory, with reset (to baseline) and snapshot/restore.
3. Static deterministic snapshot — stable, internally-consistent, reproducible data (no liveness engine in v1). Deterministic so views are easy to verify.
4. Configurable topology — YAML parametrizes spine/leaf counts; leaves can be added/removed (statically via YAML+reload, dynamically via the add_leaf_switch_pair playbook's fabricNodeIdentP write reaction). Border-leaf vPC → CSW, 2 sites + ISN, full L3Out subtree.
5. Scope — ACI + NDO REST surface (~22 REST playbooks fully testable + the ACI half of migration). NX-OS SSH is OUT of v1 (extension seam left, no auth stub).
6. Stack — Python 3.11+, FastAPI + uvicorn + Pydantic v2, PyYAML, httpx (tests). Self-signed TLS on each host (autoACI uses verify=False). Matches autoACI's stack.

Process / host layout

One process, three HTTPS apps on three localhost ports (self-signed certs in certs/): APIC Site-A :8443, APIC Site-B :8444, NDO :8445. Each site APIC serves its own MIT; both sites are built from one global topology so stretched tenants + ISN peering are consistent. The NDO app derives its schema model from the same topology (same tenant/VRF/BD names). Point autoACI: Site-A→127.0.0.1:8443, Site-B→127.0.0.1:8444, NDO→127.0.0.1:8445.

Module map + interfaces (the contracts subagents implement against)

mit/ — object store (foundation; no deps)

• mo.py: MO = {class_name:str, attrs:dict[str,str], children:list[MO]}; attrs["dn"] is the key. Helpers: MO(cls, **attrs), .dn, .add_child(mo), .flat() (self+descendants).
• dn.py: bracket-aware DN utils — dn_split(dn)->list[str] (never split inside []), parent_dn(dn), rn_of(dn), dn_class_hint, pod_from_dn, name_from_dn.
• store.py: MITStore — add(mo), upsert(mo) (merge attrs + recurse children; honor status=deleted), get(dn)->MO|None, children(dn, classes=None), subtree(dn, classes=None), by_class(cls)->list[MO], all(); maintains a dn→MO map and a parent→[child dn] index. Deterministic iteration order (insertion order).

query/ — ACI query engine (deps: mit)

• filters.py: parse eq/wcard/and/or(...) strings → predicate fn(MO)->bool. Bracket/quote aware.
• engine.py: run_class_query(store, cls, params), run_mo_query(store, dn, params), run_node_scoped(store, node_dn, cls, params). Params object carries filter, subtree mode (rsp-subtree children|full or legacy query-target=subtree), subtree classes, page, page_size. Returns (imdata:list[dict], total:int) in the exact {cls:{attributes,children}} shape. Two subtree shapes (must match real APIC + autoACI): (a) legacy query-target=subtree (autoACI's query_dn(subtree=True)) → descendants FLAT as top-level imdata, filtered by target-subtree-class (root included only if it matches); consumers read the target class at the top level, never recurse into children. (b) modern rsp-subtree=children|full (autoACI's query_class(subtree=…)) → matched objects with NESTED children (children=one level, full=recursive). Pagination slices the result set; total = pre-pagination count.

topology/ — the YAML contract (deps: none)

• schema.py: Pydantic v2 models for the whole topology (see topology.yaml + §Topology schema below). Validates counts, references (bd→vrf, epg→bd, l3out→vrf), site/ISN config. Fail loudly on bad refs. Also validates (PR-8, findings #27/#28):
  • fabric.loopback_pool/fabric.oob_pool fall within the fixed 10.0.0.0/8 / 192.168.0.0/16 ranges that build/fabric.py's loopback_ip()/oob_ip() actually generate (those functions are not parameterized by an arbitrary CIDR — 6+ call sites across build/*.py call them with just (pod, node_id) — so the pool fields describe, and are checked against, the builder's real output rather than being purely cosmetic).
  • CSW peering params duplicated between site.border_leaves[].csw (authoritative — carries local_ip/vlan too, used by l3out.py for the physical sub-interface/l3extMember) and tenant.l3outs[].csw_peer (used for the L3Out's bgpPeerP/static-route nexthop) must agree on name/asn/peer_ip for any border-leaf the l3out references, or loading fails with a clear error.
• loader.py: load_topology(path)->Topology.

build/ — generators (deps: topology, mit)

Each sub-builder is build(topo, site, store) and only ADDS MOs. orchestrator.build_site(topo, site) -> MITStore runs them in dependency order and returns a populated store; build_all(topo)->{site:store}. Sub-builders (one concern each, <~200 lines):

• fabric.py — fabricNode (roles/ids/model/serial/version from YAML), topSystem, fabricHealthTotal, vpcDom/vpcIf (border-leaf pair), infraWiNode.
• cabling.py — fabricLink (with /lnk-…/ DN), lldpAdjEp + cdpAdjEp derived STRICTLY from the cabling graph, isisAdjEp.
• underlay.py — bgpInst (local ASN), ospfAdjEp/ospfIf✅ (spine↔leaf), IS-IS dom.
• overlay.py — intra-site BGP-EVPN (spines as RR, leaves as clients): bgpPeer/bgpPeerEntry(established)/bgpPeerAfEntry; ISN: on each spine's sys/bgp/inst, add inter-site bgpPeer with peer addr /32 + remote (other-site) ASN.
• endpoints.py — fvCEp/fvIp distributed across leaves+EPGs (encap/fabricPathDn consistent), epmMacEp/epmIpEp, fvIfConn. Runs BEFORE tenants.py (PR-3b-batch1 reorder) so tenants.py can read back real endpoint paths for fvRsPathAtt.
• tenants.py — fvTenant/fvCtx/fvBD/fvSubnet/fvAp/fvAEPg + fvRs* rels + vzBrCP/vzSubj/vzFilter/vzEntry from YAML, plus fvRsPathAtt✅ (static binding, reusing the EPG's own endpoint path) and vzRsSubjGraphAtt✅.
• access.py — infra AAEP/IPG/port-profiles/selectors/blocks + physDomP/l3extDomP + fvnsVlanInstP/fvnsEncapBlk + rels, plus the leaf-interface-policy cluster✅ (infraAccPortP/infraHPortS/infraPortBlk/infraRsAccBaseGrp/infraAccBndlGrp) and an infraInfra root MO at uni/infra.
• l3out.py — full l3extOut subtree (LNodeP/LIfP/InstP/subnets/RsEctx/RsL3DomAtt/RsPathAtt) + bgpPeerP/bgpAsP to CSW, plus the operational bgpPeer/bgpPeerEntry(established) representing the CSW eBGP session; actrlRule for contracts; plus the route-control cluster✅ (rtctrlProfile/rtctrlCtxP/rtctrlSubjP/rtctrlMatchRtDest/rtctrlSetComm/rtctrlSetPref), static routes✅ (ipRouteP/ipNexthopP), l3extMember✅, and ospfExtP✅.
• routing.py (new, PR-3b-batch1) — per-(tenant,vrf) uribv4Route/uribv4Nexthop✅ RIB entries on every leaf/border-leaf (BD subnets as connected routes + a 0.0.0.0/0 default via the L3Out CSW peer on border leaves), and arpAdjEp✅ — one per locally-learned endpoint, reusing that endpoint's real MAC/IP. Runs after endpoints.
• interfaces.py — l1PhysIf + ethpmPhysIf (per node, ports incl. fabric uplinks + host ports), ethpmFcot, eqptcapacityPolUsage5min.
• health_faults.py — healthInst per node/tenant (…/sys/health, /tn-X/health), a seeded believable faultInst set (node-local DNs so node-scoped queries work), configExportP/configJob, coopPol/coopInst.
• userprofile.py — static uni/userprofile-admin (aaaUserEp) + aaaUserDomain child (name="all"), so the CONTRACT §1 login probe (query-target=subtree&target-subtree-class=aaaUserDomain) resolves instead of 404ing.

rest_aci/ — APIC FastAPI app (deps: query, mit, build)

• auth.py — aaaLogin/aaaRefresh/aaaLogout, token+cookie, the topSystem/userprofile login probes.
• app.py — make_apic_app(site_state): routes /api/aaaLogin.json, /api/aaaRefresh.json, /api/aaaLogout.json, /api/class/{cls}.json, /api/mo/{dn:path}.json (GET+POST), /api/node/class/{dn:path}/{cls}.json (node-scoped) and /api/node/class/{cls}.json (fabric-wide — equivalent to /api/class/{cls}.json; PR-6 FIX 3). Parses query params → query engine → envelope. Error mapping (§5 of CONTRACT).
• writes.py — POST body → upsert into MIT (recurse children, honor status); fabricNodeIdentP reaction materializes a full node-registration set (fabricNode + topSystem + healthInst + fabricLink/lldpAdjEp/cdpAdjEp cabling to every spine + l1PhysIf inventory), fires for a fabricNodeIdentP anywhere in the planned write (not just top-level), and derives pod from the target site instead of hardcoding it (PR-6 FIX 2; shared with /_sim/add-leaf). Returns created/modified imdata.

ndo/ — NDO (deps: topology, mit optional)

• model.py — build_ndo_model(topo): sites, tenants, schemas/templates (vrfs/bds/anps/epgs/contracts/filters/externalEpgs with /vrfs/… /bds/… refs), template summaries + tenantPolicy DHCP, fabric-connectivity, policy-states, audit records. Names MUST match the APIC MIT (cross-plane consistency).
• app.py — make_ndo_app(ndo_state): login/version + all §7 GETs + schema/deploy POST (store + return id/status).

runtime/ — supervisor (deps: all)

• config.py — ports, cert paths, topology path, host→site map (env-overridable).
• supervisor.py — load topology → build_all + build_ndo_model → start 3 uvicorn servers (asyncio) with TLS. Keeps a baseline deep-copy of each store for reset.

control/ — admin API (mounted on each app under /_sim/…, deps: runtime state)

• reset (restore baseline), snapshot/restore (named), reload (re-read YAML + rebuild against the FRESH matching Site object from the just-loaded topology — matched by Site.id, falling back to Site.name — not the stale pre-reload one, so edits to a site's nodes/leaves actually take effect; returns a 400 APIC error envelope if the site no longer exists in the reloaded topology instead of crashing or silently keeping stale state; PR-6 FIX 1), add-leaf/remove-leaf (runtime mutate + rebuild affected derived data — add-leaf shares writes.py's enriched node-registration reaction, not a bare fabricNode). Not part of the APIC/NDO contract — a side control plane for testing.

Internal-consistency rules (the "尽可能真实" core — enforce in build/)

• Nothing free-floating. Every operational MO derives from a topology fact.
• Cabling graph is the single source for fabricLink, lldpAdjEp, cdpAdjEp (neighbors ↔ actual links).
• IS-IS/OSPF underlay adjacencies only between physically-cabled spine↔leaf pairs.
• BGP-EVPN overlay: spines = route-reflectors, leaves = clients (sessions to spine loopbacks), operSt=established.
• ISN: for each spine, an inter-site eBGP-EVPN peer to the remote site's spines — addr /32, remote ASN = the OTHER site's fabric ASN (this is what makes autoACI draw the ISN cloud). EVPN AF up.
• L3Out: border leaves peer eBGP to CSW (asn from YAML); the l3extOut config subtree AND the operational bgpPeer(Entry) to the CSW must both exist and agree (addr/asn).
• Endpoints: each fvCEp lives on a real leaf port, in a real EPG/BD, with a plausible IP in the BD subnet; epmMacEp/epmIpEp mirror them on the owning node. Endpoint host ports never overlap the APIC-controller port range on the same leaf (see PR-5 FIX 3 below). A stretched-tenant endpoint is local (front-panel port) on exactly ONE site and tunnel/vxlan-learned on the other — never front-panel-local on both (PR-5 FIX 4).
• Health/faults: healthInst per node/tenant/controller; a small seeded faultInst set with node-local DNs so counts render and node-scoped fault queries return.
• Two sites share stretched tenants (per YAML stretch:true) → same tenant/VRF/BD names present in both MITs
  • NDO schema references them; site-local tenants present only in their site.

PR-5 topology-consistency fixes (findings #13-#18)

Six internal-consistency findings, all fixed in build/. Every port an operational MO references (l3extRsPathL3OutAtt.tDn, ospfAdjEp interface, fvCEp.fabricPathDn) must resolve against a real l1PhysIf built by interfaces.py for that exact node — this is now a checked invariant (tests/test_pr5_topology_consistency.py).

Full port inventory per node role (see interfaces.py module docstring for the authoritative version):

• Spines: eth1/1..4 fabric uplinks (one per downlink leaf/BL) + eth1/49[,50,...] dedicated ISN/IPN uplink ports, multi-site topologies only, one per spine (eth1/{49+spine_index} — matches the port underlay.py's ospfAdjEp already referenced; previously these ports didn't exist at all → FIX 2).
• Leaves + border-leaves: eth1/1..8 host-facing access ports, eth1/49[,50,...] fabric uplinks (one per spine). Border-leaves ONLY also get a dedicated eth1/48 routed L3Out sub-interface port (previously l3out.py referenced this port but nothing ever built it → FIX 1).

FIX 1 (#13) — L3Out port. Chose to extend the border-leaf port inventory with a dedicated eth1/48 routed port (mode="routed", usage="l3out") rather than reassign the L3Out onto an existing host port (e.g. eth1/8), because real ACI border-leaf L3Out sub-interfaces conventionally live on a dedicated front-panel port, not one shared with regular EPG/host traffic. l3out.py was unchanged — it already referenced eth1/48; the fix makes that port real.

FIX 2 (#14) — spine OSPF/ISN port. Chose to extend the spine port inventory with a dedicated eth1/{49+i} ISN uplink (one per spine, multi-site only) rather than reuse an existing fabric-uplink port, because every spine port in eth1/1-4 is already consumed by the intra-fabric spine↔leaf mesh (full mesh = 2 leaves + 2 border-leaves = 4 downlinks per spine in the default topology) — there is no free fabric port to reuse. This also matches real ACI: multi-site spines use a dedicated ISN-facing port, distinct from the leaf-facing fabric ports. underlay.py was unchanged — its ospfAdjEp already used eth1/{49+si}; the fix makes that port real via the same index arithmetic.

FIX 3 (#15) — endpoint/APIC port separation. fabric.py wires each APIC-{cid} (cid 1.. site.controllers, default 3) to eth1/{cid} on the first two leaves (site.leaf_nodes()[:2]) — the same leaves endpoints.py places host endpoints on. endpoints.py now reserves eth1/1..{controllers} on those leaves and starts endpoint port allocation at eth1/{controllers+1}, wrapping within the remaining _HOST_PORTS - controllers budget. Non-APIC-attached leaves are unaffected (still start at eth1/1).

FIX 4 (#16) — multi-site endpoint semantics. See below.

FIX 5 (#17) — BD unicastRoute. unicastRoute now reflects whether the BD actually has fvSubnet children (routed) rather than l2stretch — a stretched BD can still carry routed subnets, and this topology's stretched BDs (bd-BD3_LAB0, bd-BD4_LAB0) do. This agrees with ndo/model.py, which unconditionally reports unicastRouting: True for every BD (all BDs in topology.yaml have subnets).

FIX 6 (#18) — controller health. health_faults.py now emits a healthInst at topology/pod-{pod}/node-{cid}/sys/health for every controller node id (1..site.controllers), matching the topSystem fabric.py already builds there. Real APIC exposes controller health the same way as switch health.

FIX 4 — multi-site endpoint representation (design)

Real ACI never shows the identical endpoint as locally front-panel-attached at both sites of a stretched tenant simultaneously. One site has it physically attached; the other only knows about it via the multi-site overlay (ISN) tunnel. endpoints.py now models this:

• HOME site (physically attached): fvCEp.lcC="learned", fabricPathDn = the real front-panel leaf path (topology/pod-{pod}/paths-{leaf_id}/pathep-[eth1/{port}]), epm mirrors use flags="local", and a fvIfConn is emitted (real front-panel port).
• REMOTE/peer site (learned via the ISN): same MAC/IP/DN, but fvCEp.lcC="learned,vxlan" and fabricPathDn points at a tunnelIf on one of the local spines (topology/pod-{pod}/paths-{spine_id}/pathep-[tunnel{remote_spine_id}]), epm mirrors use flags="learned,vxlan", and no fvIfConn is emitted (there is no front-panel port to attach one to).
• A minimal tunnelIf MO is created per (local spine, remote spine) pair used (topology/pod-{pod}/node-{spine_id}/sys/tunnel-[tunnel{remote_spine_id}]), dest = the remote spine's loopback (representing its spine-proxy anycast TEP), src = the local spine's own loopback. overlay.py does not yet build tunnel/TEP objects, so this is a new minimal object, not a reuse.
• HOME-site assignment is deterministic and stateless across the two independent per-site build() calls: for the i-th endpoint in a stretched tenant's sequence, home site = tenant.sites[i % len(tenant.sites)]. Since both sites iterate the exact same tenant/AP/EPG/per-EPG sequence in the same order, they agree on which site is home for a given MAC without sharing any state — each site's independent build() computes the same i for the same endpoint and reaches the same conclusion.
• Non-stretched (site-local) tenants are unaffected — every endpoint is simply local, exactly as before.

Topology schema (topology.yaml) — shape

fabric: { name, evpn_rr: spine }
sites:
  - name, id, apic_host, asn, pod
    spines: N        # count OR explicit list [{id,name,model,serial}]
    leaves: M        # count OR explicit list
    border_leaves: [{id, name, vpc_domain, csw:{name,asn,peer_ip,local_ip,vlan}}]
    cabling: auto|explicit   # auto = full spine×leaf mesh + border-leaf uplinks
isn: { enabled: true, transit_asn?, peer_mesh: full }   # drives inter-site EVPN
tenants:
  - name, stretch: bool, sites: [..]
    vrfs: [{name}]
    bds:  [{name, vrf, subnets:[cidr], l2stretch?}]
    aps:  [{name, epgs:[{name, bd, contracts:{provide:[],consume:[]}}]}]
    contracts: [{name, scope, filters:[{name, entries:[{proto,from,to,ether}]}]}]
    l3outs: [{name, vrf, border_leaves:[..], csw_peer:{...}, ext_epgs:[{name, subnets:[cidr]}]}]
endpoints: { per_epg: K, ip_pool_from_subnet: true }   # generation knobs
faults:    { seed: [{severity,code,node,descr}] , health_defaults:{node:95,tenant:98} }
access:    { vlan_pools:[{name,from,to}], phys_domains:[..], aaeps:[..] }   # sensible defaults if omitted

Default topology.yaml ships the spec baseline: 2 sites × (2 spine + 2 leaf + 1 border-leaf vPC pair to CSW), ISN between them, a couple stretched + site-local tenants, one L3Out per site to its CSW, seeded endpoints/faults.

Phase plan (each phase: implementer subagent(s) → independent reviewer per 第十律)

• P1 foundation — mit/ (mo, dn, store) + query/ (filters, engine). Unit-tested in isolation.
• P2 topology — topology/schema.py + loader.py + default topology.yaml.
• P3 fabric builders — fabric, cabling, underlay, overlay(+ISN), health_faults, interfaces → Topology view renders.
• P4 tenant builders — tenants, access, l3out, endpoints → config + data-plane plugins render.
• P5 ACI REST app — auth + GET routes + POST writes + reactions + runtime supervisor + TLS + control API.
• P6 NDO — model + app.
• P7 verify — verification harness that mimics autoACI's queries for every view/plugin + write→readback + add_leaf; README + run scripts.

Verification (第二律 evidence-first)

tests/verify_autoaci.py replays the exact query patterns from docs/CONTRACT.md §8–9 against the running sim and asserts non-empty, shape-correct, internally-consistent results for: Topology (2 sites + ISN cloud + vpc pairs + faults/health), every plugin's class set, L3Out BGP-to-CSW, NDO sites/tenants/schemas, and a write→read-back cycle (POST a tenant → GET returns it) + an add_leaf reaction. Real command output is the acceptance evidence.

PR-3b-batch1 — 21 previously-catalogued-but-unbuilt classes seeded

CONTRACT.md §6 catalogued 21 classes across 6 concern groups that had no builder ever emitting them, so their class queries returned totalCount: 0 and the depending autoACI plugins (access_policy.py, epg_detail.py, contract_detail/contract_map/contract_debug.py, l3out_detail.py, l3out_status.py, route_control.py, routing_state.py, route_table.py, fabric_ospf.py) rendered nothing. All 21 are now seeded, derived from the existing topology/tenant objects rather than invented data.

Access-policy cluster (build/access.py) — one leaf-interface-policy tree per border-leaf vPC domain (infraAccBndlGrp lagT="node", consistent with the vpcDom/vpcIf pair fabric.py already builds for the same border leaves) plus one for the regular-leaf pair (lagT="link", a PC bundle). Each infraAccPortP gets one infraHPortS (type="range") selector spanning the real host-port inventory (eth1/1..eth1/_HOST_PORTS) via one infraPortBlk + one infraRsAccBaseGrp pointing at the bundle group. Also seeds an infraInfra MO at uni/infra itself — previously a structural-only DN with no MO, which 404'd GET /api/mo/uni/infra.json even under rsp-subtree=full (fixed as part of this batch since the TESTS requirement needed it; the generic subtree engine already worked correctly once a root MO existed).

Route-control cluster (build/l3out.py) — one rtctrlProfile (export) per L3Out, with a nested rtctrlCtxP (+ rtctrlSetComm/rtctrlSetPref children using real ACI's fixed short-form RNs scomm/spref). One tenant-level rtctrlSubjP match rule per L3Out (not nested under the L3Out — autoACI's route_control.py queries rtctrlSubjP with wcard(dn,"tn-{tenant}"), tenant-scoped, verified read-only against the plugin source) with an rtctrlMatchRtDest child reusing a real tenant BD subnet. A static default route (ipRouteP ip="0.0.0.0/0") nested under each L3Out's l3extRsNodeL3OutAtt, with an ipNexthopP child pointing at the same CSW peer IP the eBGP session builder (_upsert_ebgp_sessions) already peers with. l3extMember (side="A") on the existing l3extRsPathL3OutAtt — this topology's L3Out path is a single port per border-leaf (not vPC-bundled), so only one member is emitted, per the batch-1 placement note. ospfExtP added to the SAME L3Out that already carries bgpExtP (real ACI supports OSPF+BGP coexistence on one L3Out) rather than fabricating a second OSPF-only L3Out in topology.yaml — the simpler faithful choice since this topology has no existing OSPF-only L3Out to attach to.

EPG/contract classes (build/tenants.py) — fvRsPathAtt (static binding) per EPG, pointing at the SAME front-panel path that EPG's own endpoints already use. This required reordering the orchestrator's builder list (endpoints now runs BEFORE tenants, moved up from after l3out) so tenants.py can read back the real fvCEp.fabricPathDn/encap endpoints.py already wrote for that EPG, instead of independently re-deriving the port-allocation algorithm (which risked silently drifting out of sync). Every other builder remains pure add-only; endpoints.py itself never reads the store, so the reorder is safe. A stretched EPG whose only endpoints at this site are REMOTE (tunnel-learned) gets no fvRsPathAtt — a static binding must reference a real local port. vzRsSubjGraphAtt attached to exactly one contract's subject per tenant (the tenant's first contract), with a plausible tnVnsAbsGraphName; vnsAbsGraph itself stays out of scope.

Operational/routing classes (new build/routing.py + build/underlay.py) — uribv4Route/uribv4Nexthop seeded per (tenant, vrf) on every leaf/border-leaf: BD subnets as directly-connected routes (nexthop = the node's own loopback) plus a 0.0.0.0/0 default via the L3Out's CSW peer on border leaves, matching the ipRouteP static route already seeded on the same node. VRF "dom" name reuses the exact {tenant}:{vrf} convention l3out.py's BGP dom already established. uribv4Nexthop's RN uses the 4-bracket nh-[proto]-[addr]-[ifname]-[vrf] shape autoACI's _RE_NH_PARENT regex (route_table.py/routing_state.py, identical in both files) requires — verified read-only against the plugin source, not re-derived from guesswork. arpAdjEp — one per locally-learned (front-panel) endpoint, reusing that endpoint's real MAC/IP from the store endpoints.py already populated (routing.py runs after endpoints in the orchestrator order); REMOTE/tunnel-learned endpoints get no ARP entry (real ACI resolves those via the overlay, not local ARP). ospfIf nested between the existing ospfDom and ospfAdjEp in underlay.py, on the same eth1/{49+i} ISN uplink port interfaces.py already builds a dedicated l1PhysIf for — so the pre-existing ospfAdjEp now sits on a real, matching ospfIf.

Writes — rest_aci/writes.py _RN_TEMPLATES gained entries for all POST-able batch-1 classes so a child posted without an explicit dn lands on the same canonical DN a GET or builder-created MO would use.

PR-3b-batch2/3 — remaining previously-catalogued-but-unbuilt classes seeded

Batches 2+3 of the seeding effort closed the rest of CONTRACT.md §6's gap: 9 more classes across build/fabric.py, build/interfaces.py, build/tenants.py, a new build/zoning.py, and build/overlay.py, plus a deliberate decision NOT to build infraNodeIdentP. DN/attribute shapes were verified read-only against the specific autoACI consumer plugins (a targeted grep of autoACI's backend/plugins/*.py) before writing any builder code, per the same discipline batch-1 established.

Port-channel/vPC (build/fabric.py) — pcAggrIf + pcRsMbrIfs, one pair per vPC leg (border-leaf), consistent with the vpcDom/vpcIf this module already builds for the same pairs. vpc_status.py matches pcAggrIf to vpcIf by (node, name) — so pcAggrIf.name intentionally reuses the exact same "po{bl.id}" string vpcIf.name already carries (not access.py's differently-named infraAccBndlGrp, a different IPG identity autoACI's plugin does not join on). Naming correction: CONTRACT.md catalogued this relation class as vpcRsMbrIfs; a read-only grep of vpc_status.py showed it actually queries pcRsMbrIfs (the real ACI class, child of pcAggrIf, RN rsmbrIfs-[{ifDn}]) — built as pcRsMbrIfs, with a CONTRACT.md §6 changelog note recording the correction. Member ports are two real host-facing l1PhysIf per leg (eth1/1, eth1/2 — always present on every leaf/border-leaf) so pcRsMbrIfs.tDn always resolves.

Optics (build/interfaces.py) — ethpmFcot, sibling of ethpmPhysIf under the same port DN (own fcot RN). Emitted only for fabric-uplink, ISN-uplink, and L3Out routed ports — real optics-bearing front-panel ports — not plain host-access ports, mirroring real labs where copper/DAC host ports often report no transceiver inventory while fabric/WAN-facing optical ports do (a deliberate realism choice, documented in the module docstring, not an omission).

Cluster health (build/fabric.py) — infraWiNode, the APIC appliance-vector: every controller carries its own view of every cluster member (an N×N matrix for N=site.controllers), all seeded health="fully-fit" to match this sim's other all-healthy defaults (fabricHealthTotal.cur=100).

Zoning-rule / pcTag plumbing (build/tenants.py + new build/zoning.py) — real ACI zoning rules key off per-EPG/VRF pcTag ("sclass") + scopeId, neither of which existed anywhere in this sim before batch-2 (verified: zero prior pcTag references). tenants.py gained pctag_for/scope_id_for — small CRC32-derived helpers (same technique endpoints.py's _vnid already uses) offset clear of autoACI's 5 hardcoded system pcTags (1/13/14/15/16384) — assigning a stable pcTag to every fvAEPg and fvCtx (+ scope on fvCtx), and seeding one fvEpP (uni/epp/fv-[{epg_dn}]) per EPG carrying the matching pcTag, the EPG's VRF scopeId, and epgPKey=the EPG's own dn (zoning_rules.py extracts the friendly EPG name back out of epgPKey's trailing epg- segment). New build/zoning.py derives actrlRule — one permit rule per (contract subject filter, provider EPG, consumer EPG) triple, emitted on every leaf where either side has a locally-learned endpoint (read back from endpoints.py's fvCEp, same technique tenants.py's fvRsPathAtt already established), with sPcTag/dPcTag/scopeId reusing the exact pctag_for/scope_id_for values the owning EPGs/VRF already carry. Also seeds real MOs at the .../sys/actrl and .../sys/actrl/scope-[...] structural containers — the same class of gap batch-1's infraInfra fix addressed for uni/infra: without a real MO at the exact DN a subtree query targets, GET /api/mo/{dn}.json 404s even though the generic subtree engine itself works fine once a root MO exists. zoning runs right after tenants in the orchestrator (both already after endpoints).

EVPN routes (build/overlay.py) — bgpVpnRoute + bgpPath, the per-spine EVPN VPN route family fabric_bgp_evpn.py's "vpn_routes" view two-pass parses (pass 1 collects routes by dn; pass 2 recovers the parent route dn from a path dn via path_dn.rsplit("/path-", 1)[0] — so bgpPath's RN must start with path- directly under the route's own dn, verified read-only against the plugin). Seeded per spine (the EVPN route-reflector role) for each BD subnet deployed at the site, with one bgpPath per leaf that actually hosts a locally-learned endpoint for that BD (read back from fvRsBd+fvCEp). This needs the same post-tenants/endpoints data zoning.py needs, but overlay.build() itself must stay in its original EARLY orchestrator slot (BGP peer setup only needs fabric/topology data) — so the new logic lives in a second function, overlay.build_vpn_routes(), called by the orchestrator as an explicit extra step after the main _BUILDERS loop completes, rather than moving/splitting overlay.build()'s existing peer logic.

Node registration (build/fabric.py + rest_aci/writes.py) — fabricNodeIdentP, boot-seeded one per real switch fabricNode (spines/leaves/border-leaves — NOT controllers; real ACI's Fabric Membership registers switches joining the fabric, not the APICs themselves) at uni/controller/nodeidentpol/nodep-{id}, so a GET returns already-registered nodes without requiring a prior POST. Keys on the node id, not serial, to stay consistent with writes.py's pre-existing _materialize_fabric_node reaction (which parses the id back out of the same nodep-(\d+)$ pattern) — a boot-seeded and a later-POSTed registration must agree on the identifier scheme, or they'd silently diverge into two different DNs for the same node. _RN_TEMPLATES gained a fabricNodeIdentP -> nodep-{nodeId} entry (the one batch-2/3 class an actual Fabric Build playbook POSTs, per CONTRACT.md §3). Regression-tested: tests/test_pr3b_batch2.py re-verifies test_rest_aci.py::test_add_leaf_reaction's exact scenario (POSTing a NEW nodep-950 must still materialize a matching fabricNode) to confirm boot-seeding existing nodes doesn't interfere with the write-path reaction for a genuinely new one.

infraNodeIdentP — deliberately NOT built. No autoACI plugin references it (verified via a fabric-wide grep across autoACI's backend/plugins/*.py). Real ACI's infraNodeIdentP is a node provisioning-policy object, a different concept from fabricNodeIdentP's Fabric Membership registration despite the similar name — building it with fabricNodeIdentP-like semantics would be a wrong object satisfying no real consumer. Removed from CONTRACT.md's catalog entirely rather than built incorrectly; see that file's batch-2/3 changelog note for the full reasoning.

PR-9 — Ansible cisco.aci fidelity

The owner's primary public use case for this simulator is running Ansible cisco.aci playbooks against it directly. PR-9 hardens the write/delete/auth surface to match what cisco.aci's plugins/module_utils/aci.py actually sends, verified read-only against the upstream collection source (https://github.com/CiscoDevNet/ansible-aci). Full behavioral contract: docs/CONTRACT.md §11. Highlights:

• DELETE /api/mo/{dn}.json added (state=absent), server-side idempotent (200-empty on both existing-then-removed and already-missing DNs) — see CONTRACT.md §11 for why idempotent-on-missing was chosen over 404 despite the module never actually exercising that exact case itself.
• GET-on-missing-DN is 200-empty, not 404 (FEATURE 6, a live blocker found running an actual playbook against the sim: fatal: APIC Error 103 on the very first task of a state=present create). This superseded the pre-PR-9 design (404-on-missing), which had been justified as matching a DIFFERENT consumer's (autoACI's ACINotFoundError) expectation — cisco.aci is now this simulator's primary target per the owner's stated use case, and its api_call() treats any non-200 GET as fatal with no "not found is fine" exception. /api/class/* and node-scoped queries already had no such 404 case (they return 200 + empty on a no-match query); this brings /api/mo/* in line for "the DN itself doesn't exist" too.
• status="deleted" cascade — already correctly removed the full subtree pre-PR-9 (MITStore.upsert's status="deleted" branch predates this PR); PR-9 added regression coverage proving it (not just "skips planning children", which is a separate, narrower behavior the write-planning layer also has). status="created,modified" (and any status other than "deleted") is a plain upsert — no special-casing needed since the store only ever special-cases the literal "deleted" string.
• annotation/nameAlias/descr passthrough — no fix needed; MO.attrs is a generic dict and the write path never allowlists attribute names. PR-9 added the regression test proving the round-trip survives POST → mo GET → class query, since this was previously undocumented/unverified rather than actually broken.
• firmwareCtrlrRunning — new, small: one per controller in build/fabric.py, version reusing the same site.apic_version the controller's topSystem already carries.

Certificate accept-mode trust model (cert-auth, PR-9)

cisco.aci supports password-less certificate-signature auth: instead of calling aaaLogin, every request carries a single Cookie header with four fields (APIC-Certificate-Algorithm, APIC-Certificate-DN, APIC-Certificate-Fingerprint, APIC-Request-Signature — an RSA-SHA256 signature over METHOD + path + body, base64-encoded). Real APIC verifies that signature against a public key previously uploaded for the named user (aaaUserCert).

This simulator does NOT verify the signature. require_session() (rest_aci/auth.py) implements accept-mode: if all four cookies are present and APIC-Certificate-DN parses to uni/userext/user-{user}/usercert-{name} with user == SIM_USERNAME, the request is treated as authenticated for that user — full stop. The APIC-Request-Signature bytes are read (cookie must be non-empty) but never decoded or checked against any key. This is a deliberate, documented trust shortcut, not an oversight:

• The simulator has no real user-certificate registry (aaaUserCert upload flow is out of scope, same tier as NX-OS SSH being out of v1 per the "Locked decisions" above) — implementing real verification would require building that entire subsystem for a security property the simulator doesn't need (there is no real secret being protected; it's a local test fixture). Trusting the claimed identity once the DN's user matches SIM_USERNAME is sufficient to unblock cert-auth playbooks running unmodified against the sim, but is insufficient as a real security boundary.
• Do not point this accept-mode design at anything internet-reachable or multi-tenant. Anyone who can reach the sim's port can authenticate as SIM_USERNAME by sending four cookies with no cryptographic proof of possession.
• SIM_CERT_STRICT env var is reserved (parsed but currently a no-op) for a future mode that would register a real public key per user and verify APIC-Request-Signature against it. Not implemented in PR-9 — out of scope; noted here so a future PR doesn't have to rediscover the gap.

PR-18 — Tier-1 fabric configuration parameters

Exposes real-ACI-lab-variable Tier-1 fabric knobs that Ansible/cisco.aci/aci-py fidelity depends on, all optional with defaults matching the pre-PR-18 hardcodes so the existing topology.yaml (and any hand-authored one) validates and builds unchanged.

• site.controllers: int (default 3) — already existed pre-PR-18 (introduced with the APIC-cluster work) and was already fully wired: build/fabric.py's controller fabricNode/topSystem/ firmwareCtrlrRunning loop, the infraWiNode appliance-vector double loop (both viewer and member ranges), build/health_faults.py, and build/endpoints.py's APIC-reserved-port accounting all already read site.controllers — no hardcoded 3 remained to replace. PR-18 only added --controllers to aci-sim new and surfaced the value in aci-sim show.
• fabric.tep_pool: str (default "10.0.0.0/16") — the infra TEP address space. Stored + CIDR- validated + surfaced (aci-sim show/new). No builder derives a per-node address from this field — build/fabric.py's loopback_ip()/oob_ip() are the actual address source (see the pre-existing loopback_pool/oob_pool note in topology/schema.py's Fabric docstring, which documents the same "hardcoded family, pool is descriptive metadata" pattern this field follows). Treat tep_pool as store-only/informational until a future PR threads it into an actual infra-address builder.
• fabric.infra_vlan: int (default 3967, range 1-4094) — the fabric infrastructure VLAN. Stored + range-validated + surfaced. No MIT class in this sim currently carries a distinct infra-VLAN attribute to wire it into (real ACI's infraSetPol.fabricInfraVlan classes are not part of the current CONTRACT.md catalog) — store-only, same caveat as tep_pool.
• fabric.gipo_pool: str (default "225.0.0.0/15") — the multicast GIPo pool. Stored + CIDR-validated
  • surfaced. Store-only, same caveat.
• Deterministic node serials — build/fabric.py's default_serial(site_id, node_id) replaces the previous inline f"SIM{node.id:08d}" fallback used whenever an auto-generated Node.serial is empty (fabricNode.serial and fabricNodeIdentP.serial). New format: f"SAL{site_id}{node_id:04d}" (e.g. site "1" node 101 → "SAL10101") — encodes the site id so serials stay visibly distinct per site. Explicit serial: values in a YAML Node entry are unaffected (still override — this is only the serial or default_serial(...) fallback path). No test asserted the old "SIM..." format, so this is a safe behavior change; aci-sim show now prints the resolved serial per node.
• site.pod: int (already existed, default 1) — verified honored end-to-end: every builder that emits a topology/pod-{N}/... DN reads site.pod (grepped all of build/*.py, rest_aci/writes.py, control/admin.py — zero remaining pod-1/pod=1 literals). A site with pod: 2 builds all its fabricNode/topSystem/etc. DNs under pod-2 (verified: topology/pod-2/node-201, ...).

N-site limit (verified, not fixed by PR-18)

aci-sim new --sites N (N>2) and the builders that only depend on site.pod/site.controllers/node-ID offsets work for arbitrary N with zero node-ID collisions (verified with a 3-site scaffold: sites at offsets 0/200/400, infraWiNode count = controllers² per site, all DNs correct).

However, Topology.other_site(name) (topology/schema.py) is hard-coded to 2-site semantics: it returns "the first site in self.sites that is not name", not "every other site". Two builders call it to resolve the ISN/multi-site remote peer:

• build/overlay.py's ISN inter-site eBGP-EVPN peering (build(), "ISN: each local spine → each remote-site spine loopback").
• build/endpoints.py's stretched-tenant HOME/REMOTE endpoint placement (is_stretched branch).

Verified with a live 3-site topology (generate_topology(sites=3, ...), ISN peer_mesh: full): site C (3rd site, asn 65003) peers ISN eBGP ONLY with site A (asn 65001) — other_site("LABC") picks the first non-C site in list order (LABA) and never reaches LABB. This means with 3+ sites:

• ISN is NOT a true full mesh across all sites, despite isn.peer_mesh: full — each site peers with exactly one other (the first site in the YAML's sites: list order that isn't itself), not N-1 others.
• A tenant with stretch: true and 3+ sites: entries passes schema validation (the validator only requires len(tenant.sites) >= 2, not == 2) but endpoints.py's remote-peer logic will treat only one of the non-home sites as the ISN-reachable remote for tunnel/REMOTE-endpoint purposes.

This is a pre-existing limitation, not introduced by PR-18 (confirmed via git diff main -- aci_sim/topology/schema.py: other_site() is byte-identical to what PR-18 branched from). Fixing it to a real N-site full mesh is a bigger architectural change (both call sites would need to iterate [s for s in topo.sites if s.name != site.name] instead of a single other_site() call, and endpoints.py's single remote_site/tunnel_spine/home_spine variables would need to become per-remote-site collections) — out of scope for a Tier-1-parameters PR. Practical limit: ISN/stretch correctness is verified for exactly 2 sites; 3+ site topologies build without collision but do not get a correct N-site ISN mesh or N-site stretched-endpoint placement.

PR-19 — Tier-2 topology-elasticity parameters

Exposes ISN underlay knobs (ospf_area, mtu) and a VMM domain vCenter-connection model, plus fixes a real leaf/spine-count-elasticity bug in aci-sim new's node-ID generator. All new fields optional with defaults matching pre-PR-19 hardcoded behavior, so the existing topology.yaml validates and builds byte-identical output.

isn.ospf_area: str (default "0.0.0.0") and isn.mtu: int (default 9150)

Both were previously hardcoded literals in the builders:

• build/underlay.py's ospfIf/ospfAdjEp area attribute was hardcoded to "0.0.0.0" — now reads topo.isn.ospf_area. Accepts either dotted-decimal ("0.0.0.0") or the decimal-integer shorthand ACI also renders ("0") — validated in Topology.normalize_and_validate via IPv4Address parse or str.isdigit().
• build/interfaces.py's dedicated ISN/IPN spine uplink port (eth1/{49+i}, multi-site only) was built via the same _add_port() helper as every other port, which hardcoded mtu="9216" (the intra-fabric default) for every port including this one. _add_port() now takes an mtu keyword (default "9216", unchanged for all ordinary ports); the ISN uplink call site alone passes str(topo.isn.mtu). Verified: a leaf's eth1/49 fabric-uplink port and a spine's eth1/49 ISN uplink port are different physical interfaces on different node roles (interfaces.py's port-numbering docstring), so this change touches only the true ISN port — ordinary fabric ports (including any leaf port that happens to share the same port number) are unaffected. isn.mtu range-validated 576-9216 (real ACI inter-pod/inter-site MTU range, same ceiling as the fabric's own 9216).
• isn.peer_mesh: partial remains a documented validate-and-reject (unchanged from PR-17/18's decision in build/overlay.py) — re-examined for PR-19 and kept: a partial mesh has no sensible default partition without additional per-site config this schema doesn't carry (which spine pairs with which isn't inferable from spine count alone). Silently building full-mesh (ignoring the setting) or an arbitrary subset (guessing a partition no author asked for) would both be worse than failing fast with a clear message, so PR-19 keeps the existing rejection rather than inventing a partition scheme.

VMM domain vCenter connection model — pure addition, not a rewire

access.vmm_domains: list[VMMDomain] (default: empty list, unlike phys_domains/l3_domains/ aaeps, which each default to one seeded entry). Each VMMDomain carries name (required) plus optional vcenter_ip, vcenter_dvs, vlan_pool, datacenter.

Why this is additive, not a wire-up of an existing gap (verified read-only against a real aci-py + cisco.mso production chain on a lab test host):

• ~/aci-py/aci_py/playbooks.py's bind_epg_to_vmm_domain playbook (MSO section) drives mso_schema_site_anp_epg_domain (shimmed in ~/aci-py/aci_py/shims/mso_mso_epg.py), which appends a JSON-Patch op add /sites/{site}-{tpl}/anps/{anp}/epgs/{epg}/domainAssociations/- with value.dn = "uni/vmmp-VMware/dom-{domain_profile}" — a DN string, not a live object reference. The sim's NDO layer (aci_sim/ndo/patch.py) stores this JSON-Patch document verbatim; nothing cross-validates the referenced dn against a real vmmDomP in any APIC-side MIT store. ~/aci-py/environments/lab-multisite-sim/vars/bind_epg_to_vmm_domain_MS-TN1.yml confirms the domain names actually exercised in the live MS-TN1 chain (vmm-vmw-lab1, vmm-vmw-lab2) are just opaque strings from this var file's perspective.
• The single-fabric branch of the same playbook template (bind_epg_to_vmm_domain.j2.yml's APIC section) DOES drive cisco.aci.aci_domain (shimmed in ~/aci-py/aci_py/shims/access.py's _domain_dn_class, which maps vmmDomain → vmmDomP at uni/vmmp-{provider}/dom-{name}) — but this branch is gated on single_fabric|default(false) and is not what the MS-TN1 multi-site chain (scripts/live_chain.py environments/lab-multisite-sim MS-TN1) exercises.
• Conclusion: no consumer in either verified production chain (autoACI sandbox e2e, aci-py MS-TN1) reads a vmmDomP/vmmCtrlrP object from this sim today. Adding one is genuinely new surface, not a fix for an existing silently-broken path — hence the empty-list default (true backward-compat baseline: omitting access.vmm_domains produces zero vmmDomP/vmmCtrlrP/vmmUsrAccP MOs, byte-identical to pre-PR-19) rather than seeding a default domain the way phys_domains/l3_domains/aaeps do.

MO shape (build/access.py::_build_vmm_domain), verified against the real DN scheme in ~/aci-py/aci_py/shims/access.py's _domain_dn_class ("vmmDomP", "uni/vmmp-{provider}/dom-{name}") and ~/aci-py/aci_py/shims/mso_mso_epg.py's dn_map["vmmDomain"] = "uni/vmmp-VMware/dom-{0}":

• vmmDomP at uni/vmmp-VMware/dom-{name} — always built (only "VMware" is modeled; the sole provider either verified call site actually exercises).
• vmmCtrlrP at {vmmDomP dn}/ctrlr-{name} — built only when vcenter_ip is set: hostOrIp = vcenter_ip, rootContName = datacenter (falls back to the domain's own name if datacenter is unset). dvsName is additionally set when vcenter_dvs is provided.
• vmmUsrAccP at {vmmCtrlrP dn}/usracc-{name} — built alongside vmmCtrlrP (placeholder credential-profile reference; no real credentials in a sim, matching the no-secret pattern phys/l3 domains already use for their own child objects).
• infraRsVlanNs at {vmmDomP dn}/rsvlanNs — built only when vlan_pool is set, pointing at the first access.vlan_pools[] entry's DN (same convention _build_phys_dom/_build_l3_dom already use). vlan_pool is cross-referenced against access.vlan_pools[].name in Topology.normalize_and_validate — an unknown pool name is a validation error, same discipline as tenant.bds[].vrf/epg.bd.

_build_epg() in build/tenants.py (the actual EPG-domain binding builder) is untouched — EPGs still bind only to the first phys_domains[] entry via fvRsDomAtt, exactly as before PR-19. This is intentional: the real production chain's bind_epg_to_vmm_domain never touches this MIT tree (see above), so there was nothing to rewire.

Leaf/spine count elasticity — a real generator bug found and fixed

aci-sim new's node-ID scheme (topology/schema.py's docstring: leaves base 101, spines base 201, site_offset=200) has a documented ceiling — "max ~40 leaves + ~40 spines per site before the ranges touch" — but prior to PR-19, generate_topology() (cli.py) had no guard enforcing that ceiling. Stress-testing the exact boundary (required by this PR's acceptance bar) found three distinct ways a large --leaves-per-site/--spines-per-site/--border-pairs value silently produced an invalid topology.yaml dict that only failed deep inside Topology.normalize_and_validate, with a generic "node ids collide" message that didn't name the offending flag:

1. --leaves-per-site > 100 — the leaf block itself (101..100+N) runs into the spine block (201+). Reproduced: --leaves-per-site 150 --spines-per-site 4 → leaf ids run 101-250, directly overlapping spine ids 201-204.
2. --spines-per-site > 100 — the spine block itself (201..200+N) runs into the NEXT site's leaf block (301+). Reproduced: --leaves-per-site 2 --spines-per-site 101 → site 0's spine ids run 201-301, colliding with site 1's leaf ids starting at 301.
3. Large --border-pairs — the border-leaf block (already pushed past whichever of leaves/spines extends further, per the pre-existing PR-17 border-ID fix) can itself run past the next site's leaf block if border_pairs is large enough relative to a small leaf/spine count. Reproduced: --leaves-per-site 2 --spines-per-site 2 --border-pairs 50 → border ids run up to 300+, colliding with the next site's leaf block at 301.

Fix: generate_topology() now computes the same leaf/spine/border-top arithmetic the ID-assignment loop uses and raises a ValueError naming the specific offending flag before building any node dicts, for all three cases above. Verified boundaries: --leaves-per-site 100/--spines-per-site 100 (exact max) still validate and build cleanly; 101 of either is rejected with a clear message; --border-pairs 49 at the default leaf/spine count is the max safe value, 50 is rejected. The acceptance-bar case from this PR's scope (--leaves-per-site 8 --spines-per-site 4, well within the safe range) validates clean and builds a full MITStore with the expected node count, verified via orchestrator.build_site() (not just schema validation) in tests/test_pr19_tier2.py's TestLeafSpineElasticity.test_full_build_succeeds_for_8_leaves_4_spines.

This is a genuine bug fix (not a new feature) — the ID scheme's stated ceiling was never actually enforced anywhere before this PR, meaning any topology author who requested more than ~100 leaves/spines-per-site via the CLI would previously get a confusing Pydantic stack trace instead of an actionable error.

PR-20 — Tier-3 fine-tuning parameters (final param tier)

The last planned parameter tier. Exposes commonly-tuned mgmt-subnet/BD-MAC/routing-timer/APIC-version knobs as optional topology.yaml fields, wired into a real MO attribute wherever this sim already builds one — and honestly marked store-only where it doesn't (BFD timers, inb_subnet). All new fields optional with defaults matching pre-existing hardcoded behavior (or the real ACI factory default where there was no prior hardcoded value at all), so the existing topology.yaml validates and builds byte-identical output — verified via the full tests/test_pr20_tier3.py::TestBackwardCompat class plus the unchanged 672-test suite passing alongside 41 new tests (713 total).

bd.mac / fabric.default_bd_mac — the critical backward-compat case

build/tenants.py's _build_bd hardcoded fvBD.mac = "00:22:BD:F8:19:FF" (the real ACI default gateway MAC) for every BD before this PR. fabric.default_bd_mac (fabric-wide default) reproduces that exact literal as its own schema default, and bd.mac (per-BD, optional) overrides it when set — resolution order in _build_bd is bd.mac or topo.fabric.default_bd_mac. Because the fabric-wide default equals the pre-existing hardcoded literal exactly, a topology with neither field set produces byte-identical fvBD.mac output to pre-PR-20 — this is the single highest-risk field in this PR (any drift from the existing literal would visibly change every BD's gateway MAC that a consumer like autoACI might read), so it got its own dedicated backward-compat test (test_repo_fvBD_mac_is_real_aci_default) asserting the literal value directly against the real repo topology.yaml, not just "no schema-level default drift."

BGP keepalive/hold — l3out.csw_peer.keepalive/.hold → bgpPeerP

Real ACI defaults 60s/180s (the standard 1:3 keepalive:hold ratio). build/l3out.py's _build_node_profile already builds a bgpPeerP MO (control-plane BGP peer policy, distinct from the operational bgpPeer/bgpPeerEntry session MOs _upsert_ebgp_sessions builds) with only addr/ peerCtrl attrs pre-PR-20; this PR adds keepAliveIntvl/holdIntvl to the same object. Validated: hold must exceed keepalive (a hold at or below the keepalive interval can never survive a single missed keepalive) — same sanity-check pattern as PR-19's isn.mtu range check. The operational bgpPeer/bgpPeerEntry MOs are not touched — real ACI's operational BGP session table doesn't carry configured-timer attrs (those live on the policy object), so there's nothing to wire there.

OSPF hello/dead — isn.ospf_hello/.ospf_dead → ospfIf

Real ACI defaults 10s/40s. build/underlay.py's ospfIf MO (already built for isn.ospf_area, PR-19) gains helloIntvl/deadIntvl attrs. Validated: ospf_dead must exceed ospf_hello, matching real ACI's own OSPF sanity check (a dead timer at or below the hello interval can never detect a live neighbor before declaring it down).

BFD timers — isn.bfd_min_rx/.bfd_min_tx/.bfd_multiplier — explicitly STORE-ONLY

Real ACI defaults 50ms/50ms/3. Grepped build/*.py for "bfd"/"Bfd"/"BFD" — zero matches. This sim builds no bfdIfP/bfdRsIfPol-style MO anywhere, and adding a faithful one would additionally require a believable bfdIfStats/oper-state child tree with no verified consumer (autoACI plugin or aci-py playbook) to anchor its shape against — inventing an MO purely to hold a timer, with no real read path exercising it, would be worse than being explicit that these three fields are schema-level only. Store + range-validate (min_rx/min_tx positive; multiplier 1-50, real ACI's UI range) + surface in aci-sim show/new.

OOB/INB mgmt subnets — fabric.oob_subnet (validate-only) / fabric.inb_subnet (store-only)

fabric.oob_subnet sits alongside the existing fabric.oob_pool (PR-early/#27) — build/fabric.py's oob_ip() derives every node's OOB address from (pod, node_id) only and does not consume an arbitrary subnet/pool CIDR (documented limitation, unchanged by this PR). oob_subnet is stored + validated (must fall within 192.168.0.0/16, same family as oob_pool) + surfaced, letting a topology author record the intended OOB subnet distinctly from the pool the builder actually draws from.

fabric.inb_subnet is genuinely new surface, not a wire-up of an existing gap: grepped build/*.py for "mgmtInB"/"inbMgmt" — zero matches. Real ACI's in-band management is a distinct mgmtInB bridge-domain living in the special mgmt tenant, requiring its own tenant/BD/EPG-style builder — a materially bigger addition than "wire an existing attr," and out of scope for a fine-tuning-parameters PR. Stored + validated (must fall within a private RFC1918 range: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16)

• surfaced, explicitly marked store-only in aci-sim show's own output ((store-only) label) so a user never mistakes it for something the sim actually builds.

fabric.default_apic_version — fabric-wide override, on top of already-independent versioning

Important finding while implementing this PR: the task ask ("make apic_version drive the APIC controller topSystem/firmwareCtrlrRunning version consistently, independent of switch-node version") was already fully satisfied before this PR — site.apic_version (added in the PR-9/PR-18 lineage) has driven build/fabric.py's controller fabricNode.version/topSystem.version/firmwareCtrlrRunning.version since PR-9, completely independently of the per-node switch Node.version field (PR-18). Verified: grepped build/*.py for the literal "4.2(7f)" and found it only as Site.apic_version's own schema default in topology/schema.py — no builder hardcodes an APIC version literal that bypasses site.apic_version.

What was missing: (1) no fabric-wide override existed for topologies with many sites that all want the same APIC version, and (2) aci-sim show/new never surfaced apic_version at all (verified: zero matches for "apic_version" in pre-PR-20 cli.py). This PR adds fabric.default_apic_version (optional, default None = unchanged behavior — each site keeps using its own site.apic_version) with resolution topo.fabric.default_apic_version or site.apic_version in build/fabric.py, and surfaces the effective resolved value per-site in aci-sim show's new apic_version= column — closing the surfacing gap without touching the wiring that already worked.

Backward-compat verification performed vs. NOT performed (be precise)

Performed in this sandboxed development environment: aci-sim validate on the repo topology.yaml (OK), the full unit suite (713 passed: 672 pre-existing + 41 new, zero regressions), and aci-sim show/ new manual smoke checks confirming every new field surfaces correctly.

NOT performed — no access from this environment: the real production-chain regression gate this project's acceptance bar calls for (autoACI sandbox e2e 11/11, aci-py MS-TN1 live_chain 0-fail) requires an SSH-reachable hardware test host, which does not exist in this sandboxed clone. The fvBD.mac default was double-checked against the pre-PR-20 hardcoded literal in build/tenants.py specifically because it is the one Tier-3 change with plausible autoACI-regression risk (autoACI reads fvBD attrs) — but this is static-code verification, not a live e2e run. The main thread's independent re-run of the real Ansible/aci-py chains is the authoritative backward-compat gate for this PR, per this project's established process (several prior PRs passed unit tests but broke the real production chain).

PR-21 — single-APIC default + per-controller OOB IPs

Design intent (owner directive): APIC cluster size is irrelevant to Ansible playbook deployment testing — a playbook connects to and pushes config through exactly one APIC endpoint, regardless of how many controllers back it in real ACI. Cluster size only shows up in cluster-health/appliance-vector tooling, e.g. autoACI's health_score.py, which reads infraWiNode across every controller. This PR therefore flips Site.controllers's default from 3 (pre-PR-21) to 1 (single-APIC-per-site), while keeping the field fully configurable (validated range 1-5) for anyone who does need to exercise multi-controller/cluster-health semantics.

Site.controllers default 3 → 1, validated range 1-5

This is a breaking change to generated node counts, not merely a config default: every site's fabricNode/topSystem count drops by (old controllers − new controllers) = 2 by default (e.g. a 2-spine/2-leaf/2-border-leaf/3-controller site was 9 fabricNodes; the same shape at the new 1-controller default is 7). fabricLink/lldpAdjEp/cdpAdjEp counts for the APIC↔leaf attachment drop proportionally (2 leaves × N controllers, so N=3→6 links becomes N=1→2 links), and infraWiNode count (N²) drops from 9 to 1. Every hardcoded count in the test suite and e2e scripts that assumed the old default was audited and updated to the new default — see the PR-21 commit message / CHANGELOG for the full list of touched assertions (the pattern used throughout: prefer a test that derives the expected count from site.controllers dynamically, as test_pr3b_batch2.py's infraWiNode test already did pre-PR-21, over one that hardcodes a literal that will need updating again if the default ever changes). Range validation (1-5) lives in Topology.normalize_and_validate alongside the other per-site checks — 6 (or 0, or negative) is rejected with a clear error naming the site.

Site.controller_ips — per-controller OOB mgmt IP (optional)

Pre-PR-21, every controller's topSystem.oobMgmtAddr was derived from (pod, controller_id) via the same oob_ip() helper switch nodes use — never from the site's own mgmt_ip, which is the address the sim's REST server actually binds to (controller-1 / the cluster VIP in sandbox mode). That's fine for a single controller (there's only one address to assign, and CI/tests never asserted a specific controller OOB IP), but becomes a believability gap for controllers > 1: real multi-controller clusters assign each APIC its own OOB IP typically adjacent to the cluster's primary address, not derived from an arbitrary per-node formula.

controller_ips: list[str] (optional, default unset) fixes this: when set, controller_ips[i] is controller i+1's OOB IP verbatim (length must equal controllers, each entry a valid IP — validated in Topology.normalize_and_validate, same place as the controllers range check). When unset, Site. controller_oob_ips() (new helper, topology/schema.py) auto-derives sequentially from mgmt_ip: controller 1 = mgmt_ip, controller 2 = mgmt_ip + 1, controller 3 = mgmt_ip + 2, etc. (plain ipaddress integer arithmetic, so it correctly rolls over octets). If mgmt_ip is also unset (empty string — some hand-authored topologies never set it), controller_oob_ips() returns [] and build/fabric.py falls back to the legacy oob_ip(pod, cid) scheme, so a topology with neither field set keeps its pre-PR-21 controller OOB addresses unchanged.

Wired into build/fabric.py's controller loop for both topSystem.oobMgmtAddr and the leaf-side lldpAdjEp/cdpAdjEp.mgmtIp (the neighbor-reported address must match the controller's own OOB IP, or autoACI's LLDP-based topology rendering would show a controller "announcing" an IP that doesn't match its own topSystem) — both read the same per-call controller_oob_ips list so they can never drift apart.

Shipped topology.yaml — dropped to the new default, not left at 3

Both LAB1/LAB2 already omitted an explicit controllers: field pre-PR-21 (it was commented out at its then-default of 3), so they automatically pick up the new default of 1 with no YAML edit required. The commented example line was updated to describe the current default (1) rather than the old one (3), and a new commented block was added showing the controllers: 3 + controller_ips: [...] opt-in shape for anyone who copies this file as a starting point and wants multi-controller semantics.

Backward-compat verification performed vs. NOT performed (be precise)

Performed in this sandboxed development environment: aci-sim validate on the repo topology.yaml (OK, now reporting 1 controller/site), the full unit suite (counts below), and manual aci-sim show checks confirming controller_ips surfaces correctly for both the auto-derived and explicit cases.

NOT performed from this sandboxed environment (no SSH/network access here): the real production-chain regression gate this project's acceptance bar calls for (autoACI sandbox e2e, aci-py MS-TN1 live_chain.py) — this PR explicitly changes generated node counts, which is exactly the kind of change that gate exists to catch (a hardcoded node-count assertion inside autoACI itself, if one exists, would only surface there). The main thread's independent re-run of the real Ansible/aci-py/autoACI chains on ACI hardware is the authoritative backward-compat gate for this PR.