FeatCal: Feature Calibration for Post-Merging Models

Abstract

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

Figure 1. Feature drift after Task Arithmetic (TA) merging and FeatCal calibration in CLIP-ViT-B/32. Panels (a,b) use Stanford Cars: FeatCal moves features toward expert features, raises their mean cosine similarity from 0.60 to 0.84, and reduces mean L2 feature drift, with 46% less final-layer drift. Panel (c) reports per-task accuracy in the 8-task setting. Appendix B gives full 8-task feature views.

1. Introduction

Model merging composes task experts into one model, avoiding joint training, retraining, or deploying a separate model for each task. However, the merged model often still underperforms the experts it is intended to combine. This leaves a clear performance gap after merging in practice.

We analyze this gap through feature drift, the difference between features of the merged model and the task expert on the same input sample. Our layer by layer analysis decomposes this drift into upstream propagation and local mismatch at expert input features, then shows how local mismatches propagate through later layers in forward order and combine into final feature drift. We further analyze how final feature drift reaches the model output and becomes output drift, explaining how feature drift can affect output scores. Figure 1 illustrates this behavior on CLIP-ViT-B/32 merged with Task Arithmetic (TA), where final features move away from the Stanford Cars expert features and drift appears across multiple layers of the network after TA merging.

Surgery and ProbSurgery are related post-merging calibration methods: they identify feature drift at the final layer and train extra modules for calibration. Other intervention methods point to the same lesson: expert signals can help, but current methods often rely on task specific intervention parameters, extra modules at inference, or iterative optimization. These choices make calibration less direct for an already merged model and can leave the final model with an auxiliary inference path. This motivates efficient, direct calibration that preserves inference speed.

FeatCal uses our drift analysis as a design cue for calibrating the merged model. It uses a forward-order schedule suggested by the propagation view and calibrates the model layer by layer. We introduce a regularization term that keeps the calibrated model close to the merged model, which helps preserve the benefits of model merging and reduce overfitting to the small calibration set. The resulting objective has an efficient closed form solution, so calibration needs no gradient descent, iterative optimization, or extra modules at inference. We further introduce feature interpolation and anchor regularization to balance expert signals with the merged model and improve performance.

Empirically, Figure 1 shows the practical effect: FeatCal moves merged features toward expert features, reduces feature drift, and raises per-task accuracy after TA merging. On CLIP-ViT-B/32 8-task TA, it reaches 85.5% versus 77.0%/78.8% for Surgery/ProbSurgery, and on FLAN-T5-base GLUE it reaches 85.2% versus 83.7%/82.2%. The same trend holds on CLIP-ViT-L/14 WUDI, FLAN-T5-large, and MergeBench Llama-family LLM merging, where FeatCal improves TA by $+2.0$/$+2.3$ average points on 3B/8B models. On CLIP-ViT-B/32 TA, 8 examples per task reach 82.9%, and calibration with 256 examples per task takes 53 seconds, about 4x faster than both baselines under the same calibration protocol.

We summarize the main contributions of this work as follows: ① We develop a theory of feature drift after merging, with an exact decomposition into local mismatch and upstream propagation, forward order propagation, and a link to output drift. ② We introduce FeatCal, which efficiently calibrates merged model weights in forward order with closed form updates, without gradient descent, architecture changes, or extra modules at inference. ③ We validate FeatCal on CLIP, FLAN-T5, and MergeBench LLM benchmarks, where it outperforms related post-merging calibration baselines while using fewer samples and lower calibration cost without adding inference-time modules.

2. Related Work

Model Merging. Most model merging methods build a fused model by merging task experts directly in parameter space. Methods based on feature statistics or feature drift, such as RegMean, RegMean++, and LOT Merging, are closer in mechanism: they derive layer updates during merging from feature statistics, regression objectives, or an explicit feature drift objective. These methods define how to build the merged model. In contrast, FeatCal treats that model as the starting point, uses a small calibration set and task experts for post-merging calibration, and controls calibration strength through regularization. This stage separation matters because calibration must work with the feature drift left by a chosen merger instead of changing the merge rule itself. It helps preserve the benefits of model merging while reducing the risk of overfitting to calibration data.

Post-Merging Feature Calibration. Representation Surgery and follow-up methods operate on merged-model features through task-specific plugins, deeper interventions, probabilistic feature-drift modeling, or parameter-efficient modules. These closest post-merging alternatives establish that expert-guided feature calibration is useful. FeatCal differs in parameterization and deployment: instead of learning or deploying auxiliary intervention modules, it folds the calibration into the original linear module weights through closed-form regularized updates, leaving a single architecture-preserving model at inference time rather than an auxiliary intervention path.

3. Post-Merging Feature Drift: Problem Formulation and Properties

Before introducing FeatCal, we formalize post-merging feature drift and show how local mismatch is defined at expert input features, then propagated and combined in forward order across depth.

3.1 Layer-Wise Feature Drift

We consider $N$ task experts $\{M_i^{\expert}\}_{i=1}^N$ and a merged model $M^{\merge}$. As in standard weight-space merging, experts are fine-tuned from a common pretrained base. All models share the same architecture and contain $L$ layers. Let $\mathcal D_i$ be the data distribution of task $i$. For layer $\ell$, $f_{i,\ell}^{\expert}$ and $f_{\ell}^{\merge}$ denote the corresponding layer functions of the task expert and the merged model, respectively.

For an input sample $x$ from task $i$, define the expert and merged layer output features recursively as

Definition 1. [Layer-wise feature drift]

For task $i$, sample $x$, and layer $\ell$, the layer-wise feature drift is the difference between the merged-model feature and the corresponding task-expert feature at that layer:

This pointwise drift signal is the object propagated in forward order in the analysis below.

3.2 Local Mismatch and Drift Propagation

Proposition 1. [Exact layer-wise drift decomposition]

For every task $i$, sample $x$, and intermediate layer $\ell$, the drift decomposes as follows. By Eq. 1, $p_{i,1}(x)=0$.

Proof. The algebraic identity and its regularity details are deferred to Appendix C.

Interpretation. Proposition 1 decomposes layer-wise feature drift into two terms: local mismatch and upstream-drift propagation. The local mismatch $m_{i,\ell}(x)$ measures the feature mismatch caused by the difference between the merged and expert layer maps at the same expert input feature. The propagation term $p_{i,\ell}(x)$ measures how feature drift inherited from earlier layers changes the input feature of layer $\ell$ and is then carried into the output feature of this layer.

Proposition 2. [Layer-wise propagation of local mismatch]

Fix a task $i$, sample $x$, and layers $1,\dots,L$. Suppose that, for each layer $\ell$, $f_\ell^{\merge}$ is continuously differentiable on an open neighborhood containing the segment between $h_{i,\ell-1}^{\expert}(x)$ and $h_{i,\ell-1}^{\merge}(x)$. Let $A_{i,\ell}(x)$ denote the corresponding path-averaged local sensitivity operator, defined explicitly in Eq. 18. Then the drift obeys the layer-by-layer recursion

Since $e_{i,0}(x)=0$, the final feature drift is

The product composes compatible local maps. See Eq. 21 for the $s$-to-$t$ form.

Proof. The local sensitivity derivation and unrolled expansion are deferred to Appendix C.

Interpretation. Proposition 2 gives a simple forward-order view: local mismatches arise at individual layers, and their induced feature drift propagates through downstream layers to the final layer. Thus, final feature drift is a downstream combination of local mismatches from different layers. For residual networks, we further show that residual paths carry upstream feature drift through the skip connection. The drift can also grow under specific conditions. See Appendix D.

3.3 From Feature Drift to Output Drift

Definition 2. [Output drift for task scores]

For the merged model $M^{\merge}$, let $z_i^{\expert}(x)$ and $z_i^{\merge}(x)$ be the expert and merged task scores, each represented as a score vector. Let $\psi_i^{\expert}$ and $\psi_i^{\merge}$ map final features to task scores, with $z_i^{\expert}(x)=\psi_i^{\expert}(h_{i,L}^{\expert}(x))$ and $z_i^{\merge}(x)=\psi_i^{\merge}(h_{i,L}^{\merge}(x))$. The post-merging output drift is

In the logit or similarity settings used below, $z_i$ can contain class logits, CLIP candidate scores (scaled similarity scores over a fixed candidate set), or fixed-prefix decoder vocabulary logits.

Proposition 3. [Feature-to-output perturbation]

Suppose $\psi_i^{\merge}$ is locally $B_i^{\merge}(x)$-Lipschitz on the segment between $h_{i,L}^{\expert}(x)$ and $h_{i,L}^{\merge}(x)$. Here $B_i^{\merge}(x)$ locally bounds how much the merged task score map can change when its final-feature input moves along this segment. Then

where

is the score map mismatch. If the task score map is shared, this term is $0$.

Proof. This is the perturbation bound proved in Proposition 6.

Interpretation. Proposition 3 shows that final feature drift can propagate to output drift, thus further affect model outputs and task loss. For example, in a language model under a fixed prefix, token probabilities are obtained by applying softmax to output logits. Holding other logits fixed, a larger token logit gives a larger token probability, so output-logit drift can potentially induce probability drift and change the next-token choice. Cross-entropy loss is also tied to the probability assigned to the target token, so probability drift can induce loss drift. Detailed analysis is deferred to Appendix E.

4. FeatCal: Feature Calibration for Post-Merging Models

The preceding drift analysis guides the design of a direct calibration procedure for an already merged model. FeatCal visits layers in forward order, takes a current feature snapshot after earlier layer updates, and solves expert-guided calibration objectives for the modules in that layer with explicit regularization.

4.1 Basic Calibration Objective

Forward-order calibration. The drift analysis in Section 3.2 shows that feature drift arises from local mismatch terms that propagate layer by layer in forward order. FeatCal follows the same order by calibrating the merged model layer by layer: after earlier layers are calibrated, it recollects current calibrated-model features for the next layer before fitting that layer's module objectives. This schedule also reduces mismatch between features used during calibration and features exposed by the deployed calibrated model; Appendix F formalizes this source/deployed feature mismatch.

Why calibrate linear modules? Under this schedule, each layer provides a shared feature snapshot. Given the cached input features and expert target features for a module in that snapshot, the local surrogate can be defined for any linear module. In practice, FeatCal applies it to the modules configured for calibration. Linear modules are natural feature-mixing and projection points in the architectures we study, including attention and MLP linear modules. Once their calibrated-model input features are fixed, each linear module gives a tractable regularized regression problem with a closed-form update. The update replaces the existing merged weight, preserving the architecture without gradient descent, adapters, or inference-time modules. For other affine components, including bias parameters and LayerNorm affine parameters, we also design calibration updates, as detailed in Appendix G. In practice, these extra updates give limited gains: on CLIP, their average accuracy improvement is less than 0.5 percentage points. The main gains come from calibrating linear modules.

Linear module feature drift. For a fixed linear module inside the current layer, we define a module-local version of feature drift using the variables available at that module. Let $W_i,W^{\merge},W^{\base}\in\mathbb{R}^{m\times d}$ be the task-$i$ expert, merged, and base weights for this module. When processing the layer, we cache fixed input feature matrices $X_i^{\cali}$ and $X_i^{\expert}$ for this module on the same task-$i$ calibration samples. Here $X_i^{\cali}$ is produced by the prefix-calibrated model and is therefore the deployed input source, while $X_i^{\expert}$ is the corresponding task-expert feature matrix.

Following the layer-wise feature drift definition in Eq. 2, we define the linear module feature drift of a candidate calibrated weight $W$ by

As in the layer-wise analysis, this module-level drift can be interpreted as a combination of module-local mismatch and upstream-drift propagation.

Basic calibration objective. With input features fixed, the per-module calibration objective minimizes the overall linear module feature-drift error with a merged-weight penalty:

The quadratic penalty controls how far a single module update can move from the merged weights. This objective is a tractable module-local surrogate for reducing linear module feature drift, rather than an exact objective for end-to-end task risk or all-layer feature drift.

4.2 Feature Interpolation for Calibration Targets

Under the forward-order schedule, each layer's calibration should focus on the local mismatch at its current linear module rather than compensate for feature drift caused by upstream layers. Directly using expert input features for calibration can violate this goal. The gap between $X_i^{\cali}$ and $X_i^{\expert}$ already contains upstream feature drift, so fitting the direct target $W_iX_i^{\expert}$ may force the current linear module to fit drift left by earlier layers. We therefore introduce an interpolated target feature:

In Eq. 9, we replace the target term $W_iX_i^{\expert}$ with $W_iX_i^{\tgt}$. This target keeps the expert signal while making the local regression less aggressive under upstream drift.

4.3 Anchor Regularization

During calibration, the objective should use targets formed from expert features while keeping the update tied to the merged weights. The base model provides another useful reference. It is pretrained on large data before task specialization and can contain knowledge that a small calibration set does not cover. To use this reference, we include the base weight in the regularization term. The coefficients below let us control how calibration uses the merged and base references. For a linear module, we define the anchor weight as

The coefficient $\rho$ controls the anchor used by the quadratic penalty.

Combining the target interpolation in Eq. 10 and the anchor in Eq. 11 gives the practical objective:

The first term uses the interpolated target from Eq. 10 to fit the expert feature signal without using the raw expert input feature as a hard target. The second term uses the anchor from Eq. 11 to keep the update close to the chosen reference, with $\lambda>0$ controlling the regularization strength.

4.4 Task-Wise Scale Normalization and Closed-Form Solution

This subsection describes the update applied after feature collection. At each forward-order layer, FeatCal first caches current calibrated-model features and expert features for the modules calibrated in that layer. The cached features are then used to compute each linear module update separately, and the layer parameters are loaded after the layer's updates are formed. Thus the statistics below are module-local, even though feature collection is organized by layer.

For a fixed linear module in the current layer, let $n$ denote the calibration sample count for each task at this module. The $1/n$ factors form per-task empirical moments and prevent each task contribution from scaling directly with the sample count. We summarize task $i$ by the empirical feature statistics

Here, $G_i$ is the input second moment and $C_i$ is the target-input cross moment.

The objective in Eq. 12 is a matrix-valued ridge regression problem. To reduce scale sensitivity, we use stabilized inverse task weights with $\epsilon>0$,

With these stabilized task weights fixed, the module-wise objective becomes

The corresponding stationary condition for this quadratic objective is

Solving this linear system gives the closed-form update for this module

Because $G_i\succeq 0$, $\omega_i>0$, and $\lambda>0$, the inverse is well defined. In implementation, we also add $\epsilon I_d$ to the solve matrix as a numerical stabilizer.

The complete forward-order calibration procedure is given in Appendix H.

5. Experiments

5.1 Setup

Benchmarks. We use two public FusionBench settings and one MergeBench LLM setting : ① CLIP image classification. We use the FusionBench CLIP model merging benchmark with CLIP-ViT-B/32 and CLIP-ViT-L/14. The primary setting has 8 image tasks: SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD. The 14-task suite adds Flowers102, PCAM, FER2013, Oxford-IIIT Pet, STL10, and CIFAR100. The 20-task suite further adds CIFAR10, Food101, Fashion-MNIST, EMNIST Letters, KMNIST, and Rendered SST2. We follow prior merging protocols, report top-1 accuracy and task averages, and defer full per-task extended results to Appendix I. ② FLAN-T5 text generation. We evaluate FusionBench FLAN-T5-base and FLAN-T5-large merging on 8 prompted GLUE tasks: CoLA, MNLI, MRPC, QNLI, QQP, RTE, SST-2, and STS-B. The base experts are full fine-tuned models, while the large experts use LoRA fine-tuning. We merge task experts and evaluate generated text outputs. We report exact match accuracy except for STS-B, where we report Spearman's $\rho$, and average the 8 task scores. ③ MergeBench LLM merging. We evaluate Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct in the MergeBench domain-expert setting. The task suite covers mathematics, coding, instruction following, and general knowledge through MATH-500, GSM8K, HumanEval+, MBPP+, IFEval, and ARC-Challenge. HumanEval+ and MBPP+ report pass@1, and each table average is the mean over all 6 reported MergeBench tasks in this LLM setting.

Compared methods. We compare against pre-trained, single-task, and multi-task references when available. For CLIP, upstream mergers include Simple Averaging, Task Arithmetic, AdaMerging, and WUDI-Merging; for FLAN-T5 and MergeBench, we use Task Arithmetic. We apply FeatCal on top of different upstream mergers and compare with Surgery and ProbSurgery where available. Unless otherwise stated, upstream and baseline hyperparameters follow the FusionBench recipes.

Calibration setup. By default, FeatCal uses 256 calibration samples per task, calibrates layers in forward order, and applies the linear-weight update in Eq. 17. When enabled, it also applies the bias and LayerNorm affine updates in Appendix G; the main CLIP runs enable both, while the FLAN-T5 runs calibrate linear-module bias parameters but not LayerNorm affine parameters. For the main CLIP accuracy tables, we fix $\lambda=0.05$, $\rho=2.0$, and $\alpha=0.3$. For FLAN-T5, we use $\lambda=10^{-5}$, $\rho=2.0$, and $\alpha=0.6$. For MergeBench, we use $(\lambda,\rho,\alpha)=(10^{-5},0.5,0.15)$ for Llama-3.2-3B-Instruct and $(3.0,0.5,0.15)$ for Llama-3.1-8B-Instruct. Test sets are used only for final reporting. Section 5.5 includes a compact sensitivity diagnostic for the 8-task TA setting.

5.2 Results on CLIP Models

Table 1. Multi-task performance of CLIP-ViT-B/32 models on 8 image-classification tasks. All numbers are top-1 accuracy (%). Gray arrows show changes over upstream mergers.

Table 2. CLIP Extended Average Accuracy.

On B/32 8-task CLIP, FeatCal raises the 4 upstream averages from 66.3/67.5/82.7/84.5 to 83.2/85.5/88.1/88.8, beating Surgery and ProbSurgery in each block. The gains are largest for the weaker upstream mergers, but FeatCal also improves AdaMerging and WUDI-Merging, where the merged models are already close to the multi-task reference. This pattern aligns with the feature drift motivation: the same calibration step can recover large lost accuracy and still refine strong merged models without changing the merger itself. The best average, 88.8, is close to the 90.3 task expert average and above the 88.6 multi-task reference. For the TA and WUDI rows in Table 2, FeatCal remains above Surgery and ProbSurgery across all extended averages. On 20-task TA, FeatCal reaches 79.4 on B/32 and 84.7 on L/14, giving $+18.8$ and $+14.4$ points over TA. Full per-task results are in Appendix I.

5.3 Results on FLAN-T5 Models

Table 3. FLAN-T5 GLUE generation results. Scores are percentages: exact-match accuracy except STS-B, which reports Spearman's $\rho$; arrows show changes over Task Arithmetic, and bold marks best non-reference entries.

Table 3 shows that the GLUE gains hold for both FLAN-T5-base and FLAN-T5-large. For base, FeatCal improves Task Arithmetic by $+6.3$ average points, exceeds Surgery and ProbSurgery, and improves all 8 tasks, with the largest gains on MNLI and STS-B. For large, where LoRA-based Task Arithmetic is already strong, FeatCal gives smaller gains but still reaches the best post-TA average. This contrast suggests that feature calibration helps most when the merged generator has clear headroom, while still helping in the stronger setting.

5.4 Results on MergeBench LLMs

Table 4. MergeBench results on Llama-family models. All Task Arithmetic rows use scale $0.3$; arrows show changes over Task Arithmetic within each model block. Bold marks the best non-reference entry for each metric within the corresponding model block.

Table 4 shows that the gains extend to LLM domain expert merging. FeatCal improves the 6-task average over Task Arithmetic by $+2.0$ points on the 3B model and $+2.3$ points on the 8B model, outperforming Surgery and ProbSurgery in both model blocks. The largest gain is on IFEval for the 8B setting, where FeatCal improves Task Arithmetic by $+15.$ points.

5.5 Analysis

We collect 4 diagnostics that probe the mechanism, practical cost, robustness, and stability of post-merging feature calibration.

Figure 2. Feature-calibration diagnostics for TA on CLIP-ViT-B/32. (a) Task-wise final-feature cosine to experts; (b) per-sample cosine on Cars; (c) expert-cosine gain versus accuracy gain over TA w/ Surgery; (d) backbone task-vector cosine to same-task experts.

Feature-calibration diagnostics. Figure 2 tests whether TA w/ FeatCal gains coincide with final features closer to task experts. Panels (a) and (b) show larger expert cosine than Surgery in this setting: the macro average rises from 0.785 to 0.850 over TA w/ Surgery, and Stanford Cars has a mean per-sample gain of 0.084. This matches the intended mechanism because FeatCal calibrates the feature distribution reached by the deployed merged model. Panel (c) shows that expert-feature cosine is a diagnostic rather than a complete explanation: larger cosine gains often track accuracy gains, with an average $+8.64$-point gain over TA w/ Surgery, but EuroSAT and MNIST improve with small cosine changes, while GTSRB improves despite a slight decrease. Panel (d) highlights a feature-space and parameter-space mismatch: although FeatCal moves final features closer to experts, the calibrated backbone has lower same-task expert task-vector cosine than the TA baseline on every task. Thus, FeatCal calibrates the deployed feature distribution without bringing the backbone parameters into closer task-vector alignment, and the ProbSurgery comparison in Appendix J shows the same pattern with a stronger adapter baseline.

Table 5. Post-TA runtime at $n=256$, excluding final evaluation. Speedup is relative to Surgery.

[2][4.35em][c]#1$\times$ #2

Figure 5. Sample Efficiency.

Sample efficiency and calibration cost. Figure 5 reports post-TA sample efficiency on CLIP-ViT-B/32 TA with matched per-task budgets $n$. All methods start from the same Task Arithmetic model and differ only in the post-merging calibration stream. With $8$ examples per task, FeatCal already reaches $82.9\%$ average accuracy and then saturates near $85.5\%$, while Surgery and ProbSurgery rise more slowly. Table 5 fixes $n=256$: excluding final evaluation, the closed-form update takes 53s, $4.1\times$ faster than Surgery and $4.2\times$ faster than ProbSurgery, with lower GPU energy and much less peak CPU RSS. Together, FeatCal reaches most of its TA gain with few examples and keeps calibration cost low at $n=256$. Full per-budget accuracy, resource averages, hardware, and logging details are in Appendix K.

Robustness to corrupted calibration data.

Table 6. 8-task TA accuracy under corrupted calibration. Avg. includes clean, Gaussian, blur, and fog.

Figure 6. Calibration examples under corruptions.

We corrupt only the images used for post-merging calibration and evaluate on clean test sets from 8-task TA, isolating calibration data quality rather than test-time corruption robustness. Protocol details are in Appendix L.

FeatCal remains strongest in every reported setting, with a $78.0\%$ average over clean, Gaussian noise, motion blur, and fog, compared with $71.5\%$ for Surgery and $71.6\%$ for ProbSurgery. Corrupted runs remain below the clean reference, but the stable ordering in Figure 6 suggests that FeatCal extracts a reliable expert-feature calibration signal from imperfect data.

Ablation study.

Figure 3. CLIP-ViT-B/32 TA coefficient sweeps.

Figure 3 reports single-factor sensitivity in the 8-task TA setting. FeatCal works best in a conservative calibration regime. Overly aggressive targets or regularization can degrade performance. The ridge sweep is flat over small positive values, so the closed-form solve is not tied to a narrow regularization choice, while very large ridge strength suppresses the update. The interpolation sweep has a sharper target-side tradeoff: small and medium $\alpha$ values help, whereas the hard-expert endpoint can force local updates to absorb upstream drift that should be calibrated gradually across layers. The merged-base blend curve is most asymmetric: moderate extrapolation improves both upstream mergers, but large $\rho$ values push calibrated weights too far from the merged-base reference and especially hurt the WUDI variant. Overall, the ablation is a stability diagnostic with a broad conservative region rather than a single sharp optimum.

6. Conclusion

Feature drift frames post-merging calibration by showing how local mismatches can propagate through a merged model and affect outputs. FeatCal uses this signal by capturing features layer by layer and applying closed-form updates to linear modules, improving CLIP and FLAN-T5 mergers over Surgery and ProbSurgery in the main settings without adding modules at inference.

Acknowledgments

This work was supported by the Hong Kong RGC (TRS: T41-517/25-N; GRF: 15228325), ITC RAISe+ (RAI/24/1/086A), and the Research Institute for Generative Artificial Intelligence at PolyU.

_2026

BibTeX

@misc{gu-2026-featcal,
      title={FeatCal: Feature Calibration for Post-Merging Models},
      author={Yanggan Gu and Shuo Cai and Zihao Wang and Wenjun Wang and Yuanyi Wang and Pengkai Wang and Sirui Huang and Su Lu and Jianmin Wu and Hongxia Yang},
      year={2026},
      eprint={2605.13030},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2605.13030},
}