SKIM: Pruning Large Language Model Agents via Selective Knowledge Informed Masking

What we want from a pruned agent

	free decodingno grammar	constrained decodingdeployment
dense
standard
SKIM	not needed	best

Accuracy under constrained decoding

0.8B 2B 4B 9B

Wanda-spFLAPSlimGPTSKIM (ours)

Agents are served with constrained decoding: a grammar writes most of every tool call, and the model decides only a few tokens. SKIM prunes and distills for exactly those decisions, and its lead over standard compression grows with sparsity.

Abstract

Large Language Model (LLM) agents are increasingly deployed under structured generation, where grammar constraints enforce output format at inference time. Whereas existing compression methods target unconstrained generation, we first identify a fundamental mismatch: under structured generation, grammar constraints reduce the model's output distribution to valid tokens, and the logits matter most at the small set of positions that decide the agent's action. Motivated by this finding, we propose SKIM (Selective Knowledge-Informed Masking), the first pruning and distillation framework for LLM agents under structured generation, with both stages guided by token role. SKIM partitions token positions into context, decision, and format categories, and this partition drives two complementary axes: (1) category-aware pruning saliency scores that preserve parameters most relevant for decision positions; and (2) an on-policy distillation objective that aligns student and teacher on valid tokens, paired with category-aware feature matching. We evaluate SKIM on tool-calling benchmarks and within multi-step agent loops, across multiple model scales and sparsity levels. SKIM consistently improves accuracy over baselines, with its superiority becoming most striking in high-sparsity regimes where conventional methods struggle, demonstrating the potential of category-aware compression for deployment-time grammar-constrained agents.

Motivation

Under constrained decoding, only a few positions are the model's to decide

1The grammar blocks format-breaking tokens, so most positions have one valid token

2Standard compression preserves every token; SKIM preserves the decisions

Method

Results

Full-match accuracy of the generated tool calls. All pruned models are healed with the same distillation budget; structured is decoding under the grammar, the deployment setting, and free is decoding without it. SKIM is the best pruned model in most columns, and its margin is largest at 50% sparsity, where the baselines lose the most.

Analysis

BibTeX

@inproceedings{choi2026skim,
  title     = {{SKIM}: Pruning Large Language Model Agents via Selective Knowledge Informed Masking},
  author    = {Choi, Moonseok and Jeong, Jongwon and Kang, Minki and Nam, Giung and Lee, Juho},
  booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}