Detectability Bounds and Decision-Aware Evaluation for Change Detection in Heterogeneous Relational Event Streams: A Reproducible, Fully-Labeled Benchmark as the Empirical Instrument

Aditya Singh

Abstract: Change detection over heterogeneous relational event streams- where a fault is a temporal process that perturbs one of many correlated channels, the same symptom can arise from distinct latent causes, and detection must ultimately be judged by intervention cost rather than ranking alone- lacks two things the problem needs: a theory that says which faults are detectable and how quickly, and an evaluation protocol that scores detectors where operators act. We address both. First, we develop a decision-aware evaluation theory for budgeted intervention: an optimality result identifying the cost-minimizing budget-constrained policy under calibration, and a calibration-regret bound that turns probability quality into a quantified decision cost rather than a diagnostic convention. Second, we give per-fault information-theoretic detectability bounds via quickest-change-detection theory, computed from a generator's own parameters, that partition faults into singular (arrival-limited), regular (finite-information), and granularity-limited regimes; and a scope-invariance result characterizing when symmetric monitors generalize across fault scopes while trained detectors need not. To make these results measurable we require a stream with exact, timestamp-level ground truth - which commercial data cannot supply- so we instantiate them in CRM-IntegrityBench, an open, seed-reproducible generator for relational CRM event streams (five entity classes with foreign-key relations, seasonality, and provenance) with a parameterized corruption-injection framework implementing twelve fault families with exact onset, duration, scope, and per-event labels. The accompanying protocol scores detectors on discrimination, timeliness, calibration, and budget-constrained expected loss under leakage-safe chronological replay. Evaluating five CPU-reproducible detector families (rule-based monitoring, per-channel CUSUM, unsupervised anomaly detection, calibrated linear models, and gradient-boosted trees) across fifteen seeds (5.7 million events, 2,160 injected incidents), gradient-boosted trees lead ranking and calibration- AUPRC 0.833 versus 0.723 for a robust rules monitor (paired difference +0.109, 95% bootstrap CI [0.078, 0.145]), with the lowest Brier score and expected calibration error- while at a 1% intervention budget the budgeted expected loss is statistically indistinguishable across the calibrated detectors, because the budget saturates against the base rate; the detectors separate on loss only at looser budgets (≥ 1.5%). Detection separates sharply at the family level rather than the aggregate: each detector attains near-perfect detection on some fault families and near-zero on others, and several families are hard for every detector. A feature-leakage ablation shows the absolute numbers are optimistic- removing six fault-specific indicator channels costs every detector 0.10-0.44 AUPRC- but the comparative finding strengthens: the gradient-boosted margin over the deviation monitors more than doubles, from +0.109 to +0.243, once hand-built channels are withheld. No detector family dominates every family; the measured per-family detection ordering closely tracks the theory's predicted difficulty (Spearman ρ = -0.79, p = 0.002) - exactly the discrimination that multi-axis, decision-aware evaluation is designed to surface. Generator, configurations, and evaluation code are released for independent reproduction and extension.

Keywords: Change Detection, Relational Event Streams, Fault Detection, Decision Aware Evaluation, CRM Integrity Bench

How to Cite?: Aditya Singh, "Detectability Bounds and Decision-Aware Evaluation for Change Detection in Heterogeneous Relational Event Streams: A Reproducible, Fully-Labeled Benchmark as the Empirical Instrument", Volume 15 Issue 8, August 2026, International Journal of Science and Research (IJSR), Pages: 231-244, https://www.ijsr.net/getabstract.php?paperid=SR26804042413, DOI: https://dx.doi.org/10.21275/SR26804042413

Download Citation: APA | MLA | BibTeX | EndNote | RefMan