Why black-box AI fails in maintenance.

What a black box means on the plant floor

In predictive maintenance, a black-box model is one that maps condition-monitoring data (vibration, temperature, motor current, oil analysis) to an output (an anomaly score, a failure probability, a remaining-life estimate) through internal parameters that carry no engineering meaning. A deep neural network is the typical case: millions of weights, none of which corresponds to a crack length, a bearing clearance, or a load. Ask it why it raised an alert and there is nothing to inspect. The reasoning is distributed across the weights, and the weights mean nothing individually.

The label is not about secrecy, and it is not about complexity either. A model can be proprietary yet interpretable, or open-source yet opaque. Nor do post-hoc explanation tools change the category: as Cynthia Rudin argued in her influential Nature Machine Intelligence paper, an explanation generated after the fact is a second model approximating the first, and it can be plausible while being unfaithful to what the original model actually computed. A black box with an explanation bolted on is still a black box.

None of this matters much when a model recommends a film or ranks advertisements, because being wrong is cheap there. Maintenance is a high-stakes setting: acting on a prediction costs a maintenance window and lost production, while ignoring one risks a failure. That asymmetry is why the three failure modes below bite harder here than almost anywhere else.

Failure one: the training data does not exist

A supervised model learns failure only from examples of failure. To predict a bearing seizure, it needs many complete run-to-failure trajectories, recorded under conditions like yours, with the failure mode labeled. Critical industrial equipment is maintained precisely so those trajectories never complete. A fleet of turbines might yield a handful of genuine failures in a decade, each a different mode under different conditions. The asset owner's success at maintenance is the model's data famine.

The field's own literature concedes the point. Lei and colleagues' systematic review of machinery prognostics found that most published methods are validated on laboratory or simulated data, with industrial deployments far scarcer than the publication volume suggests. Fink and colleagues, in a review written with practitioners from the rail and aviation industries, list the scarcity of labeled failure data among the central obstacles to deploying deep learning in prognostics and health management at all.

On the plant floor the consequences take two familiar shapes. A model trained mostly on healthy data can flag that something looks unusual, but cannot name the failure mode or time it, which turns every alert into an investigation. And a model trained on other plants' failures transfers poorly, because degradation signatures depend on the specific machine, mounting, and duty. A model that has never seen your failure can neither recognize it nor predict it.

Failure two: it breaks silently when the plant changes

Plants change constantly. A new product mix shifts loads, summer raises ambient temperatures, an overhaul resets wear surfaces, a sensor gets recalibrated or drifts. A black-box model learned correlations that held in its training data, and when the operating distribution shifts, those correlations quietly stop holding. The model keeps producing numbers with the same confidence as before, because nothing inside it can notice that the world has moved.

Machine learning research has given this failure a precise name. D'Amour and colleagues showed that modern training pipelines are underspecified: many different models achieve identical test scores, yet behave very differently once conditions shift, and validation metrics cannot tell you which one you deployed. Translated to maintenance: you cannot tell from a good test score whether the model learned something that tracks the physics or a shortcut that merely correlated with it during training.

A physical model fails differently. Fatigue follows the same law after a regime change; conservation of energy survives a product changeover. Physics keeps constraining predictions exactly where the data has run out. A black box has no such fallback, so its characteristic failure is quiet wrongness discovered later. In maintenance the worst version is the confident all-clear issued during the very period when changed conditions made the model blind.

Failure three: predictions nobody will stake a shutdown on

Picture the decision the model exists to trigger. An alert says a gearbox will fail within three weeks and recommends pulling the next outage forward, at a six-figure cost in lost production. The reliability engineer asks why. The honest answer from a black-box system is that a score crossed a threshold. No load, no damage mechanism, no remaining margin, nothing to check against the machine. The rational response is to wait for corroboration, and by the time corroboration arrives the prediction has added nothing.

After the first false alarm handled this way, rational caution hardens into alert fatigue, and the system gets demoted to one more trend line somebody glances at. This is not operator stubbornness. Fink and colleagues list the lack of interpretability, and the resulting lack of trust, among the main reasons deep learning solutions stall before industrial deployment. A prediction that cannot be verified does not just fail to help; it spends the credibility the next prediction would have needed.

Rudin's argument lands hardest exactly here: for high-stakes decisions, stop explaining black boxes and use models that are interpretable in the problem's own terms. In maintenance the problem's own terms are physical quantities. A prediction stated as loads, damage state, and remaining margin can be audited against the actual machine, argued with, and acted on. A score cannot.

The same questions, asked of both model types

The contrast becomes concrete when you put the same operational questions to a black-box model and to one built around the physics of the equipment:

Directional comparison; individual implementations vary. Physics-informed approaches are surveyed in Karniadakis et al. (2021).

This is why the research field's response to the black-box problem has not been better explanation tools but a different class of model. Physics-informed machine learning builds the governing equations into the model itself, attacking the data problem and the trust problem at the same time: the physics supplies what the missing failure data cannot, and the predictions stay in terms an engineer can verify. Our guide on physics-informed neural networks explains how that works in detail.

Five questions to ask before trusting a maintenance AI

Whatever you make of the physics-informed argument, the failure modes above translate directly into procurement questions. A system worth trusting with critical equipment should have a convincing answer to all five:

1. Show me a prediction in engineering terms. Loads, temperatures, damage state, remaining margin. If the system can only show a score, every alert will end in an argument instead of a work order.
2. Tell me what happens outside training conditions. How does the model behave after a regime change, an overhaul, or a sensor swap, and how does it signal that it has left familiar territory?
3. State the data you need from me, honestly. If the answer is years of labeled failure history, ask how many of your critical failure modes have even one complete recorded example.
4. Quantify the uncertainty. A remaining-life estimate without a confidence interval is a guess wearing a number. The width of the interval is itself information a planner needs.
5. Prove it cannot predict the impossible. What mechanism stops the model from outputting a state that violates conservation laws or the machine's own physical limits?

These five questions are not neutral: they are precisely where black-box systems have no good answers and physics-grounded systems do. That is the point. The questions a reliability team naturally asks before betting a shutdown on a prediction are the questions a model with no knowledge of the machine cannot answer.

How Vermilion answers them

Vermilion was built on the other side of this divide from the start. Its Large Reasoning Model reasons against first-principles digital twins of the equipment it watches (bearing dynamics, gear-mesh harmonics, fluid-film stability), with constitutive parameters fitted per asset under known physical priors. Degradation is tracked as a physical state, not a score, and every prediction is the output of running that state forward under the asset's expected duty.

That design answers the five questions directly. Predictions arrive in engineering quantities a reliability team can check against the machine. The physics keeps constraining estimates after regime changes, where training data has run out. No years of labeled failure history are required, because the equations already encode how failures unfold. Uncertainty is carried through to every estimate. And a state that violates the physics is refused before it ever reaches an operator.