Predictive modeling is pervasive across a variety of industries including energy, urban air mobility, additive manufacturing, finance, epigenetics, and insurance to cite a few. But over time, prediction accuracy inevitably deteriorates for most machine learning (ML) models due to the drift despite having acceptable performance in a test setting.
Drift is a natural occurrence as cumulative information changes over time but should be regulated to ensure an ML model’s prediction remains relevant in real-time when decisions are being made. This research proposes a drift mitigation framework (DMF) to obtain an effective sample size in a scalable and unbiased manner for sustaining the model’s performance. The team applies this model to decision making in microgrid energy systems.
To mitigate from drifting, scenario selection was used as an alternative sampling technique to obtain an effective sample from historical data. Basically, the team looked back at different situations from the past to make sure their prediction tool didn't miss anything important. These scenarios were then evaluated to ensure the obtained sample effectively captures the general population behavior. The findings indicated that the selected sample maintained similar statistical traits as the theoretical distributions tailored to the full original population and training data. Therefore, there was no notable change observed between the population data and the selected sample across each factor.
The assessments revealed that their approach captured essential specifics without overlooking the broader context. This suggests that when employing these prediction tools for crucial decisions in the future, whatever application they might be, they may possess enhanced accuracy and dependability amidst evolving circumstances.
This work was supported in part by funding from the AFOSR Award No: FA9550-18-1-0075.
Darville, J., Yavuz, A., Runsewe, T., & Celik, N. (2023). Effective sampling for drift mitigation in machine learning using scenario selection: A microgrid case study. Applied Energy, 341, 121048. https://doi.org/10.1016/j.apenergy.2023.121048
