Area of Applicability for Deep Learning: Exploring Latent Space Geometry of Earth Observation Models

Problem setting

In the real world, models might be used for inference on samples which are outside of their domain of application.

Figure 1: Example of overconfident predictions for OOD samples (Hou 2023.)

Problem setting

We learn a model that maps from input space to output space \(f: \mathcal X \mapsto \mathcal Y\) and combine it with a confidence score function \(s: \mathbb{R^d} \mapsto \mathbb{R^1}\).

Combined with a threshold \(\lambda\), we induce a decision function \(g\):

\[ g_{\lambda}(x|s) = \mathbb{1}{[s(x)>\lambda]}. \tag{1}\]

We can now either evaluate if \(g\) is successful in detecting OOD-samples: \[ g(x) = \begin{cases} OOD, \; \text{if} \; g(x) = 1, \\ ID, \; \text{if} \; g(x) = 0, \end{cases} \tag{2}\]

or, we must be successful in reducing the model’s risk on the non-rejected samples:

\[ (f,g)(x) = \begin{cases} reject, \; \text{if} \; g(x) = 1, \\ f(x), \; \text{if} \; g(x) = 0. \end{cases} \tag{3}\]

Area of Applicability

• Area of Applicability (AOA) defines the domain in which a model is trusted to operate within estimated performance measures

• Prior-art in shallow machine learning:

  • AOA based on dissimilarity index (DI) in input space (Meyer and Pebesma 2021)
  • recently extended by the local point density (LPD) in input space (Schumacher et al. 2025)

• How to translate to deep learning models?

  • inputs are often high-dimensional and complex (images, time series, etc.)
  • input space distances ignore the model’s internal representation of learned (spatial) features
  • CNNs learn compact and task-relevant latent representations of the input data

Working hypothesis

Latent representations learned by deep models provide a computationally efficient and reliable basis for defining an Area of Applicability for Earth‑observation models.

• Do KNN-based confidence scores in the latent space of a deep model correlate with model performance?
• How does the chosen distance function and its parameters affect the results?

Dataset

• We use the refined BigEarthNet (reBEN) dataset (Clasen et al. 2025), which is a large-scale benchmark for remote sensing image analysis
• It consists of 480,038 Sentinel-2 image patches from 10 European countries, covering 125 tiles
• Each image patch comes with a pixel-wise land cover classification based on the CORINE Land Cover dataset
• This allows to construct and analyze three different tasks:
  • C: image-level multi-label classification
  • R: image-level multi-target regression
  • S: pixel-level segmentation

Dataset

Dataset

Methodology

Figure 3: Workflow figure indicating the studies methodology.

Results

• the intercepts for all three tasks are positive and significant, indicating that the error increases when applying leave-one-country-out evaluation (33% to 56%)
• applying selection reduces the error (30-35%), the effect is significant for all tasks.
• for the regression task using cosine distance reduces the error (3%)
• using cosine distance increases the error for the classification and segmentation task (13% and 6%)

Results

• for classification and segmentation, increasing K reduces the error, but the effect is not significant for all values of K
• for regression, no effect of increasing K is observed
• applying PCA decreases the error for the regression task, but increases it for the classification and segmentation task

Discussion

• Investigating the Area of Applicability for deep learning models in Earth observation is crucial for ensuring reliable and trustworthy model performance in real-world applications
• Compared to the classical OOD detection setting, the AOA framework emphasizes the importance of defining a domain of application for a model based on estimated performance measures, rather than just detecting OOD samples
• KNN-based confidence scores in the latent space of a deep model correlate with model performance, with a considerable effect size that depends on the task and the choice of distance function
• the choice of distance function and its parameters moderately effect the results, but there is no clear pattern across tasks
• further research is needed to explore if these findings translate to other datasets and tasks

Classification results

Classification results

In-distribution error versus out-of-distribution error (each dot represents the average over a country dataset).

Regression results

Regression results

In-distribution error versus out-of-distribution error (each dot represents the average over a country dataset).

Segmentation results

Segmentation results

In-distribution error versus out-of-distribution error (each dot represents the average over a country dataset).