CAFRI Labs: Model Updates

Daniel Phoenix

knitr::opts_chunk$set(echo = FALSE)
library(dplyr)
library(tidyr)
library(ggplot2)
library(ggridges)
library(kableExtra)

Evaluation Results

Model training with WWE Linear Ensemble raster with LCMAP masking - stratified sampling

	RF (ranger)	GBM (LightGBM)	SVM (kernlab)	Ensemble (model weighted)	Ensemble (RMSE weighted)
RMSE	40.626	40.312	41.880	39.112	39.404
MBE	0.137	-0.375	0.242	-0.542	-0.002
R2	0.752	0.746	0.727	0.762	0.759

AGB Distribution

summary(bind_rows(training, testing)$agb)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00   71.41  140.75  140.23  208.25  294.52

Bootstrapping Results

Across 100 bootstrap iterations, our ensemble model had a mean RMSE of 40.131 \(\pm\) 0.163.

RMSE Distribution

Plot Errors

Validation Results

RMSE	Min	Median	Max
Rf	38.255	40.588	42.189
Lgb	38.204	39.901	41.810
Svm	39.039	41.735	44.960
Ensemble	36.728	39.070	41.066

R2	Min	Median	Max
rf	0.734	0.759	0.789
lgb	0.733	0.754	0.775
svm	0.693	0.731	0.765
ensemble	0.742	0.768	0.797

Metadata

Ensembles

RMSE-weighted model weights:

      lgb        rf       svm 
0.3381861 0.3340915 0.3277224

Linear model weights:


Call:
lm(formula = agb ~ rf_pred * lgb_pred * svm_pred, data = pred_values)

Residuals:
     Min       1Q   Median       3Q      Max 
-218.013  -21.018    1.227   22.487  196.860 

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
(Intercept)                7.663e+00  6.401e-01  11.971  < 2e-16 ***
rf_pred                    2.814e-01  1.827e-02  15.398  < 2e-16 ***
lgb_pred                   9.818e-02  1.540e-02   6.375 1.84e-10 ***
svm_pred                   4.179e-03  1.506e-02   0.277    0.781    
rf_pred:lgb_pred           2.867e-03  1.093e-04  26.234  < 2e-16 ***
rf_pred:svm_pred           1.263e-03  1.049e-04  12.035  < 2e-16 ***
lgb_pred:svm_pred          1.695e-03  1.111e-04  15.249  < 2e-16 ***
rf_pred:lgb_pred:svm_pred -1.358e-05  3.285e-07 -41.322  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 38.36 on 147892 degrees of freedom
Multiple R-squared:  0.7736,    Adjusted R-squared:  0.7736 
F-statistic: 7.218e+04 on 7 and 147892 DF,  p-value: < 2.2e-16

\(n\) and \(p\)

7043 observations
- 4930 training
- 2113 testing
18 predictors
- precip, tmax, tmin, twi, dem, aspect, slope, lt_nbr, lt_tcb, lt_tcg, lt_tcw, delta_nbr, delta_tcb, delta_tcg, delta_tcw, ysd, mag, chm

Component Models

Tuning used 5-fold CV
Final hyperparameters:

Random forest:

$num.trees
[1] 500

$mtry
[1] 4

$min.node.size
[1] 1

$sample.fraction
[1] 0.95

$splitrule
[1] "extratrees"

$replace
[1] FALSE

$formula
agb ~ .

LGB:

$learning_rate
[1] 0.1

$nrounds
[1] 500

$num_leaves
[1] 9

$max_depth
[1] -1

$extra_trees
[1] FALSE

$min_data_in_leaf
[1] 10

$bagging_fraction
[1] 0.9

$bagging_freq
[1] 1

$feature_fraction
[1] 0.4

$min_data_in_bin
[1] 10

$lambda_l1
[1] 0

$lambda_l2
[1] 0

$force_col_wise
[1] TRUE

SVM:

$x
agb ~ .

$kernel
[1] "laplacedot"

$type
[1] "eps-svr"

$kpar
$kpar$sigma
[1] 0.00390625


$C
[1] 64

$epsilon
[1] 0.03125

$nu
[1] 0.2

Model Updates