1.2.0: This time it’s LCMAP

Lidar-based models using LCMAP primary and secondary classes as predictors. 2021-08-05

Evaluation Results

Last iteration of these models: 2021-05-30

Change Summary

• Added LCMAP primary and secondary land cover classifications as predictors
• Removed “n” (number of returns per plot) as a predictor, due to concerns about being non-standard across coverages with different return spacings

	RF (ranger)	GBM (LightGBM)	SVM (kernlab)	Ensemble (model weighted)	Ensemble (RMSE weighted)
RMSE	37.591	37.232	36.854	36.723	36.499
MBE	1.894	1.843	-0.726	1.821	1.015
R2	0.771	0.775	0.780	0.780	0.785

AGB Distribution

summary(bind_rows(training, testing)$agb_mgha)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.000   9.107  85.393  91.573 149.091 425.000 

Bootstrapping Results

Across 1000 bootstrap iterations, our ensemble model had a mean RMSE of 36.693 \(\pm\) 0.327.

RMSE Distribution

Plot Errors

Validation Results

RMSE	Min	Median	Max
Rf	34.695	40.522	45.031
Lgb	35.272	40.364	44.787
Svm	34.435	40.756	64.763
Ensemble	34.394	40.005	45.806

“Ensemble” here refers to the RMSE-weighted ensemble; the linear model ensemble is trained using the predictions from the validation stage (so doesn’t have any “test” data until we do the final evaluation).

R2	Min	Median	Max
rf	0.681	0.739	0.797
lgb	0.681	0.742	0.790
svm	0.408	0.734	0.796
ensemble	0.688	0.746	0.793

Metadata

Ensembles

• RMSE-weighted model weights:

      lgb        rf       svm 
0.3370106 0.3342122 0.3287773 

• Linear model weights:

Call:
lm(formula = agb_mgha ~ rf_pred * lgb_pred * svm_pred, data = pred_values)

Residuals:
     Min       1Q   Median       3Q      Max 
-126.487  -22.287   -0.039   14.119  219.826 

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
(Intercept)               -2.605e+00  5.866e-01  -4.440 9.04e-06 ***
rf_pred                    7.367e-01  6.368e-02  11.568  < 2e-16 ***
lgb_pred                   2.591e-01  7.381e-02   3.510 0.000448 ***
svm_pred                   6.707e-02  4.079e-02   1.644 0.100180    
rf_pred:lgb_pred          -1.172e-03  3.183e-04  -3.683 0.000231 ***
rf_pred:svm_pred          -1.590e-03  5.023e-04  -3.166 0.001549 ** 
lgb_pred:svm_pred          2.153e-03  5.963e-04   3.610 0.000306 ***
rf_pred:lgb_pred:svm_pred  2.471e-06  1.416e-06   1.745 0.080921 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 39.66 on 23492 degrees of freedom
Multiple R-squared:  0.7436,    Adjusted R-squared:  0.7435 
F-statistic:  9731 on 7 and 23492 DF,  p-value: < 2.2e-16

Coverages

• 17 coverages:
  • FEMA_FranklinStLawrence2016, FEMA_FultonSaratogaHerkimerFranklin2017, FEMA_GreatLakes2014, FEMA_OniedaSubbasin2016, NYSGPO_AlleganySteuben2016, NYSGPO_CayugaOswego_2018, NYSGPO_ColumbiaRensselaer2016, NYSGPO_ErieGeneseeLivingston2019, NYSGPO_MadisonOtsego_2015, NYSGPO_Southwest_spring_2017, NYSGPO_SouthwestB_fall_2017, NYSGPO_WarrenWashingtonEssex_2015, USGS_3County2014, USGS_ClintonEssexFranklin2014, USGS_LongIsland2014, USGS_NorthEast2011, USGS_Schoharie2014

\(n\) and \(p\)

• 1136 observations
  • 786 training
  • 350 testing
• 84 predictors
  • zmean, zmean_c, max, quad_mean, quad_mean_c, cv, cv_c, z_kurt, z_skew, L2, L3, L4, L_cv, L_skew, L_kurt, h10, h20, h30, h40, h50, h60, h70, h80, h90, h95, h99, hvol, cancov, rpc1, d10, d20, d30, d40, d50, d60, d70, d80, d90, precip, tmin, tmax, twi, slope, aspect, elev, tax_code_105, tax_code_112, tax_code_120, tax_code_210, tax_code_240, tax_code_241, tax_code_260, tax_code_270, tax_code_280, tax_code_312, tax_code_314, tax_code_322, tax_code_323, tax_code_910, tax_code_911, tax_code_912, tax_code_930, tax_code_931, tax_code_932, tax_code_941, tax_code_1000, tax_category_100, tax_category_200, tax_category_300, tax_category_900, tax_code_2000, lcpri4, lcpri6, lcpri1, lcpri2, lcpri3, lcpri5, lcsec3, lcsec2, lcsec1, lcsec4, lcsec5, lcsec8, lcpri8

Component Models

• Tuning used 5-fold CV
• Final hyperparameters:

Random forest:

$num.trees
[1] 500

$mtry
[1] 33

$min.node.size
[1] 9

$sample.fraction
[1] 0.25

$splitrule
[1] "maxstat"

$replace
[1] TRUE

$formula
agb_mgha ~ .

LGB:

$learning_rate
[1] 0.1

$nrounds
[1] 50

$num_leaves
[1] 24

$max_depth
[1] -1

$extra_trees
[1] TRUE

$min_data_in_leaf
[1] 10

$bagging_fraction
[1] 0.6

$bagging_freq
[1] 1

$feature_fraction
[1] 0.9

$min_data_in_bin
[1] 3

$lambda_l1
[1] 10

$lambda_l2
[1] 3

$force_col_wise
[1] TRUE

SVM:

$x
agb_mgha ~ .

$kernel
[1] "laplacedot"

$type
[1] "eps-svr"

$kpar
$kpar$sigma
[1] 0.001953125


$C
[1] 21

$epsilon
[1] 0.125