LT Tuning: Round 1 & 1.5

Summary of intial tuning results for the Landtrendr algorithm, plus bonus results from tunings using the multiple disturbance accuracy assesment protocol.

Madeleine Desrochers
2023-02-14

Initial Tuning: Single Disturbance Detection

Table 1: Summary of accuracy metrics for all parameter combinations using Landtrendr’s default ‘most reccent’ disturbance mapping protocol. Assesment was conducted against harvest records from UHW and HWF.
file name short name Precision Recall F1 Accuracy
lt_nbr_greatest_tuning_recoverythreshold_75 recovery threshold 0.75 0.6402 0.2964 0.4052 0.7796
lt_nbr_greatest_tuning_defaults defaults 0.6671 0.2214 0.3325 0.7738
rasters/lt_gee_nbr_greatest_tuning_maxsegments_8 max segmetns 08 0.6716 0.2193 0.3306 0.7740
rasters/lt_gee_nbr_greatest_tuning_maxsegments_12 max segments 12 0.6686 0.2243 0.3359 0.7743
lt_gee_nbr_greatest_tuning_vertexovershoot_1 vertex overshoot 1 0.6724 0.2213 0.3330 0.7744
lt_gee_nbr_greatest_tuning_vertexovershoot_5 vertex overshoot 5 0.6689 0.2229 0.3344 0.7742
lt_gee_nbr_greatest_tuning_recoverythreshold_05 recovery threshold 0.5 0.6818 0.2884 0.4054 0.7853
lt_gee_nbr_greatest_tuning_pval_1 pval 0.1 0.6688 0.2236 0.3352 0.7743
lt_gee_nbr_greatest_tuning_pval_2 pval 0.2 0.6720 0.2259 0.3381 0.7751
lt_gee_nbr_greatest_tuning_modelproportion_1 modelproportion 1 0.6692 0.2203 0.3315 0.7739
lt_gee_nbr_greatest_tuning_modelproportion_5 model proportion 0.5 0.6692 0.2203 0.3315 0.7739
lt_gee_nbr_greatest_tuning_maxsegments_16 max segments 16 0.6687 0.2250 0.3367 0.7744
lt_gee_nbr_greatest_tuning_maxsegments_14 max segments 14 0.6671 0.2237 0.3351 0.7741
lt_gee_nbr_greatest_tuning_recoverythreshold_08 recovery threshold 0.8 0.6290 0.2977 0.4041 0.7778
lt_gee_nbr_greatest_tuning_recoverythreshold_09 recovery threshold 0.9 0.6135 0.2992 0.4022 0.7752
lt_gee_nbr_greatest_tuning_recoverythreshold_1 recoverythreshold 1 0.6026 0.3001 0.4006 0.7732
lt_gee_nbr_greatest_tuning_combo_1 combo 01 0.6216 0.3016 0.4062 0.7772
lt_gee_nbr_greatest_tuning_combo_2 combo 02 0.6078 0.3067 0.4077 0.7752
lt_gee_nbr_greatest_tuning_combo_3 combo 03 0.6102 0.3074 0.4088 0.7757
lt_gee_nbr_greatest_tuning_combo_4 combo 04 0.6267 0.3037 0.4092 0.7783
lt_gee_nbr_greatest_tuning_combo_5 combo 05 0.6636 0.2979 0.4112 0.7840
lt_gee_nbr_greatest_tuning_combo_6 combo 06 0.6751 0.2954 0.4110 0.7853
lt_gee_nbr_greatest_tuning_combo_7 combo 07 0.6101 0.3088 0.4100 0.7757
lt_gee_nbr_greatest_tuning_combo_8 combo 08 0.6260 0.3051 0.4102 0.7783
lt_gee_nbr_greatest_tuning_combo_10 combo 10 0.6106 0.3067 0.4083 0.7756
lt_gee_nbr_greatest_tuning_combo_9 combo 09 0.6680 0.2964 0.4106 0.7845
lt_gee_nbr_greatest_tuning_combo_11 combo 11 0.6676 0.2945 0.4087 0.7841
lt_gee_nbr_greatest_tuning_combo_12 combo 12 0.6602 0.2995 0.4121 0.7836
lt_gee_nbr_greatest_tuning_combo12_dur_1 duration 1 NA 0.0000 NA 0.7419
lt_gee_nbr_greatest_tuning_combo12_dur_2 duration 2 0.6416 0.2428 0.3523 0.7740
lt_gee_nbr_greatest_tuning_combo12_mmu_10 mmu 10 0.6765 0.2877 0.4037 0.7844
lt_gee_nbr_greatest_tuning_combo12_mmu_15 mmu 15 0.6900 0.2743 0.3925 0.7842
Table 2: Parameters used in named tuning combinations found in Table 1.
Short Name Max Segments Vertex Count Overshoot Recovery Threshold P Value Threshold Best Model Proportion
combo 01 12 1 0.75 0.10 0.75
combo 02 16 1 0.75 0.10 0.75
combo 03 16 5 0.75 0.10 0.75
combo 04 12 5 0.75 0.10 0.75
combo 05 16 5 0.50 0.10 0.75
combo 06 12 5 0.50 0.10 0.75
combo 07 16 5 0.75 0.20 0.75
combo 08 12 5 0.75 0.20 0.75
combo 10 16 1 0.75 0.05 0.75
combo 09 16 5 0.50 0.05 0.75
combo 11 12 1 0.50 0.10 0.75
combo 12 16 1 0.50 0.10 0.75
Results of intial tuning with single disturbance detection. Values for accuracy metrics are shown relative to the results from the default parameter set.

Figure 1: Results of intial tuning with single disturbance detection. Values for accuracy metrics are shown relative to the results from the default parameter set.

Multiple Disturbance Detection Results

Tunings selected for evaluation with the multiple disturbance detection protocol were
- Combo 12: highest F1 score
- Combo 5: 2nd highest F1 and accuracy
- Combo 6: highest accuracy
- Combo 7: highest recall (excluding map generation parameters that can’t be used with MDDP)
- Recovery Threshold 0.75: highest precision, also the combination of parameters used in the JOF paper
- Recovery Threshold

Results of selected high performance tunings applied with the multiple disturbance detection protocol. Results are reported relative to the default parameter results.

Figure 2: Results of selected high performance tunings applied with the multiple disturbance detection protocol. Results are reported relative to the default parameter results.

Comparison of tuning results between the single detection outputs and the multiple disturbance detection protocol. Positive values indicate an increase in the metric value from evaluation of single disturbances to evaluation of multiple disturbances, whereas a negative value indicates a decrease in the metric.

Figure 3: Comparison of tuning results between the single detection outputs and the multiple disturbance detection protocol. Positive values indicate an increase in the metric value from evaluation of single disturbances to evaluation of multiple disturbances, whereas a negative value indicates a decrease in the metric.

Table 3: Results of application of high performing tuning combinations to the multiple disturbance detection protocol.
short_name Precision Recall F1 Accuracy
multi disturbance combo 12 0.4069283 0.4872448 0.4434795 0.9224310
multi disturbance combo 5 0.4185054 0.4799470 0.4471253 0.9247123
multi disturbance combo 6 0.4372268 0.4557587 0.4463005 0.9282677
multi disturbance combo 7 0.4152033 0.4884253 0.4488477 0.9239141
multi disturbance recovery threshold 0.75 0.4443259 0.4345781 0.4393979 0.9296608
multi disturbance defaults 0.5219555 0.3032785 0.3836437 0.9381872

Corrections

If you see mistakes or want to suggest changes, please create an issue on the source repository.

Citation

For attribution, please cite this work as

Desrochers (2023, Feb. 13). CAFRI Labs: LT Tuning: Round 1 & 1.5. Retrieved from https://cafri-labs.github.io/acceptable-growing-stock/posts/lt-tuning-round-1-15/

BibTeX citation

@misc{desrochers2023lt,
  author = {Desrochers, Madeleine},
  title = {CAFRI Labs: LT Tuning: Round 1 & 1.5},
  url = {https://cafri-labs.github.io/acceptable-growing-stock/posts/lt-tuning-round-1-15/},
  year = {2023}
}