其他
mlr3：嵌套重抽样
主要包括4个部分的内容，今天学习第二部分。
模型调优
调整超参数
方法一：通过`tuninginstancesinglecrite`和`tuner`训练模型
方法二：通过`autotuner`训练模型
超参数设定的方法
参数依赖
嵌套重抽样
进行嵌套重抽样
评价模型
把超参数应用于模型
Hyperband调参
特征选择
filters
计算分数
计算变量重要性
组合方法（wrapper methods）
自动选择
嵌套重抽样
既有外部重抽样，也有内部重抽样，彼此嵌套，可以很好的解决过拟合问题，得到更加稳定的模型。
对于概念不清楚的可以自行百度学习，就不在这里赘述了。
可使用下图帮助理解：
进行嵌套重抽样
内部使用4折交叉验证：
rm(list = ls())
library(mlr3verse)
library(mlr3tuning)
learner <- lrn("classif.rpart")
resampling <- rsmp("cv", folds = 4)
measure <- msr("classif.ce")
search_space <- ps(cp = p_dbl(lower = 0.001, upper = 0.1))
terminator <- trm("evals", n_evals = 5)
tuner <- tnr("grid_search", resolution = 10)
at <- AutoTuner$new(learner, resampling, measure, terminator, tuner,search_space)
外部使用3折交叉验证：
task <- tsk("pima")
outer_resampling <- rsmp("cv", folds = 3)
rr <- resample(task, at, outer_resampling, store_models = T)
## INFO [20:51:33.072] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 3/3)
## INFO [20:51:33.094] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]'
## INFO [20:51:33.096] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.102] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.106] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.119] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.131] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.143] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.156] [mlr3] Finished benchmark
## INFO [20:51:33.174] [bbotk] Result of batch 1:
## INFO [20:51:33.175] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.175] [bbotk] 0.012 0.2519531 0 0 0.04
## INFO [20:51:33.175] [bbotk] uhash
## INFO [20:51:33.175] [bbotk] 0758955e-fa36-4f81-88c6-76a7175cc14c
## INFO [20:51:33.175] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.182] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.185] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.198] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.210] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.222] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.239] [mlr3] Finished benchmark
## INFO [20:51:33.258] [bbotk] Result of batch 2:
## INFO [20:51:33.259] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.259] [bbotk] 0.089 0.2636719 0 0 0.03
## INFO [20:51:33.259] [bbotk] uhash
## INFO [20:51:33.259] [bbotk] 6555ef8e-faa6-4673-8bed-2afcd8845e08
## INFO [20:51:33.260] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.266] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.270] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.282] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.294] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.305] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.318] [mlr3] Finished benchmark
## INFO [20:51:33.337] [bbotk] Result of batch 3:
## INFO [20:51:33.338] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.338] [bbotk] 0.067 0.2519531 0 0 0.02
## INFO [20:51:33.338] [bbotk] uhash
## INFO [20:51:33.338] [bbotk] f621d4cb-3168-48ba-8353-54a9cad824de
## INFO [20:51:33.339] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.345] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.349] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.361] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.374] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.386] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.399] [mlr3] Finished benchmark
## INFO [20:51:33.419] [bbotk] Result of batch 4:
## INFO [20:51:33.420] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.420] [bbotk] 0.023 0.2480469 0 0 0.04
## INFO [20:51:33.420] [bbotk] uhash
## INFO [20:51:33.420] [bbotk] 580ad0d0-3c1b-4307-986f-7522f5373c7d
## INFO [20:51:33.421] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.428] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.431] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.444] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.456] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.472] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.484] [mlr3] Finished benchmark
## INFO [20:51:33.503] [bbotk] Result of batch 5:
## INFO [20:51:33.504] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.504] [bbotk] 0.078 0.2636719 0 0 0.04
## INFO [20:51:33.504] [bbotk] uhash
## INFO [20:51:33.504] [bbotk] c296d2bb-a989-4839-a6c0-9f0cc07952e7
## INFO [20:51:33.507] [bbotk] Finished optimizing after 5 evaluation(s)
## INFO [20:51:33.507] [bbotk] Result:
## INFO [20:51:33.508] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:33.508] [bbotk] 0.023 <list[2]> <list[1]> 0.2480469
## INFO [20:51:33.528] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 1/3)
## INFO [20:51:33.551] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]'
## INFO [20:51:33.552] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.558] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.562] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.574] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.586] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.598] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.610] [mlr3] Finished benchmark
## INFO [20:51:33.628] [bbotk] Result of batch 1:
## INFO [20:51:33.629] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.629] [bbotk] 0.078 0.28125 0 0 0.03
## INFO [20:51:33.629] [bbotk] uhash
## INFO [20:51:33.629] [bbotk] 04026d47-855c-4db5-8668-56a9ea1625f6
## INFO [20:51:33.630] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.636] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.640] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.652] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.664] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.681] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.693] [mlr3] Finished benchmark
## INFO [20:51:33.713] [bbotk] Result of batch 2:
## INFO [20:51:33.713] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.713] [bbotk] 0.067 0.2871094 0 0 0.03
## INFO [20:51:33.713] [bbotk] uhash
## INFO [20:51:33.713] [bbotk] 8d4b8536-d360-4da5-a3cd-aa030728ef07
## INFO [20:51:33.714] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.721] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.724] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.736] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.748] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.760] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.772] [mlr3] Finished benchmark
## INFO [20:51:33.792] [bbotk] Result of batch 3:
## INFO [20:51:33.793] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.793] [bbotk] 0.1 0.28125 0 0 0.02
## INFO [20:51:33.793] [bbotk] uhash
## INFO [20:51:33.793] [bbotk] 479564cb-312e-4f30-8879-bd198a57a60e
## INFO [20:51:33.794] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.800] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.804] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.816] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.828] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.841] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.853] [mlr3] Finished benchmark
## INFO [20:51:33.874] [bbotk] Result of batch 4:
## INFO [20:51:33.875] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.875] [bbotk] 0.089 0.28125 0 0 0.03
## INFO [20:51:33.875] [bbotk] uhash
## INFO [20:51:33.875] [bbotk] c27a8bb6-68bf-436e-9a44-959adc485d2e
## INFO [20:51:33.875] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:33.885] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:33.889] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:33.901] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:33.913] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:33.925] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:33.937] [mlr3] Finished benchmark
## INFO [20:51:33.956] [bbotk] Result of batch 5:
## INFO [20:51:33.957] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:33.957] [bbotk] 0.023 0.2949219 0 0 0.04
## INFO [20:51:33.957] [bbotk] uhash
## INFO [20:51:33.957] [bbotk] a47f52d1-3225-4898-b755-3c360b74365d
## INFO [20:51:33.960] [bbotk] Finished optimizing after 5 evaluation(s)
## INFO [20:51:33.960] [bbotk] Result:
## INFO [20:51:33.961] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:33.961] [bbotk] 0.078 <list[2]> <list[1]> 0.28125
## INFO [20:51:33.982] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 2/3)
## INFO [20:51:34.004] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]'
## INFO [20:51:34.005] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.012] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.015] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.028] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.040] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.053] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.066] [mlr3] Finished benchmark
## INFO [20:51:34.087] [bbotk] Result of batch 1:
## INFO [20:51:34.088] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.088] [bbotk] 0.023 0.2382812 0 0 0.02
## INFO [20:51:34.088] [bbotk] uhash
## INFO [20:51:34.088] [bbotk] 4b196172-eab1-4adc-aaa2-12b51dca7b06
## INFO [20:51:34.088] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.095] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.098] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.110] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.122] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.134] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.146] [mlr3] Finished benchmark
## INFO [20:51:34.166] [bbotk] Result of batch 2:
## INFO [20:51:34.167] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.167] [bbotk] 0.089 0.2617188 0 0 0.02
## INFO [20:51:34.167] [bbotk] uhash
## INFO [20:51:34.167] [bbotk] 531313fe-9a50-4402-bda8-c4b773c989e4
## INFO [20:51:34.167] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.174] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.177] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.189] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.202] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.214] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.227] [mlr3] Finished benchmark
## INFO [20:51:34.247] [bbotk] Result of batch 3:
## INFO [20:51:34.248] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.248] [bbotk] 0.078 0.2617188 0 0 0.03
## INFO [20:51:34.248] [bbotk] uhash
## INFO [20:51:34.248] [bbotk] d5d914f0-0808-49e6-adce-7c5f0898d306
## INFO [20:51:34.249] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.256] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.259] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.271] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.284] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.299] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.311] [mlr3] Finished benchmark
## INFO [20:51:34.331] [bbotk] Result of batch 4:
## INFO [20:51:34.331] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.331] [bbotk] 0.034 0.2421875 0 0 0.01
## INFO [20:51:34.331] [bbotk] uhash
## INFO [20:51:34.331] [bbotk] b49f1619-d251-4464-8efe-48f32373d753
## INFO [20:51:34.332] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.339] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.342] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.354] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.366] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.378] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.391] [mlr3] Finished benchmark
## INFO [20:51:34.411] [bbotk] Result of batch 5:
## INFO [20:51:34.412] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.412] [bbotk] 0.012 0.2382812 0 0 0.02
## INFO [20:51:34.412] [bbotk] uhash
## INFO [20:51:34.412] [bbotk] 375b973a-a946-4c77-94f7-451b29f07cb6
## INFO [20:51:34.415] [bbotk] Finished optimizing after 5 evaluation(s)
## INFO [20:51:34.415] [bbotk] Result:
## INFO [20:51:34.416] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:34.416] [bbotk] 0.023 <list[2]> <list[1]> 0.2382812
这里演示的数据集比较小，大数据可以使用并行化技术，将在后面介绍。
评价模型
提取内部抽样的模型表现：
extract_inner_tuning_results(rr)
## iteration cp classif.ce learner_param_vals x_domain task_id
## 1: 1 0.078 0.2812500 <list[2]> <list[1]> pima
## 2: 2 0.023 0.2382812 <list[2]> <list[1]> pima
## 3: 3 0.023 0.2480469 <list[2]> <list[1]> pima
## learner_id resampling_id
## 1: classif.rpart.tuned cv
## 2: classif.rpart.tuned cv
## 3: classif.rpart.tuned cv
提取内部抽样的存档：
extract_inner_tuning_archives(rr)
## iteration cp classif.ce x_domain_cp runtime_learners timestamp
## 1: 1 0.078 0.2812500 0.078 0.03 2022-02-27 20:51:33
## 2: 1 0.067 0.2871094 0.067 0.03 2022-02-27 20:51:33
## 3: 1 0.100 0.2812500 0.100 0.02 2022-02-27 20:51:33
## 4: 1 0.089 0.2812500 0.089 0.03 2022-02-27 20:51:33
## 5: 1 0.023 0.2949219 0.023 0.04 2022-02-27 20:51:33
## 6: 2 0.023 0.2382812 0.023 0.02 2022-02-27 20:51:34
## 7: 2 0.089 0.2617188 0.089 0.02 2022-02-27 20:51:34
## 8: 2 0.078 0.2617188 0.078 0.03 2022-02-27 20:51:34
## 9: 2 0.034 0.2421875 0.034 0.01 2022-02-27 20:51:34
## 10: 2 0.012 0.2382812 0.012 0.02 2022-02-27 20:51:34
## 11: 3 0.012 0.2519531 0.012 0.04 2022-02-27 20:51:33
## 12: 3 0.089 0.2636719 0.089 0.03 2022-02-27 20:51:33
## 13: 3 0.067 0.2519531 0.067 0.02 2022-02-27 20:51:33
## 14: 3 0.023 0.2480469 0.023 0.04 2022-02-27 20:51:33
## 15: 3 0.078 0.2636719 0.078 0.04 2022-02-27 20:51:33
## batch_nr warnings errors resample_result task_id learner_id
## 1: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 2: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 3: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 4: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 5: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 6: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 7: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 8: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 9: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 10: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 11: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 12: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 13: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 14: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 15: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## resampling_id
## 1: cv
## 2: cv
## 3: cv
## 4: cv
## 5: cv
## 6: cv
## 7: cv
## 8: cv
## 9: cv
## 10: cv
## 11: cv
## 12: cv
## 13: cv
## 14: cv
## 15: cv
可以看到和上面的结果是不一样的哦，每一折都有5次迭代，这就和我们设置的参数有关系了。
查看外部重抽样的模型表现：
rr$score()[,9]
## classif.ce
## 1: 0.2460938
## 2: 0.2656250
## 3: 0.2890625
查看平均表现：
rr$aggregate()
## classif.ce
## 0.2669271
把超参数应用于模型
at$train(task)
## INFO [20:51:34.578] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=5, k=0]'
## INFO [20:51:34.580] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.587] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.590] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.603] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.616] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.629] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.642] [mlr3] Finished benchmark
## INFO [20:51:34.660] [bbotk] Result of batch 1:
## INFO [20:51:34.660] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.660] [bbotk] 0.045 0.25 0 0 0.02
## INFO [20:51:34.660] [bbotk] uhash
## INFO [20:51:34.660] [bbotk] c7c8bda8-3435-45db-8950-d316cde2cf0a
## INFO [20:51:34.661] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.668] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.671] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.684] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.700] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.711] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.724] [mlr3] Finished benchmark
## INFO [20:51:34.743] [bbotk] Result of batch 2:
## INFO [20:51:34.744] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.744] [bbotk] 0.1 0.2682292 0 0 0.02
## INFO [20:51:34.744] [bbotk] uhash
## INFO [20:51:34.744] [bbotk] 313d9692-68fc-4010-978f-e8aed3183c0e
## INFO [20:51:34.745] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.751] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.755] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.767] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.779] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.792] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.804] [mlr3] Finished benchmark
## INFO [20:51:34.825] [bbotk] Result of batch 3:
## INFO [20:51:34.825] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.825] [bbotk] 0.067 0.2552083 0 0 0.05
## INFO [20:51:34.825] [bbotk] uhash
## INFO [20:51:34.825] [bbotk] 6e4a5402-b0f7-4753-9b4e-9148943c0d96
## INFO [20:51:34.826] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.833] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.836] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.852] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.864] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.876] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.888] [mlr3] Finished benchmark
## INFO [20:51:34.907] [bbotk] Result of batch 4:
## INFO [20:51:34.908] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.908] [bbotk] 0.089 0.2682292 0 0 0.03
## INFO [20:51:34.908] [bbotk] uhash
## INFO [20:51:34.908] [bbotk] c2bcd001-218b-4d5b-a3fe-040f757ecdf3
## INFO [20:51:34.909] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:34.915] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:34.919] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:34.931] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:34.944] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:34.956] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:34.970] [mlr3] Finished benchmark
## INFO [20:51:34.990] [bbotk] Result of batch 5:
## INFO [20:51:34.991] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:34.991] [bbotk] 0.012 0.2434896 0 0 0.04
## INFO [20:51:34.991] [bbotk] uhash
## INFO [20:51:34.991] [bbotk] fdca679b-4117-4d26-974c-26509cba1d9d
## INFO [20:51:34.993] [bbotk] Finished optimizing after 5 evaluation(s)
## INFO [20:51:34.994] [bbotk] Result:
## INFO [20:51:34.994] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:34.994] [bbotk] 0.012 <list[2]> <list[1]> 0.2434896
现在模型就可以应用于新的数据集了。
以上过程也是有简便写法的，但是需要注意，这里的mlr3tuning需要用github版的，cran版的还有bug，不知道修复了没：
rr1 <- tune_nested(
method = "grid_search",
resolution = 10,
task = task,
learner = learner,
inner_resampling = resampling,
outer_resampling = outer_resampling,
measure = measure,
term_evals = 20,
search_space = search_space
)
## INFO [20:51:35.045] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 1/3)
## INFO [20:51:35.067] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [20:51:35.069] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.075] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.079] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.091] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.103] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.115] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.128] [mlr3] Finished benchmark
## INFO [20:51:35.145] [bbotk] Result of batch 1:
## INFO [20:51:35.146] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.146] [bbotk] 0.1 0.2578125 0 0 0.01
## INFO [20:51:35.146] [bbotk] uhash
## INFO [20:51:35.146] [bbotk] 80577e6f-8f05-4c37-b5ef-134057641a2a
## INFO [20:51:35.147] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.154] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.157] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.170] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.182] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.194] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.207] [mlr3] Finished benchmark
## INFO [20:51:35.230] [bbotk] Result of batch 2:
## INFO [20:51:35.231] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.231] [bbotk] 0.034 0.2578125 0 0 0.03
## INFO [20:51:35.231] [bbotk] uhash
## INFO [20:51:35.231] [bbotk] 4ee26d60-fd52-4092-9dce-e76256898fb5
## INFO [20:51:35.232] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.238] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.242] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.254] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.266] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.278] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.291] [mlr3] Finished benchmark
## INFO [20:51:35.311] [bbotk] Result of batch 3:
## INFO [20:51:35.312] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.312] [bbotk] 0.001 0.2832031 0 0 0.04
## INFO [20:51:35.312] [bbotk] uhash
## INFO [20:51:35.312] [bbotk] b1fbb284-6124-4b56-aa19-8e74a0e444be
## INFO [20:51:35.312] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.319] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.323] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.335] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.348] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.360] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.373] [mlr3] Finished benchmark
## INFO [20:51:35.396] [bbotk] Result of batch 4:
## INFO [20:51:35.397] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.397] [bbotk] 0.023 0.2734375 0 0 0.05
## INFO [20:51:35.397] [bbotk] uhash
## INFO [20:51:35.397] [bbotk] ea1b6bff-ca1b-47f0-8297-363cd92bac23
## INFO [20:51:35.397] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.404] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.407] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.419] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.431] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.443] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.456] [mlr3] Finished benchmark
## INFO [20:51:35.475] [bbotk] Result of batch 5:
## INFO [20:51:35.476] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.476] [bbotk] 0.078 0.2578125 0 0 0.03
## INFO [20:51:35.476] [bbotk] uhash
## INFO [20:51:35.476] [bbotk] 1dd96aca-e602-44ce-91b6-3f74c3dad71d
## INFO [20:51:35.477] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.484] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.487] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.500] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.512] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.525] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.541] [mlr3] Finished benchmark
## INFO [20:51:35.560] [bbotk] Result of batch 6:
## INFO [20:51:35.561] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.561] [bbotk] 0.067 0.2578125 0 0 0.04
## INFO [20:51:35.561] [bbotk] uhash
## INFO [20:51:35.561] [bbotk] 4a0910ae-8c83-4359-af5b-f78d61228ab1
## INFO [20:51:35.562] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.568] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.571] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.584] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.596] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.608] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.620] [mlr3] Finished benchmark
## INFO [20:51:35.641] [bbotk] Result of batch 7:
## INFO [20:51:35.642] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.642] [bbotk] 0.012 0.2910156 0 0 0.01
## INFO [20:51:35.642] [bbotk] uhash
## INFO [20:51:35.642] [bbotk] 57d4c50c-659a-4cb3-a2ac-ba46685d0ce2
## INFO [20:51:35.643] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.650] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.653] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.666] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.678] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.690] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.706] [mlr3] Finished benchmark
## INFO [20:51:35.725] [bbotk] Result of batch 8:
## INFO [20:51:35.726] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.726] [bbotk] 0.089 0.2578125 0 0 0.01
## INFO [20:51:35.726] [bbotk] uhash
## INFO [20:51:35.726] [bbotk] 8155a3ea-3ce3-4b7c-8df0-c9e98fe87096
## INFO [20:51:35.727] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.733] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.736] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.748] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.760] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.772] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.785] [mlr3] Finished benchmark
## INFO [20:51:35.805] [bbotk] Result of batch 9:
## INFO [20:51:35.806] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.806] [bbotk] 0.056 0.2578125 0 0 0.03
## INFO [20:51:35.806] [bbotk] uhash
## INFO [20:51:35.806] [bbotk] 795783dc-cc4d-44e4-938c-49eb43ddc814
## INFO [20:51:35.807] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.813] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.817] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:35.829] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:35.874] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.888] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:35.900] [mlr3] Finished benchmark
## INFO [20:51:35.920] [bbotk] Result of batch 10:
## INFO [20:51:35.921] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:35.921] [bbotk] 0.045 0.2578125 0 0 0.04
## INFO [20:51:35.921] [bbotk] uhash
## INFO [20:51:35.921] [bbotk] 5fe8732f-9047-43fb-9f51-b2805e245fda
## INFO [20:51:35.924] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [20:51:35.924] [bbotk] Result:
## INFO [20:51:35.925] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:35.925] [bbotk] 0.1 <list[2]> <list[1]> 0.2578125
## INFO [20:51:35.945] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 2/3)
## INFO [20:51:35.968] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [20:51:35.969] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:35.976] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:35.979] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:35.992] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.004] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.016] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.029] [mlr3] Finished benchmark
## INFO [20:51:36.047] [bbotk] Result of batch 1:
## INFO [20:51:36.047] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.047] [bbotk] 0.089 0.2597656 0 0 0.02
## INFO [20:51:36.047] [bbotk] uhash
## INFO [20:51:36.047] [bbotk] d91839a4-f116-4314-bdba-f6ac7b1c522c
## INFO [20:51:36.052] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.061] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.065] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.077] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.090] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.102] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.114] [mlr3] Finished benchmark
## INFO [20:51:36.133] [bbotk] Result of batch 2:
## INFO [20:51:36.134] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.134] [bbotk] 0.056 0.2597656 0 0 0.03
## INFO [20:51:36.134] [bbotk] uhash
## INFO [20:51:36.134] [bbotk] ab912dd5-08ab-44aa-9a1d-8b5ce074eb13
## INFO [20:51:36.135] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.142] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.145] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.158] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.170] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.182] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.194] [mlr3] Finished benchmark
## INFO [20:51:36.214] [bbotk] Result of batch 3:
## INFO [20:51:36.215] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.215] [bbotk] 0.1 0.2636719 0 0 0.04
## INFO [20:51:36.215] [bbotk] uhash
## INFO [20:51:36.215] [bbotk] 9568b45d-fb79-44b0-8406-11af967fbb8f
## INFO [20:51:36.216] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.222] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.226] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.238] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.250] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.262] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.280] [mlr3] Finished benchmark
## INFO [20:51:36.303] [bbotk] Result of batch 4:
## INFO [20:51:36.303] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.303] [bbotk] 0.067 0.2519531 0 0 0.02
## INFO [20:51:36.303] [bbotk] uhash
## INFO [20:51:36.303] [bbotk] 5c06d6a7-ca4a-4f21-85f5-99bd6708166f
## INFO [20:51:36.304] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.311] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.314] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.326] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.338] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.351] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.363] [mlr3] Finished benchmark
## INFO [20:51:36.383] [bbotk] Result of batch 5:
## INFO [20:51:36.384] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.384] [bbotk] 0.045 0.2558594 0 0 0.02
## INFO [20:51:36.384] [bbotk] uhash
## INFO [20:51:36.384] [bbotk] 0c659322-58b5-4533-a9b2-0656e3440c8a
## INFO [20:51:36.384] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.391] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.395] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.407] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.420] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.432] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.445] [mlr3] Finished benchmark
## INFO [20:51:36.471] [bbotk] Result of batch 6:
## INFO [20:51:36.472] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.472] [bbotk] 0.001 0.2675781 0 0 0.05
## INFO [20:51:36.472] [bbotk] uhash
## INFO [20:51:36.472] [bbotk] 7c690a3e-0a40-4644-8f78-d82448b220f8
## INFO [20:51:36.472] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.479] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.483] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.495] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.507] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.519] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.531] [mlr3] Finished benchmark
## INFO [20:51:36.551] [bbotk] Result of batch 7:
## INFO [20:51:36.552] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.552] [bbotk] 0.078 0.2597656 0 0 0.01
## INFO [20:51:36.552] [bbotk] uhash
## INFO [20:51:36.552] [bbotk] e7ac0c70-0aaa-4127-9044-48650b9d0f71
## INFO [20:51:36.553] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.559] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.563] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.576] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.588] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.600] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.613] [mlr3] Finished benchmark
## INFO [20:51:36.633] [bbotk] Result of batch 8:
## INFO [20:51:36.634] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.634] [bbotk] 0.034 0.2558594 0 0 0.04
## INFO [20:51:36.634] [bbotk] uhash
## INFO [20:51:36.634] [bbotk] 6bb79880-9b17-4ed7-832b-1d8782ba1472
## INFO [20:51:36.635] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.641] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.645] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.657] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.675] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.688] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.700] [mlr3] Finished benchmark
## INFO [20:51:36.720] [bbotk] Result of batch 9:
## INFO [20:51:36.721] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.721] [bbotk] 0.012 0.25 0 0 0.03
## INFO [20:51:36.721] [bbotk] uhash
## INFO [20:51:36.721] [bbotk] b3b225e5-de05-4f40-8847-2b709a9b07ba
## INFO [20:51:36.721] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.728] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.732] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.745] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.757] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.769] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.782] [mlr3] Finished benchmark
## INFO [20:51:36.801] [bbotk] Result of batch 10:
## INFO [20:51:36.802] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.802] [bbotk] 0.023 0.2597656 0 0 0.02
## INFO [20:51:36.802] [bbotk] uhash
## INFO [20:51:36.802] [bbotk] 232394d3-a9a2-48b2-b012-afb796c993b6
## INFO [20:51:36.805] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [20:51:36.805] [bbotk] Result:
## INFO [20:51:36.806] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:36.806] [bbotk] 0.012 <list[2]> <list[1]> 0.25
## INFO [20:51:36.827] [mlr3] Applying learner 'classif.rpart.tuned' on task 'pima' (iter 3/3)
## INFO [20:51:36.850] [bbotk] Starting to optimize 1 parameter(s) with '<TunerGridSearch>' and '<TerminatorEvals> [n_evals=20, k=0]'
## INFO [20:51:36.851] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.861] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.866] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.880] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.892] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.904] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.916] [mlr3] Finished benchmark
## INFO [20:51:36.933] [bbotk] Result of batch 1:
## INFO [20:51:36.934] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:36.934] [bbotk] 0.089 0.2441406 0 0 0.02
## INFO [20:51:36.934] [bbotk] uhash
## INFO [20:51:36.934] [bbotk] 52c0b642-a12d-4dad-822f-e59b61907d2e
## INFO [20:51:36.935] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:36.942] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:36.945] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:36.957] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:36.970] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:36.982] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:36.995] [mlr3] Finished benchmark
## INFO [20:51:37.014] [bbotk] Result of batch 2:
## INFO [20:51:37.015] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.015] [bbotk] 0.034 0.25 0 0 0.03
## INFO [20:51:37.015] [bbotk] uhash
## INFO [20:51:37.015] [bbotk] 11841250-c15b-4860-9a14-03a575396f7d
## INFO [20:51:37.016] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.023] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.026] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.038] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.054] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.069] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.082] [mlr3] Finished benchmark
## INFO [20:51:37.101] [bbotk] Result of batch 3:
## INFO [20:51:37.102] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.102] [bbotk] 0.1 0.2441406 0 0 0
## INFO [20:51:37.102] [bbotk] uhash
## INFO [20:51:37.102] [bbotk] 41e89a1e-fce3-4fa5-a8f6-1ef2da7699bf
## INFO [20:51:37.103] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.110] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.113] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.125] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.137] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.150] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.162] [mlr3] Finished benchmark
## INFO [20:51:37.182] [bbotk] Result of batch 4:
## INFO [20:51:37.183] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.183] [bbotk] 0.023 0.2617188 0 0 0.04
## INFO [20:51:37.183] [bbotk] uhash
## INFO [20:51:37.183] [bbotk] 0b687b2b-f4e9-40f0-8f7c-fc163653baf2
## INFO [20:51:37.184] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.190] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.194] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.206] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.218] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.235] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.249] [mlr3] Finished benchmark
## INFO [20:51:37.269] [bbotk] Result of batch 5:
## INFO [20:51:37.270] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.270] [bbotk] 0.067 0.2441406 0 0 0.03
## INFO [20:51:37.270] [bbotk] uhash
## INFO [20:51:37.270] [bbotk] cd9643f8-bcfe-40f3-adf6-c6ba9ad7c866
## INFO [20:51:37.270] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.277] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.280] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.292] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.304] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.316] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.329] [mlr3] Finished benchmark
## INFO [20:51:37.349] [bbotk] Result of batch 6:
## INFO [20:51:37.350] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.350] [bbotk] 0.045 0.2441406 0 0 0.03
## INFO [20:51:37.350] [bbotk] uhash
## INFO [20:51:37.350] [bbotk] e8e65bcc-da0f-4048-b4e3-97f022670221
## INFO [20:51:37.351] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.357] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.361] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.373] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.386] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.398] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.417] [mlr3] Finished benchmark
## INFO [20:51:37.437] [bbotk] Result of batch 7:
## INFO [20:51:37.438] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.438] [bbotk] 0.001 0.2832031 0 0 0.03
## INFO [20:51:37.438] [bbotk] uhash
## INFO [20:51:37.438] [bbotk] efc6f90a-70ca-4b7c-8868-18e9d194bae9
## INFO [20:51:37.439] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.445] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.448] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.460] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.472] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.484] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.497] [mlr3] Finished benchmark
## INFO [20:51:37.517] [bbotk] Result of batch 8:
## INFO [20:51:37.517] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.517] [bbotk] 0.078 0.2441406 0 0 0.04
## INFO [20:51:37.517] [bbotk] uhash
## INFO [20:51:37.517] [bbotk] ab79d8ea-fee4-49aa-a1b2-ba67ba3766a9
## INFO [20:51:37.518] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.525] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.528] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.541] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.553] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.566] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.585] [mlr3] Finished benchmark
## INFO [20:51:37.605] [bbotk] Result of batch 9:
## INFO [20:51:37.606] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.606] [bbotk] 0.012 0.2675781 0 0 0.04
## INFO [20:51:37.606] [bbotk] uhash
## INFO [20:51:37.606] [bbotk] 0b549169-2d69-442c-98ce-d20651a2d46e
## INFO [20:51:37.606] [bbotk] Evaluating 1 configuration(s)
## INFO [20:51:37.613] [mlr3] Running benchmark with 4 resampling iterations
## INFO [20:51:37.616] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 3/4)
## INFO [20:51:37.628] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 2/4)
## INFO [20:51:37.640] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 1/4)
## INFO [20:51:37.652] [mlr3] Applying learner 'classif.rpart' on task 'pima' (iter 4/4)
## INFO [20:51:37.665] [mlr3] Finished benchmark
## INFO [20:51:37.684] [bbotk] Result of batch 10:
## INFO [20:51:37.685] [bbotk] cp classif.ce warnings errors runtime_learners
## INFO [20:51:37.685] [bbotk] 0.056 0.2441406 0 0 0.02
## INFO [20:51:37.685] [bbotk] uhash
## INFO [20:51:37.685] [bbotk] a289e821-a615-414e-a68f-ba66ed39508b
## INFO [20:51:37.688] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [20:51:37.688] [bbotk] Result:
## INFO [20:51:37.689] [bbotk] cp learner_param_vals x_domain classif.ce
## INFO [20:51:37.689] [bbotk] 0.089 <list[2]> <list[1]> 0.2441406
这个rr1本质上和rr是一样的，
print(rr1)
## <ResampleResult> of 3 iterations
## * Task: pima
## * Learner: classif.rpart.tuned
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations
print(rr)
## <ResampleResult> of 3 iterations
## * Task: pima
## * Learner: classif.rpart.tuned
## * Warnings: 0 in 0 iterations
## * Errors: 0 in 0 iterations
查看内部抽样表现：
extract_inner_tuning_results(rr1)
## iteration cp classif.ce learner_param_vals x_domain task_id
## 1: 1 0.100 0.2578125 <list[2]> <list[1]> pima
## 2: 2 0.012 0.2500000 <list[2]> <list[1]> pima
## 3: 3 0.089 0.2441406 <list[2]> <list[1]> pima
## learner_id resampling_id
## 1: classif.rpart.tuned cv
## 2: classif.rpart.tuned cv
## 3: classif.rpart.tuned cv
提取归档资料：
extract_inner_tuning_archives(rr1)
## iteration cp classif.ce x_domain_cp runtime_learners timestamp
## 1: 1 0.100 0.2578125 0.100 0.01 2022-02-27 20:51:35
## 2: 1 0.034 0.2578125 0.034 0.03 2022-02-27 20:51:35
## 3: 1 0.001 0.2832031 0.001 0.04 2022-02-27 20:51:35
## 4: 1 0.023 0.2734375 0.023 0.05 2022-02-27 20:51:35
## 5: 1 0.078 0.2578125 0.078 0.03 2022-02-27 20:51:35
## 6: 1 0.067 0.2578125 0.067 0.04 2022-02-27 20:51:35
## 7: 1 0.012 0.2910156 0.012 0.01 2022-02-27 20:51:35
## 8: 1 0.089 0.2578125 0.089 0.01 2022-02-27 20:51:35
## 9: 1 0.056 0.2578125 0.056 0.03 2022-02-27 20:51:35
## 10: 1 0.045 0.2578125 0.045 0.04 2022-02-27 20:51:35
## 11: 2 0.089 0.2597656 0.089 0.02 2022-02-27 20:51:36
## 12: 2 0.056 0.2597656 0.056 0.03 2022-02-27 20:51:36
## 13: 2 0.100 0.2636719 0.100 0.04 2022-02-27 20:51:36
## 14: 2 0.067 0.2519531 0.067 0.02 2022-02-27 20:51:36
## 15: 2 0.045 0.2558594 0.045 0.02 2022-02-27 20:51:36
## 16: 2 0.001 0.2675781 0.001 0.05 2022-02-27 20:51:36
## 17: 2 0.078 0.2597656 0.078 0.01 2022-02-27 20:51:36
## 18: 2 0.034 0.2558594 0.034 0.04 2022-02-27 20:51:36
## 19: 2 0.012 0.2500000 0.012 0.03 2022-02-27 20:51:36
## 20: 2 0.023 0.2597656 0.023 0.02 2022-02-27 20:51:36
## 21: 3 0.089 0.2441406 0.089 0.02 2022-02-27 20:51:36
## 22: 3 0.034 0.2500000 0.034 0.03 2022-02-27 20:51:37
## 23: 3 0.100 0.2441406 0.100 0.00 2022-02-27 20:51:37
## 24: 3 0.023 0.2617188 0.023 0.04 2022-02-27 20:51:37
## 25: 3 0.067 0.2441406 0.067 0.03 2022-02-27 20:51:37
## 26: 3 0.045 0.2441406 0.045 0.03 2022-02-27 20:51:37
## 27: 3 0.001 0.2832031 0.001 0.03 2022-02-27 20:51:37
## 28: 3 0.078 0.2441406 0.078 0.04 2022-02-27 20:51:37
## 29: 3 0.012 0.2675781 0.012 0.04 2022-02-27 20:51:37
## 30: 3 0.056 0.2441406 0.056 0.02 2022-02-27 20:51:37
## iteration cp classif.ce x_domain_cp runtime_learners timestamp
## batch_nr warnings errors resample_result task_id learner_id
## 1: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 2: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 3: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 4: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 5: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 6: 6 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 7: 7 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 8: 8 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 9: 9 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 10: 10 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 11: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 12: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 13: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 14: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 15: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 16: 6 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 17: 7 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 18: 8 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 19: 9 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 20: 10 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 21: 1 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 22: 2 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 23: 3 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 24: 4 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 25: 5 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 26: 6 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 27: 7 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 28: 8 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 29: 9 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## 30: 10 0 0 <ResampleResult[22]> pima classif.rpart.tuned
## batch_nr warnings errors resample_result task_id learner_id
## resampling_id
## 1: cv
## 2: cv
## 3: cv
## 4: cv
## 5: cv
## 6: cv
## 7: cv
## 8: cv
## 9: cv
## 10: cv
## 11: cv
## 12: cv
## 13: cv
## 14: cv
## 15: cv
## 16: cv
## 17: cv
## 18: cv
## 19: cv
## 20: cv
## 21: cv
## 22: cv
## 23: cv
## 24: cv
## 25: cv
## 26: cv
## 27: cv
## 28: cv
## 29: cv
## 30: cv
## resampling_id
查看模型表现：
rr1$aggregate()
## classif.ce
## 0.2682292
rr1$score()
## task task_id learner learner_id
## 1: <TaskClassif[49]> pima <AutoTuner[41]> classif.rpart.tuned
## 2: <TaskClassif[49]> pima <AutoTuner[41]> classif.rpart.tuned
## 3: <TaskClassif[49]> pima <AutoTuner[41]> classif.rpart.tuned
## resampling resampling_id iteration prediction
## 1: <ResamplingCV[19]> cv 1 <PredictionClassif[20]>
## 2: <ResamplingCV[19]> cv 2 <PredictionClassif[20]>
## 3: <ResamplingCV[19]> cv 3 <PredictionClassif[20]>
## classif.ce
## 1: 0.2539062
## 2: 0.2578125
## 3: 0.2929688
“注意，使用
tune_nested()之后，并没有提供方法应用于新的数据集，在咨询开发者之后，得到的说法是：tune_nested()是一种评估算法在整个数据集中的表现的方法，不是用于挑选合适的超参数的方法。重抽样过程会产生很多超参数组合，不应该用于模型中。
以上就是今天的内容，希望对你有帮助哦！欢迎点赞、在看、关注、转发！
欢迎在评论区留言或直接添加我的微信！
欢迎关注我的公众号：医学和生信笔记
“医学和生信笔记 公众号主要分享：1.医学小知识、肛肠科小知识；2.R语言和Python相关的数据分析、可视化、机器学习等；3.生物信息学学习资料和自己的学习笔记！