Scikit-learn：模型选择之调参grid search

悠悠 2022-09-25 11:26 217阅读 0赞

http://[blog.csdn.net/pipisorry/article/details/52268947][blog.csdn.net_pipisorry_article_details_52268947]

## Scikit-learn：并行调参Grid Search    ##

Grid Search: Searching for estimator parameters

scikit-learn中提供了pipeline(for estimator connection) & grid\_search(searching best parameters)进行并行调参

如使用scikit-learn做文本分类时：vectorizer取多少个word呢？预处理时候要过滤掉tf>max\_df的words，max\_df设多少呢？tfidftransformer只用tf还是加idf呢？classifier分类时迭代几次？学习率怎么设？ “循环一个个试”，这就是grid search要做的基本东西。[  
][Link 1]

[某小皮][Link 1]

# 调整模型的超参数 #

Hyper-parameters are parameters that are not directly learnt within estimators.In scikit-learn they are passed as arguments to the constructor of theestimator classes.

It is possible and recommended to search the hyper-parameter space for the best Cross-validation: evaluating estimator performance score.

Any parameter provided when constructing an estimator may be optimized in thismanner. Specifically, to find the names and current values for all parametersfor a given estimator, use:

estimator.get_params()

A search consists of:

*  an estimator (regressor or classifier such as `sklearn.svm.SVC()`);
 *  a parameter space;
 *  a method for searching or sampling candidates;
 *  a cross-validation scheme; and
 *  a [score function][].

[`GridSearchCV`][GridSearchCV] exhaustively considersall parameter combinations, while [`RandomizedSearchCV`][RandomizedSearchCV] can sample agiven number of candidates from a parameter space with a specifieddistribution.

## 穷尽网格搜索GridSearchCV ##

Gird Search：具体说，就是每种参数确定好几个要尝试的值，然后像一个网格一样，把所有参数值的组合遍历一下。优点是实现简单暴力，如果能全部遍历的话，结果比较可靠。缺点是太费时间了，特别像神经网络，一般尝试不了太多的参数组合。

param_grid = [
      {
       'C': [1, 10, 100, 1000], 'kernel': ['linear']},
      {
       'C': [1, 10, 100, 1000], 'gamma': [0.001, 0.0001], 'kernel': ['rbf']},
     ]

最好的实例 [Nested versus non-nested cross-validation][]for an example of Grid Search within a cross validation loop on the irisdataset

## 随机参数优化RandomizedSearchCV ##

Random Search：先用Gird Search的方法，得到所有候选参数，然后每次从中随机选择进行训练。

sklearn.model\_selection.[RandomizedSearchCV][](estimator, param\_distributions, n\_iter=10, scoring=None, fit\_params=None, n\_jobs=1, iid=True, refit=True, cv=None, verbose=0, pre\_dispatch='2\*n\_jobs', random\_state=None, error\_score='raise', return\_train\_score=True)

two main benefits over an exhaustive search:

*  A budget can be chosen independent of the number of parameters and possible values.
 *  Adding parameters that do not influence the performance does not decrease efficiency.

{
       'C': scipy.stats.expon(scale=100), 'gamma': scipy.stats.expon(scale=.1),
      'kernel': ['rbf'], 'class_weight':['balanced', None]}

In principle, any function can be passed that provides a `rvs` (randomvariate sample) method to sample a value.

实例[Comparing randomized search and grid search for hyperparameter estimation][] compares the usage and efficiencyof randomized search and grid search.

# 参数搜索的tips #

### Specifying an objective metric ###

\[[机器学习模型的评价指标和方法 ][Link 2]\]

### Composite estimators and parameter spaces ###

estimator类必须有的方法是有：get\_params, set\_params(\*\*params), fit(x,y), predict(new\_samples), score(x, y\_true)。其中有的可以直接从from sklearn.base import BaseEstimator中继承。

使用pipline方法 [Pipeline: chaining estimators][Pipeline_ chaining estimators]

### Model selection: development and evaluation ###

使用验证集（也就是开发集吧）来进行模型选择，输入到grid\_search中。development set (tobe fed to the GridSearchCV instance)

### Parallelism ###

[`GridSearchCV`][GridSearchCV] and [`RandomizedSearchCV`][RandomizedSearchCV] `都是并行运行的，`by using the keyword `n_jobs=-1`.

### Robustness to failure ###

参数输入后模型出错会导致整个grid serach失败，但是可以通过Setting error\_score=0(or =np.NaN)来解决。失败的issuing awarning and setting the score for that fold to 0 (or NaN)。

### 调参参数的选择 ###

\[[机器学习模型选择：调参参数选择 ][Link 3]\]

### GridSearch流程图 ###

![Center][]

[某小皮][Link 1]

# Alternatives to brute force parameter search #

### 特定模型的交叉验证 ###

...

### 信息准则 ###

###  ###

Some models can offer an information-theoretic closed-form formula of theoptimal estimate of the regularization parameter by computing a singleregularization path (instead of several when using cross-validation).

Here is the list of models benefitting from the Aikike InformationCriterion (AIC) or the Bayesian Information Criterion (BIC) for automatedmodel selection:

<table>     
 <tbody> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLarsIC.html#sklearn.linear_model.LassoLarsIC" title="sklearn.linear_model.LassoLarsIC" rel="nofollow"><code><span>linear_model.LassoLarsIC</span></code></a>([criterion,&nbsp;...])</td> 
   <td>Lasso model fit with Lars using BIC or AIC for model selection</td> 
  </tr> 
 </tbody> 
</table>

可以参考prml。

### Out of Bag Estimates ###

集成方法因为有数据抽样，多余的可以直接用于模型选择，而不需要额外独立的验证集。This left out portion can be used to estimate the generalization errorwithout having to rely on a separate validation set. This estimatecomes “for free” as no additional data is needed and can be used formodel selection.

<table> 
 <tbody> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier" title="sklearn.ensemble.RandomForestClassifier" rel="nofollow"><code><span>ensemble.RandomForestClassifier</span></code></a>([...])</td> 
   <td>A random forest classifier.</td> 
  </tr> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html#sklearn.ensemble.RandomForestRegressor" title="sklearn.ensemble.RandomForestRegressor" rel="nofollow"><code><span>ensemble.RandomForestRegressor</span></code></a>([...])</td> 
   <td>A random forest regressor.</td> 
  </tr> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier" title="sklearn.ensemble.ExtraTreesClassifier" rel="nofollow"><code><span>ensemble.ExtraTreesClassifier</span></code></a>([...])</td> 
   <td>An extra-trees classifier.</td> 
  </tr> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesRegressor.html#sklearn.ensemble.ExtraTreesRegressor" title="sklearn.ensemble.ExtraTreesRegressor" rel="nofollow"><code><span>ensemble.ExtraTreesRegressor</span></code></a>([n_estimators,&nbsp;...])</td> 
   <td>An extra-trees regressor.</td> 
  </tr> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn.ensemble.GradientBoostingClassifier" title="sklearn.ensemble.GradientBoostingClassifier" rel="nofollow"><code><span>ensemble.GradientBoostingClassifier</span></code></a>([loss,&nbsp;...])</td> 
   <td>Gradient Boosting for classification.</td> 
  </tr> 
  <tr> 
   <td><a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor" title="sklearn.ensemble.GradientBoostingRegressor" rel="nofollow"><code><span>ensemble.GradientBoostingRegressor</span></code></a>([loss,&nbsp;...])</td> 
   <td>Gradient Boosting for regression.</td> 
  </tr> 
 </tbody> 
</table>

### 贝叶斯优化Bayesian Optimization ###

考虑到了不同参数对应的实验结果值，因此更节省时间。和网络搜索相比简直就是老牛和跑车的区别。具体原理可以参考这个论文： [Practical Bayesian Optimization of Machine Learning Algorithms][] ，这里同时推荐两个实现了贝叶斯调参的Python库，可以上手即用：

*  [jaberg/hyperopt][jaberg_hyperopt], 比较简单。
 *  [fmfn/BayesianOptimization][fmfn_BayesianOptimization]， 比较复杂，支持并行调参。

## GrideSearch示例 ##

\[[Auto-scaling scikit-learn with Apache Spark][]\]

from: [http://blog.csdn.net/pipisorry/article/details/52268947][blog.csdn.net_pipisorry_article_details_52268947]

ref: \[[3.2. Tuning the hyper-parameters of an estimator][]\]\*

\[[python并行调参——scikit-learn grid\_search][python_scikit-learn grid_search]\]\*

\[[Parameter estimation using grid search with cross-validation][]\*\]

参数资料  
[Practical recommendations for gradient-based training of deep architectures by Yoshua Bengio (2012)][Practical recommendations for gradient-based training of deep architectures by Yoshua Bengio _2012]  
[Efficient BackProp, by Yann LeCun, Léon Bottou, Genevieve Orr and Klaus-Robert Müller][Efficient BackProp_ by Yann LeCun_ L_on Bottou_ Genevieve Orr and Klaus-Robert M_ller]  
[Neural Networks: Tricks of the Trade, edited by Grégoire Montavon, Geneviève Orr, and Klaus-Robert Müller.][Neural Networks_ Tricks of the Trade_ edited by Gr_goire Montavon_ Genevi_ve Orr_ and Klaus-Robert M_ller.]

[blog.csdn.net_pipisorry_article_details_52268947]: http://blog.csdn.net/pipisorry/article/details/52268947
[Link 1]: http://my.csdn.net/
[score function]: http://scikit-learn.org/stable/modules/grid_search.html#gridsearch-scoring
[GridSearchCV]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV
[RandomizedSearchCV]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html#sklearn.model_selection.RandomizedSearchCV
[Nested versus non-nested cross-validation]: http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html#sphx-glr-auto-examples-model-selection-plot-nested-cross-validation-iris-py
[Comparing randomized search and grid search for hyperparameter estimation]: http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html#sphx-glr-auto-examples-model-selection-randomized-search-py
[Link 2]: http://blog.csdn.net/pipisorry/article/details/52574156
[Pipeline_ chaining estimators]: http://scikit-learn.org/stable/modules/pipeline.html#pipeline
[Link 3]: http://blog.csdn.net/pipisorry/article/details/52902797
[Center]: /images/20220718/27d7d33ddb5e446e8b6def932be772dc.png
[Practical Bayesian Optimization of Machine Learning Algorithms]: http://link.zhihu.com/?target=http%3A//papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf
[jaberg_hyperopt]: http://link.zhihu.com/?target=https%3A//github.com/jaberg/hyperopt
[fmfn_BayesianOptimization]: http://link.zhihu.com/?target=https%3A//github.com/fmfn/BayesianOptimization
[Auto-scaling scikit-learn with Apache Spark]: https://databricks.com/blog/2016/02/08/auto-scaling-scikit-learn-with-apache-spark.html
[3.2. Tuning the hyper-parameters of an estimator]: http://scikit-learn.org/stable/modules/grid_search.html#grid-search
[python_scikit-learn grid_search]: http://blog.csdn.net/abcjennifer/article/details/23884761
[Parameter estimation using grid search with cross-validation]: http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html#example-model-selection-grid-search-digits-py
[Practical recommendations for gradient-based training of deep architectures by Yoshua Bengio _2012]: http://link.zhihu.com/?target=https%3A//arxiv.org/abs/1206.5533
[Efficient BackProp_ by Yann LeCun_ L_on Bottou_ Genevieve Orr and Klaus-Robert M_ller]: http://link.zhihu.com/?target=http%3A//yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
[Neural Networks_ Tricks of the Trade_ edited by Gr_goire Montavon_ Genevi_ve Orr_ and Klaus-Robert M_ller.]: http://link.zhihu.com/?target=http%3A//www.springer.com/computer/theoretical%2Bcomputer%2Bscience/book/978-3-642-35288-1