三種特征選擇方法及Spark MLlib調(diào)用實(shí)例Scala Java python

三種特征選擇方法及SparkMLlib調(diào)用實(shí)例(Scala/Java/python)VectorSlicer算法介紹：VectorSlicer是一個轉(zhuǎn)換器輸入特征向量，輸出原始特征向量子集。VectorSlicer接收帶有特定索引的向量列，通過對這些索引的值進(jìn)行篩選得到新的向量集?？山邮苋缦聝煞N索引.整數(shù)索引，setIndices()。.字符串索引代表向量中特征的名字，此類要求向量列有AttributeGroup，因?yàn)樵摴ぞ吒鶕?jù)Attribute來匹配名字字段。指定整數(shù)或者字符串類型都是可以的。另外，同時使用整數(shù)索引和字符串名字也是可以的。不允許使用重復(fù)的特征，所以所選的索引或者名字必須是沒有獨(dú)一的。注意如果使用名字特征，當(dāng)遇到空值的時候?qū)?bào)錯。輸出將會首先按照所選的數(shù)字索引排序(按輸入順序)，其次按名字排序(按輸入順序)。示例：假設(shè)我們有一個DataFrame含有userFeatures歹ij：userFeatures[0.0,10.0,0.5]userFeatures是一個向量列包含3個用戶特征。假設(shè)userFeatures的第一列全為0,我們希望刪除它并且只選擇后兩項(xiàng)。我們可以通過索引setIndices(1,2)來選擇后兩項(xiàng)并產(chǎn)生一個新的features列：userFeatures|features|[0.0,10.0,0.5]|[10.0,0.5]假設(shè)我們還有如同["f1","f2","f3"]的屬性，那可以通過名字setNames("f2","f3")的形式來選擇：userFeatures|features|[0.0,10.0,0.5]|[10.0,0.5]["f1","f2","f3"]|["f2","f3"]調(diào)用示例：Scala：[plain]viewplaincopyimportjava.util.Arraysimportorg.apache.spark.ml.attribute.{Attribute,AttributeGroup,NumericAttribute}importorg.apache.spark.ml.feature.VectorSlicerimportorg.apache.spark.ml.linalg.Vectorsimportorg.apache.spark.sql.Rowimportorg.apache.spark.sql.types.StructTypevaldata=Arrays.asList(Row(Vectors.dense(-2.0,2.3,0.0)))valdefaultAttr=NumericAttribute.defaultAttrvalattrs=Array("f1","f2","f3").map(defaultAttr.withName)valattrGroup=newAttributeGroup("userFeatures",attrs.asInstanceOf[Array[Attribute]])valdataset=spark.createDataFrame(data,StructType(Array(attrGroup.toStructField())))valslicer=newVectorSlicer().setInputCol("userFeatures").setOutputCol("features")slicer.setIndices(Array(1)).setNames(Array("f3"))//orslicer.setIndices(Array(1,2)),orslicer.setNames(Array("f2","f3"))valoutput=slicer.transform(dataset)println(output.select("userFeatures","features").first())Java：[java]viewplaincopyimportjava.util.List;mon.collect.Lists;importorg.apache.spark.ml.attribute.Attribute;importorg.apache.spark.ml.attribute.AttributeGroup;importorg.apache.spark.ml.attribute.NumericAttribute;importorg.apache.spark.ml.feature.VectorSlicer;importorg.apache.spark.ml.linalg.Vectors;importorg.apache.spark.sql.Dataset;importorg.apache.spark.sql.Row;importorg.apache.spark.sql.RowFactory;importorg.apache.spark.sql.types.*;Attribute[]attrs=newAttribute[]{NumericAttribute.defaultAttr().withName("f1"),NumericAttribute.defaultAttr().withName("f2"),NumericAttribute.defaultAttr().withName("f3")};AttributeGroupgroup=newAttributeGroup("userFeatures",attrs);List<Row>data=Lists.newArrayList(RowFactory.create(Vectors.sparse(3,newint[]{0,1},newdouble[]{-2.0,2.3})),RowFactory.create(Vectors.dense(-2.0,2.3,0.0)));Dataset<Row>dataset=spark.createDataFrame(data,(newStructType()).add(group.toStructField()));VectorSlicervectorSlicer=newVectorSlicer().setInputCol("userFeatures").setOutputCol("features");vectorSlicer.setIndices(newint[]{1}).setNames(newString[]{"f3"});//orslicer.setIndices(newint[]{1,2}),orslicer.setNames(newString[]{"f2","f3"})Dataset<Row>output=vectorSlicer.transform(dataset);System.out.println(output.select("userFeatures","features").first());Python：[python]viewplaincopyfrompyspark.ml.featureimportVectorSlicerfrompyspark.ml.linalgimportVectorsfrompyspark.sql.typesimportRowdf=spark.createDataFrame([Row(userFeatures=Vectors.sparse(3,{0:-2.0,1:2.3}),),Row(userFeatures=Vectors.dense([-2.0,2.3,0.0]),)])slicer=VectorSlicer(inputCol="userFeatures",outputCol="features",indices=[1])output=slicer.transform(df)output.select("userFeatures","features").show()RFormula算法介紹：RFormula通過R模型公式來選擇列。支持R操作中的部分操作，包括‘~‘，’.’，‘:’,‘+’以及‘-‘，基本操作如下：~分隔目標(biāo)和對象+合并對象，“+0”意味著刪除空格:交互(數(shù)值相乘，類別二值化).除了目標(biāo)外的全部列假設(shè)a和b為兩列：y~a+b表示模型y~w0+w1*a+w2*b其中w0為截距，w1和w2為相關(guān)系數(shù)。y~a+b+a:b-1表示模型y~w1*a+w2*b+w3*a*b,其中wl,w2,w3是相關(guān)系數(shù)。RFormula產(chǎn)生一個向量特征列以及一個double或者字符串標(biāo)簽列。如果類別列是字符串類型，它將通過Stringindexer轉(zhuǎn)換為double類型。如果標(biāo)簽列不存在，則輸出中將通過規(guī)定的響應(yīng)變量創(chuàng)造一個標(biāo)簽列。示例：假設(shè)我們有一個DataFrame含有id,country,hour和clicked四列:id|country|hour|clicked---|||7|"US"|18|1.08|"CA"|12|0.09|"NZ"|15|0.0如果我們使用RFormula公式clicked~country+hour,則表明我們希望基于country和hour預(yù)測clicked，通過轉(zhuǎn)換我們可以得到如下DataFrame：id|country|hour|clicked|features|label|||||7|"US"|18|1.0|[0.0,0.0,18.0]|1.08|"CA"|12|0.0|[0.0,1.0,12.0]|0.09|"NZ"|15|0.0|[1.0,0.0,15.0]|0.0調(diào)用示例：Scala：[plain]viewplaincopyimportorg.apache.spark.ml.feature.RFormulavaldataset=spark.createDataFrame(Seq((7,"US",18,1.0),(8,"CA",12,0.0),(9,"NZ",15,0.0))).toDF("id","country","hour","clicked")valformula=newRFormula().setFormula("clicked~country+hour").setFeaturesCol("features").setLabelCol("label")valoutput=formula.fit(dataset).transform(dataset)output.select("features","label").show()Java：[java]viewplaincopyimportjava.util.Arrays;importjava.util.List;importorg.apache.spark.ml.feature.RFormula;importorg.apache.spark.sql.Dataset;importorg.apache.spark.sql.Row;importorg.apache.spark.sql.RowFactory;importorg.apache.spark.sql.types.StructField;importorg.apache.spark.sql.types.StructType;importstaticorg.apache.spark.sql.types.DataTypes.*;StructTypeschema=createStructType(newStructField[]{createStructField("id",IntegerType,false),createStructField("country",StringType,false),createStructField("hour",IntegerType,false),createStructField("clicked",DoubleType,false)});List<Row>data=Arrays.asList(RowFactory.create(7,"US",18,1.0),RowFactory.create(8,"CA",12,0.0),RowFactory.create(9,"NZ",15,0.0));Dataset<Row>dataset=spark.createDataFrame(data,schema);RFormulaformula=newRFormula().setFormula("clicked~country+hour").setFeaturesCol("features").setLabelCol("label");Dataset<Row>output=formula.fit(dataset).transform(dataset);output.select("features","label").show();Python：[python]viewplaincopyfrompyspark.ml.featureimportRFormuladataset=spark.createDataFrame([(7,"US",18,1.0),(8,"CA",12,0.0),(9,"NZ",15,0.0)],["id","country","hour","clicked"])formula=RFormula(formula="clicked~country+hour",featuresCol="features",labelCol="label")output=formula.fit(dataset).transform(dataset)output.select("features","label").show()ChiSqSelector算法介紹：ChiSqSelector代表卡方特征選擇。它適用于帶有類別特征的標(biāo)簽數(shù)據(jù)。ChiSqSelector根據(jù)類別的獨(dú)立卡方2檢驗(yàn)來對特征排序，然后選取類別標(biāo)簽主要依賴的特征。它類似于選取最有預(yù)測能力的特征。示例：假設(shè)我們有一個DataFrame含有id,features和clicked三列，其中clicked為需要預(yù)測的目標(biāo)：id|features|clicked---|||[0.0,0.0,18.0,1.0]|1.0|[0.0,1.0,12.0,0.0]|0.0|[1.0,0.0,15.0,0.1]|0.0如果我們使用ChiSqSelector并設(shè)置numTopFeatures為1,根據(jù)標(biāo)簽clicked,features中最后一列將會是最有用特征：id|features|clicked|selectedFeatures---|||TOC\o"1-5"\h\z7|[0.0,0.0,18.0,1.0]|1.0|[1.0]8|[0.0,1.0,12.0,0.0]|0.0|[0.0]9|[1.0,0.0,15.0,0.1]|0.0|[0.1]調(diào)用示例：Scala：[plain]viewplaincopyimportorg.apache.spark.ml.feature.ChiSqSelectorimportorg.apache.spark.ml.linalg.Vectorsvaldata=Seq((7,Vectors.dense(0.0,0.0,18.0,1.0),1.0),(8,Vectors.dense(0.0,1.0,12.0,0.0),0.0),(9,Vectors.dense(1.0,0.0,15.0,0.1),0.0))valdf=spark.createDataset(data).toDF("id","features","clicked")valselector=newChiSqSelector().setNumTopFeatures(1).setFeaturesCol("features").setLabelCol("clicked").setOutputCol("selectedFeatures")valresult=selector.fit(df).transform(df)result.show()Java：[java]viewplaincopyimportjava.util.Arrays;importjava.util.List;importorg.apache.spark.ml.feature.ChiSqSelector;importorg.apache.spark.ml.linalg.VectorUDT;importorg.apache.spark.ml.linalg.Vectors;importorg.apache.spark.sql.Row;importorg.apache.spark.sql.RowFactory;importorg.apache.spark.sql.types.DataTypes;importorg.apache.spark.sql.types.Metadata;importorg.apache.spark.sql.types.StructField;importorg.apache.spark.sql.types.StructType;List<Row>data=Arrays.asList(RowFactory.create(7,Vectors.dense(0.0,0.0,18.0,1.0),1.0),RowFactory.create(8,Vectors.dense(0.0,1.0,12.0,0.0),0.0),RowFactory.create(9,Vectors.dense(1.0,0.0,15.0,0.1),0.0));StructTypeschema=newStructType(newStructField[]{newStructField("id",DataTypes.IntegerType,false,Metadata.empty()),newStructFi