2023學(xué)年完整公開(kāi)課版RDD的takeSample編程

RDD的takeSample編程目錄引言源碼參數(shù)分析212案例3223引言1引言

takeSample返回此RDD的固定大小的采樣子集。返回Array。注意：僅當(dāng)預(yù)期結(jié)果數(shù)組較小時(shí)才應(yīng)使用此方法，因?yàn)樗袛?shù)據(jù)均已加載到驅(qū)動(dòng)程序的內(nèi)存中。源碼參數(shù)分析2源碼參數(shù)分析釋

//takeSample源碼deftakeSample(withReplacement:Boolean,num:Int,seed:Long=Utils.random.nextLong):Array[T]=withScope{valnumStDev=10.0require(num>=0,"Negativenumberofelementsrequested")require(num<=(Int.MaxValue-(numStDev*math.sqrt(Int.MaxValue)).toInt),"Cannotsupportasamplesize>Int.MaxValue-"+s"$numStDev*math.sqrt(Int.MaxValue)")源碼參數(shù)分析釋

//takeSample源碼if(num==0){newArray[T](0)}else{valinitialCount=this.count()if(initialCount==0){newArray[T](0)}else{valrand=newRandom(seed)源碼參數(shù)分析釋

if(!withReplacement&&num>=initialCount){Utils.randomizeInPlace(this.collect(),rand)}else{valfraction=SamplingUputeFractionForSampleSize(num,initialCount,withReplacement)varsamples=this.sample(withReplacement,fraction,rand.nextInt()).collect()varnumIters=0源碼參數(shù)分析釋

while(samples.length<num){logWarning(s"Neededtore-sampleduetoinsufficientsamplesize.Repeat#$numIters")samples=this.sample(withReplacement,fraction,rand.nextInt()).collect()numIters+=1}Utils.randomizeInPlace(samples,rand).take(num)}}}}源碼參數(shù)分析釋

參數(shù)分析如下：1.withReplacement：是否放回抽樣。true-有放回，false-無(wú)放回。2.num：返回樣本的大小。3.seed：隨機(jī)數(shù)生成器的種子。一般默認(rèn)。案例3案例1.無(wú)放回抽樣，樣本個(gè)數(shù)>父本個(gè)數(shù)，返回父本個(gè)數(shù)。valrdd=sc.parallelize(List(2,3,7,4,8))valsampleRdd=rdd.takeSample(false,10)sampleRdd.foreach(println)2.無(wú)放回抽樣，樣本個(gè)數(shù)<=父本個(gè)數(shù)，返回樣本個(gè)數(shù)。valrdd=sc.parallelize(List(2,3,7,4,8))valsampleRdd=rdd.takeSample(false,3)sampleRdd.foreach(println)。3.有放回抽樣，返回樣本個(gè)數(shù)。valrdd=sc.pa