rStar-Math：小型LLM通過(guò)自我進(jìn)化的深度思維掌握數(shù)學(xué)推理

rStar-Math:SmallLLMsCanMasterMathReasoningwithSelf-EvolvedDeepThinking

XinyuGuan*LiLynaZhang*

YifeiLiu

NingShangYouranSunYiZhuFanYangMaoYang

arXiv:2501.04519v1[cs.CL]8Jan2025

MicrosoftResearchAsia

Abstract

WepresentrStar-Mathtodemonstratethatsmalllanguagemodels(SLMs)canrivalorevensurpassthemathreasoningcapabilityofOpenAIo1,withoutdistillationfromsuperiormodels.rStar-Mathachievesthisbyexercising“deepthinking”throughMonteCarloTreeSearch(MCTS),whereamathpolicySLMperformstest-timesearchguidedbyanSLM-basedprocessrewardmodel.rStar-MathintroducesthreeinnovationstotacklethechallengesintrainingthetwoSLMs:(1)anovelcode-augmentedCoTdatasythesismethod,whichperformsextensiveMCTSrolloutstogeneratestep-by-stepverifiedreasoningtrajectoriesusedtotrainthepolicySLM;(2)anovelprocessrewardmodeltrainingmethodthatavoidsna?vestep-levelscoreannotation,yieldingamoreeffectiveprocesspreferencemodel(PPM);(3)aself-evolutionrecipeinwhichthepolicySLMandPPMarebuiltfromscratchanditerativelyevolvedtoimprovereasoningcapabilities.Through4roundsofself-evolutionwithmillionsofsynthesizedsolutionsfor747kmathproblems,rStar-MathboostsSLMs’mathreasoningtostate-of-the-artlevels.OntheMATHbenchmark,itimprovesQwen2.5-Math-7Bfrom58.8%to90.0%andPhi3-mini-3.8Bfrom41.4%to86.4%,surpassingo1-previewby+4.5%and+0.9%.OntheUSAMathOlympiad(AIME),rStar-Mathsolvesanaverageof53.3%(8/15)ofproblems,rankingamongthetop20%thebrightesthighschoolmathstudents.Codeanddatawillbeavailableat

/microsoft/rStar

.

Task

(pass@1Acc)

rStar-Math(Qwen-7B)

rStar-Math(Qwen-1.5B)

rStar-Math(Phi3-mini)

OpenAIo1-preview

OpenAIo1-mini

QWQ

32B-preview

GPT-4oDeepSeek-V3

MATH

AIME2024

OlympiadBench

CollegeMath

Omni-Math

90.053.365.660.550.5

88.646.764.659.348.5

86.443.360.359.146.0

85.544.6

-

-

52.5

90.056.765.357.860.5

90.650.061.255.849.6

76.69.343.348.530.5

90.239.255.458.935.9

Table1:rStar-MathenablesfrontiermathreasoninginSLMsviadeepthinkingover64trajectories.

1Introduction

Recentstudieshavedemonstratedthatlargelanguagemodels(LLMs)arecapableoftacklingmathematicalproblems[Team,2024a,Yangetal.,2024,OpenAI,2024,Liuetal.,2024].However,theconventionalapproachofhavingLLMsgeneratecompletesolutionsinasingleinference–akintoSystem1thinking[Daniel,2011]–oftenyieldsfastbuterror-proneresults[Valmeekametal.,2023,OpenAI,2023].Inresponse,test-timecomputescaling[Snelletal.,2024,Qietal.,2024]suggestsaparadigmshifttowardaSystem2-stylethinking,whichemulateshumanreasoningthroughasloweranddeeperthoughtprocess.Inthisparadigm,anLLMservesasapolicymodeltogeneratemultiplemathreasoningsteps,whicharethenevaluatedbyanotherLLMactingasarewardmodel[OpenAI,2024].Thestepsandsolutionsdeemedmorelikelytobecorrectareselected.Theprocessrepeatsiterativelyandultimatelyderivesthefinalanswer.

*Equalcontribution.

Projectleader;correspondencetolzhani@

§XinyuGuanandYouranSundidthisworkduringtheinternshipatMSRA.XinyuGuan(2001gxy@)iswithPekingUniversity,YouranSuniswithTsinghuaUniversity.

2

0.7

ApplyVerifiers(PPM/python)

-0.5

-1

0.5

1

0.6

Onestep

Answerstep(correct)

Answerstep(wrong)

1

0.8

MCTS-drivendeepthinking

question

SLM.PPM

-0.7

-1

(a)step-by-stepverifiedreasoningtrajectory

..

filtering/

Q-valueo…O

Step1Step2

finalstep

..

..

..

..

fullsolutions

(b)Constructionofper-steppreferencepairsbasedonQ-values

SLM-r3

SLM-r1

Round1

SLM-r2

PPM-r2

SLM-r4

PPM-r4

Round2

Round3

Round4

PPM-r3

(c)4roundsofself-evolution

PPM-augmentedMCTS

PPM-augmentedMCTS

Terminal-guidedMCTS

Terminal-guidedMCTS

Figure1:TheoverviewofrStar-Math.

Inthetest-timecomputeparadigm,thekeyistotrainapowerfulpolicymodelthatgeneratespromisingsolutionstepsandareliablerewardmodelthataccuratelyevaluatesthem,bothofwhichdependonhigh-qualitytrainingdata.Unfortunately,itiswell-knownthatoff-the-shelfhigh-qualitymathreasoningdataisscarce,andsynthesizinghigh-qualitymathdatafacesfundamentalchallenges.Forthepolicymodel,itischallengingtodistinguisherroneousreasoningstepsfromthecorrectones,complicatingtheeliminationoflow-qualitydata.Itisworthnotingthatinmathreasoning,acorrectfinalanswerdoesnotensurethecorrectnessoftheentirereasoningtrace[Lanhametal.,2023].Incorrectintermediatestepssignificantlydecreasedataquality.Asfortherewardmodel,processrewardmodeling(PRM)showsagreatpotentialbyprovidingfine-grainedfeedbackonintermediatesteps[Lightmanetal.,2023].However,thetrainingdataisevenscarcerinthisregard:accuratestep-by-stepfeedbackrequiresintensehumanlabelingeffortsandisimpracticaltoscale,whilethoseautomaticannotationattemptsshowlimitedgainsduetonoisyrewardscores[Luoetal.,2024,Wangetal.,2024c,Chenetal.,2024].Duetotheabovechallenges,existingdistill-baseddatasynthesisapproachestotrainingpolicymodels,e.g.,scalingupGPT4-distilledCoTdata[Tangetal.,2024,Huangetal.,2024],haveshowndiminishingreturnsandcannotexceedthecapabilityoftheirteachermodel;meanwhile,asoftoday,trainingreliablePRMsformathreasoningremainsanopenquestion.

Inthiswork,weintroducerStar-Math,aself-evolvableSystem2-stylereasoningapproachthatachievesthestate-of-the-artmathreasoning,rivalingandsometimesevensurpassingOpenAIo1onchallengingmathcompetitionbenchmarkswithamodelsizeassmallas7billion.UnlikesolutionsrelyingonsuperiorLLMsfordatasynthesis,rStar-Mathleveragessmallerlanguagemodels(SLMs)withMonteCarloTreeSearch(MCTS)toestablishaself-evolutionaryprocess,iterativelygeneratinghigher-qualitytrainingdata.Toachieveself-evolution,rStar-Mathintroducesthreekeyinnovations.First,anovelcode-augmentedCoTdatasynthesismethod,whichperformsextensiveMCTSrolloutstogeneratestep-by-stepverifiedreasoningtrajectorieswithself-annotatedMCTSQ-values.Specifically,mathproblem-solvingisdecomposedintomulti-stepgenerationwithinMCTS.Ateachstep,theSLMservingasthepolicymodelsamplescandidatenodes,eachgeneratingaone-stepCoTandthecorrespondingPythoncode.Toverifythegenerationquality,onlynodeswithsuccessfulPythoncodeexecutionareretained,thusmitigatingerrorsinintermediatesteps.Moreover,extensiveMCTSrolloutsautomaticallyassignaQ-valuetoeachintermediatestepbasedonitscontribution:stepscontributingtomoretrajectoriesthatleadtothecorrectansweraregivenhigherQ-valuesandconsideredhigherquality.ThisensuresthatthereasoningtrajectoriesgeneratedbySLMsconsistofcorrect,high-qualityintermediatesteps.

Second,anovelmethodthattrainsanSLMactingasaprocesspreferencemodel,i.e.,aPPMtoimplementthedesiredPRM,thatreliablypredictsarewardlabelforeachmathreasoningstep.ThePPMleveragesthefactthat,althoughQ-valuesarestillnotpreciseenoughtoscoreeachreasoningstepdespiteusingextensiveMCTSrollouts,theQ-valuescanreliablydistinguishpositive(correct)stepsfromnegative(irrelevant/incorrect)ones.ThusthetrainingmethodconstructspreferencepairsforeachstepbasedonQ-valuesandusesapairwiserankingloss[Ouyangetal.,2022]tooptimizePPM’sscorepredictionforeachreasoningstep,achievingreliablelabeling.ThisapproachavoidsconventionalmethodsthatdirectlyuseQ-valuesasrewardlabels[Luoetal.,2024,Chenetal.,2024],whichareinherentlynoisyandimpreciseinstepwiserewardassignment.

Finally,afour-roundself-evolutionrecipethatprogressivelybuildsbothafrontierpolicymodelandPPMfromscratch.Webeginbycuratingadatasetof747kmathwordproblemsfrompubliclyavailablesources.Ineachround,weusethelatestpolicymodelandPPMtoperformMCTS,

3

generatingincreasinglyhigh-qualitytrainingdatausingtheabovetwomethodstotrainastrongerpolicymodelandPPMfornextround.Eachroundachievesprogressiverefinement:(1)astrongerpolicySLM,(2)amorereliablePPM,(3)generatingbetterreasoningtrajectoriesviaPPM-augmentedMCTS,and(4)improvingtrainingdatacoveragetotacklemorechallengingandevencompetition-levelmathproblems.

ExtensiveexperimentsacrossfourSLMs(1.5B-7B)andsevenmathreasoningtasksdemonstratetheeffectivenessofrStar-Math.Remarkably,rStar-MathimprovesallfourSLMs,matchingorevensurpassingOpenAIo1onchallengingmathbenchmarks.OnMATHbenchmark,with8searchtrajectories,rStar-MathboostsQwen2.5-Math-7Bfrom58.8%to89.4%andQwen2.5-Math-1.5Bfrom51.2%to87.8%.With64trajectories,thescoresriseto90%and88.4%,outperformingo1-previewby4.5%and2.6%andmatchingo1-mini’s90%.OntheOlympiad-levelAIME2024,rStar-Mathsolvesonaverage53.3%(8/15)oftheproblems,exceedingo1-previewby8.7%andallotheropen-sourcedLLMs.Wefurtherconductcomprehensiveexperimentstoverifythesuperiorityofstep-by-stepverifiedreasoningtrajectoriesoverstate-of-the-artdatasynthesisbaselines,aswellasthePPM’seffectivenesscomparedtooutcomerewardmodelsandQvalue-basedPRMs.Finally,wepresentkeyfindingsfromrStar-Mathdeepthinking,includingtheintrinsicself-reflectioncapabilityandPPM’spreferencefortheorem-applicationsintermediatesteps.

2RelatedWorks

MathDataSynthesis.AdvancementsinLLMmathreasoninghavelargelyreliedoncuratinghigh-qualityCoTdata,withmostleadingapproachesbeingGPT-distilled,usingfrontiermodelslikeGPT-4forsynthesis[Wangetal.,2024b,Gouetal.,2023,Luoetal.,2023].NotableworksincludeNuminaMath[JiaLIandPolu,2024a]andMetaMath[Yuetal.,2023b].Whileeffective,thislimitsreasoningtothecapabilitiesoftheteacherLLM.HardproblemsthattheteacherLLMcannotsolveareexcludedinthetrainingset.Evensolvableproblemsmaycontainerror-proneintermediatesteps,whicharehardtodetect.Althoughrejectionsamplingmethods[Yuanetal.,2023,Brownetal.,2024]canimprovedataquality,theydonotguaranteecorrectintermediatesteps.Asaresult,scalingupCoTdatahasdiminishingreturns,withgainsnearingsaturation—e.g.,OpenMathInstruct-2[Toshniwaletal.,2024]onlyseesa3.9%boostonMATHdespitean8×increaseindatasetsize.

ScalingTest-timeComputehasintroducednewscalinglaws,allowingLLMstoimproveperfor-manceacrossbygeneratingmultiplesamplesandusingrewardmodelsforbest-solutionselection[Snelletal.,2024,Wuetal.,2024,Brownetal.,2024].Varioustest-timesearchmethodshavebeenproposed[Kangetal.,2024,Wangetal.,2024a],includingrandomsampling[Wangetal.,2023]andtree-searchmethods[Yaoetal.,2024,Haoetal.,2023,Zhangetal.,2024b,Qietal.,2024]likeMCTS.However,open-sourcemethodsforscalingtest-timecomputationhaveshownlimitedgainsinmathreasoning,oftenduetopolicyLLMorrewardmodellimitations.rStar-MathaddressesthisbyiterativelyevolvingthepolicyLLMandrewardmodel,achievingSystem2mathematicalreasoningperformancecomparabletoOpenAIo1[OpenAI,2024].

RewardModelsarecrucialforeffectiveSystem2reasoningbutarechallengingtoobtain.RecentworksincludeLLM-as-a-Judgeforverification[Zhengetal.,2023,Qietal.,2024]andspecializedrewardmodelslikeOutcomeRewardModel[Yangetal.,2024,Yuetal.,2023a]andProcessRewardModel(PRM)[Lightmanetal.,2024].WhilePRMsofferpromisingdense,step-levelrewardsignalsforcomplexreasoning[Luoetal.,2024,Wangetal.,2024c],collectingstep-levelannotationsremainsanobstacle.WhileKangetal.[2024],Wangetal.[2024a]relyoncostlyhuman-annotateddatasetslikePRM800k[Lightmanetal.,2024],recentapproaches[Wangetal.,2024c,Luoetal.,2024]exploreautomatedannotationviaMonteCarloSamplingorMCTS.However,theystruggletogeneratepreciserewardscores,whichlimitsperformancegains.rStar-Mathintroducesanovelprocesspreferencereward(PPM)thateliminatestheneedforaccuratestep-levelrewardscoreannotation.

3Methodology

3.1DesignChoices

MCTSforEffectiveSystem2Reasoning.WeaimtotrainamathpolicySLMandaprocessrewardmodel(PRM),andintegratingbothwithinMonteCarloTreeSearch(MCTS)forSystem2deepthinking.MCTSischosenfortwokeyreasons.First,itbreaksdowncomplexmathproblemsintosimplersingle-stepgenerationtasks,reducingthedifficultyforthepolicySLMcomparedtoother

4

System2methodslikeBest-of-N[Brownetal.,2024]orself-consistency[Wangetal.,2023],whichrequiregeneratingfullsolutionsinoneinference.Second,thestep-by-stepgenerationinMCTSnaturallyyieldsstep-leveltrainingdataforbothmodels.StandardMCTSrolloutautomaticallyassignQ-valuetoeachstepbasedonitscontributiontothefinalcorrectanswer,obviatingtheneedforhuman-generatedstep-levelannotationsforprocessrewardmodeltraining.

Ideally,advancedLLMssuchasGPT-4couldbeintegratedwithinMCTStogeneratetrainingdata.However,thisapproachfacestwokeychallenges.First,eventhesepowerfulmodelsstruggletoconsistentlysolvedifficultproblems,suchasOlympiad-levelmathematics.Consequently,theresultingtrainingdatawouldprimarilyconsistofsimplersolvableproblems,limitingitsdiversityandquality.Second,annotatingper-stepQ-valuesdemandsextensiveMCTSrollouts;insufficienttreeexplorationcanleadtospuriousQ-valueassignments,suchasoverestimatingsuboptimalsteps.Giventhateachrolloutinvolvesmultiplesingle-stepgenerationsandthesemodelsarecomputationallyexpensive,increasingrolloutssignificantlyraisesinferencecosts.

Overview.Tothisend,weexploreusingtwo7BSLMs(apolicySLMandaPRM)togeneratehigher-qualitytrainingdata,withtheirsmallersizeallowingforextensiveMCTSrolloutsonaccessiblehardware(e.g.,4×40GBA100GPUs).However,self-generatingdatapresentsgreaterchallengesforSLMs,duetotheirweakercapabilities.SLMsfrequentlyfailtogeneratecorrectsolutions,andevenwhenthefinalansweriscorrect,theintermediatestepsareoftenflawedorofpoorquality.Moreover,SLMssolvefewerchallengingproblemscomparedtoadvancedmodelslikeGPT-4.

Thissectionintroducesourmethodology,asillustratedinFig.1.Tomitigateerrorsandlow-qualityintermediatesteps,weintroduceacode-augmentedCoTsyntheticmethod,whichperformsextensiveMCTSrolloutstogeneratestep-by-stepverifiedreasoningtrajectories,annotatedwithQ-values.TofurtherimproveSLMperformanceonchallengingproblems,weintroduceafour-roundself-evolutionrecipe.Ineachround,boththepolicySLMandtherewardmodelareupdatedtostrongerversions,progressivelytacklingmoredifficultproblemsandgeneratinghigher-qualitytrainingdata.Finally,wepresentanovelprocessrewardmodeltrainingapproachthateliminatestheneedforpreciseper-steprewardannotations,yieldingthemoreeffectiveprocesspreferencemodel(PPM).

3.2Step-by-StepVerifiedReasoningTrajectory

Westartbyintroducingourmethodforgeneratingstep-by-stepverifiedreasoningtrajectorieswithper-stepQ-valueannotations.GivenaproblemxandapolicymodelM,werunthestandardMCTStoincrementallyconstructasearchtreeforstep-by-stepsolutionexploration.AsshowninFig.1(a),therootnoderepresentsquestionx,whilechildnodescorrespondtointermediatestepssgeneratedbyM.Aroot-to-leafpathendingatterminalnodesdformsatrajectoryt=x⊕s1⊕s2⊕...⊕sd,witheachstepsiassignedaQ-valueQ(si).FromthesearchtreeT,weextractsolutiontrajectoriesT={t1,t2,...,tn}(n≥1).Ourgoalistoselecthigh-qualitytrajectoriesfromTtoconstructthetrainingset.Forthispurpose,weintroducecode-augmentedCoTsynthesismethodtofilteroutlow-qualitygenerationsandperformextensiverolloutstoimprovethereliabilityofQ-valueaccuracy.

Code-augmentedCoTGeneration.PriorMCTSapproachesprimarilygeneratenaturallanguage(NL)CoTs[Qietal.,2024,Zhangetal.,2024a].However,LLMsoftensufferfromhallucination,producingincorrectorirrelevantstepsyetstillarriveatthecorrectanswerbychance[Lanhametal.,2023].Theseflawedstepsarechallengingtodetectandeliminate.Toaddressthis,weproposeanovelcodeexecutionaugmentedCoT.AsshowninFig.2,thepolicymodelgeneratesaone-stepNLCoTalongsideitscorrespondingPythoncode,wheretheNLCoTisembeddedasaPythoncomment.OnlygenerationswithsuccessfullyexecutedPythoncodeareretainedasvalidcandidates.

Question:Billwalks$\frac{1}{2}$milesouth,then$\frac{3}{4}$mileeast,andfinally$\frac{1}{2}$milesouth.Howmanymilesishe,inadirectline,fromhisstartingpoint?Expressyouranswerasadecimaltothenearesthundredth.

#Step1:Calculatethetotaldistancewalkedsouth

total_south=1/2+1/2

#Step2:Calculatethetotaldistancewalkedeast

total_east=3/4

#Step3:UsethePythagoreantheoremtofindthedirectdistancefromthestartingpoint

importmath

direct_distance=math.sqrt(total_south**2+total_east**2)

#Step4:Roundthedirectdistancetothenearesthundredth

direct_distance_rounded=round(direct_distance,2

Fromtheresult,wecanseethatthedirectdistancefromthestartingpointis\boxed{1.25}miles

NLCoTasPythonComment

Pythoncodeexecutionforstep1:

#Step1:Calculatethetotaldistancewalkedsouth

total_south=1/2+1/2

#Step1:Calculatethetotaldistancewalkedsouth

total_south=1/2+1/2

#Step2:Calculatethetotaldistancewalkedeast

total_east=3/4

Pythoncodeexecutionforstep2:

…

Figure2:AnexampleofCode-augmentedCoT.

5

Specifically,startingfromtheinitialrootnodex,weperformmultipleMCTSiterationsthroughselection,expansion,rollout,andback-propagation.Atstepi,wecollectthelatestreasoningtrajectoryx⊕s1⊕s2⊕...⊕si?1asthecurrentstate.Basedonthisstate,weprompt(seeAppendixA.3)thepolicymodeltogeneratencandidatessi,0,...,si,n?1forstepi.Pythoncodeexecutionisthenemployedtofiltervalidnodes.AsshowninFig.2,eachgenerationsi,jisconcatenatedwiththecodefromallprevioussteps,formings1⊕s2⊕...⊕si?1⊕si,j.CandidatesthatexecutesuccessfullyareretainedasvalidnodesandscoredbythePPM,whichassignsaQ-valueq(si).Then,weusethewell-knownUpperConfidenceboundsforTrees(UCT)[KocsisandSzepesvári,2006]toselectthebestnodeamongthencandidates.Thisselectionprocessismathematicallyrepresentedas:

where(1)

whereN(s)denotesthenumberofvisitstonodes,andNparent(s)isthevisitcountofs’sparentnode.Thepredictedrewardq(s)isprovidedbythePPMandwillbeupdatedthroughback-propagation.cisaconstantthatbalancesexploitationandexploration.

ExtensiveRolloutsforQ-valueAnnotation.AccurateQ-valueQ(s)annotationinEq.1iscrucialforguidingMCTSnodeselectiontowardscorrectproblem-solvingpathsandidentifyinghigh-qualitystepswithintrajectories.ToimproveQ-valuereliability,wedrawinspirationfromGoplayers,whoretrospectivelyevaluatetherewardofeachmovebasedongameoutcomes.Althoughinitialestimatesmaybeimprecise,repeatedgameplayrefinestheseevaluationsovertime.Similarly,ineachrollout,weupdatetheQ-valueofeachstepbasedonitscontributiontoachievingthecorrectfinalanswer.AfterextensiveMCTSrollouts,stepsconsistentlyleadingtocorrectanswersachievehigherQ-values,occasionalsuccessesyieldmoderateQ-values,andconsistentlyincorrectstepsreceivelowQ-values.Specifically,weintroducetwoself-annotationmethodstoobtainthesestep-levelQ-values.Fig.1(c)showsthedetailedsettinginthefourroundsofself-evolution.

Terminal-guidedannotation.Duringthefirsttworounds,whenthePPMisunavailableorinsufficientlyaccurate,weuseterminal-guidedannotation.Formally,letq(si)kdenotetheqvalueforstepsiafterback-propagationinthekthrollout.FollowingAlphaGo[Silveretal.,2017]andrStar[Qietal.,2024],wescoreeachintermediatenodebasedonitscontributiontothefinalcorrectanswer:

q(si)k=q(si)k?1+q(sd)k;(2)wheretheinitialqvalueq(si)0=0inthefirstrollout.Ifthisstepfrequentlyleadstoacorrectanswer,itsqvaluewillincrease;otherwise,itdecreases.Terminalnodesarescoredasq(sd)=1forcorrectanswersandq(sd)=?1otherwise,asshowninFig.1.

PRM-augmentedannotation.Startingfromthethirdround,weusePPMtoscoreeachstepformoreeffectivegeneration.Comparedtoterminal-guidedannotation,whichrequiresmultiplerolloutsforameaningfulqvalue,PPMdirectlypredictsanon-zeroinitialqvalue.PPM-augmentedMCTSalsohelpsthepolicymodeltogeneratehigher-qualitysteps,guidingsolutionstowardscorrectpaths.Formally,forstepsi,PPMpredictsaninitialq(si)0valuebasedonthepartialtrajectory:

q(si)0=PPM(x⊕s1⊕s2⊕...⊕si?1⊕si)(3)Thisqvaluewillbeupdatedbasedonterminalnode’sq(sd)valuethroughMCTSback-propagationinEq.2.Forterminalnodesd,wedonotusePRMforscoringduringtrainingdatageneration.Instead,weassignamoreaccuratescorebasedongroundtruthlabelsasterminal-guidedrewarding.

3.3ProcessPreferenceModel

Processrewardmodels,whichprovidegranularstep-levelrewardsignals,ishighlydesirableforsolvingchallengingmathproblems.However,obtaininghigh-qualitystep-leveltrainingdataremainsanopenchallenge.Existingmethodsrelyonhumanannotations[Lightmanetal.,2023]orMCTS-generatedscores[Zhangetal.,2024a,Chenetal.,2024]toassignascoreforeachstep.Thesescoresthenserveastrainingtargets,withmethodssuchasMSEloss[Chenetal.,2024]orpointwiseloss[Wangetal.,2024c,Luoetal.,2024,Zhangetal.,2024a]usedtominimizethedifferencebetweenpredictedandlabeledscores.Asaresult,theprecisionoftheseannotatedstep-levelrewardscoresdirectlydeterminestheeffectivenessoftheresultingprocessrewardmodel.

Unfortunately,preciseper-stepscoringremainsaunsolvedchallenge.AlthoughourextensiveMCTSrolloutsimprovethereliabilityofQ-values,preciselyevaluatingfine-grainedstepqualitypresentsa

6

majorobstacle.Forinstance,amongasetofcorrectsteps,itisdifficulttorankthemasbest,second-best,oraverageandthenassignprecisescores.Similarly,amongincorrectsteps,differentiatingtheworstfrommoderatelypoorstepsposesanalogouschallenges.Evenexperthumanannotationstruggleswithconsistency,particularlyatscale,leadingtoinherentnoiseintraininglabels.

Weintroduceanoveltrainingmethodthattrainsaprocesspreferencemodel(PPM)byconstructingstep-levelpositive-negativepreferencepairs.AsshowninFig.1(b),insteadofusingQ-valuesasdirectrewardlabels,weusethemtoselectstepsfromMCTStreeforpreferencepairconstruction.Foreachstep,weselecttwocandidateswiththehighestQ-valuesaspositivestepsandtwowiththelowestasnegativesteps.Critically,theselectedpositivestepsmustleadtoacorrectfinalanswer,whilenegativestepsmustleadtoincorrectanswers.Forintermediatesteps(exceptthefinalanswerstep),thepositiveandnegativepairssharethesameprecedingsteps.Forthefinalanswerstep,whereidenticalreasoningtrajectoriesrarelyyielddifferentfinalanswers,werelaxthisrestriction.WeselecttwocorrecttrajectorieswiththehighestaverageQ-valuesaspositiveexamplesandtwoincorrecttrajectorieswiththelowestaverageQ-valuesasnegativeexamples.Following[Ouyangetal.,2022],wedefineourlossfunctionusingthestandardBradley-Terrymodelwithapairwiserankingloss:

Lppm(θ)=?E(x,y,y∈D)[log(σ(rθ(x,y)?rθ(x,y)))](4)

wheniisnotfinalanswerstep,y=s1⊕...⊕si?1⊕sos;y=s1⊕...⊕si?1⊕seg(5)

Here,rθ(x,yi)denotestheoutputofthePPM,wherexistheproblemandyisthetrajectoryfromthefirststeptotheithstep.

3.4Self-EvolvedDeepThinking

3.4.1TrainingwithStep-by-StepVerifiedReasoningTrajectory

MathProblemsCollection.Wecollectalargedatasetof747kmathwordproblemswithfinalanswerground-truthlabels,primarilyfromNuminaMath[JiaLIandPolu,2024a]andMetaMath[Yuetal.,2023b].Notably,onlycompetition-levelproblems(e.g.,OlympiadsandAIME/AMC)fromNuminaMathareincluded,asweobservethatgrade-school-levelproblemsdonotsignificantlyimproveLLMcomplexmathreasoning.Toaugmentthelimitedcompetition-levelproblems,wefollow[Lietal.,2024]anduseGPT-4tosynthesizenewproblemsbasedontheseedproblemsin7.5kMATHtrainsetand3.6kAMC-AIMEtrainingsplit.However,GPT-4oftengeneratedunsolvableproblemsorincorrectsolutionsforchallengingseedproblems.Tofilterthese,wepromptGPT-4togenerate10solutionsperproblem,retainingonlythosewithatleast3consistentsolutions.

ReasoningTrajectoriesCollection.Insteadofusingtheoriginalsolutionsinthe747kmathdataset,weconductextensiveMCTSrollouts(Sec.3.2)togeneratehigher-qualitystep-by-stepverifiedreasoningtrajectories.Ineachself-evolutionround,weperform16rolloutspermathproblem,whichleadsto16reasoningtrajectories.Problemsarethencategoriesbydifficultybasedonthecorrectratioofthegeneratedtrajectories:easy(allsolutionsarecorrect),medium(amixofcorrectandincorrectsolutions)andhard(allsolutionsareincorrect).Forhardproblemswithnocorrecttrajectories,anadditionalMCTSwith16rolloutsisperformed.Afterthat,allstep-by-steptrajectoriesandtheirannotatedQ-valuesarecollectedandfilteredtotrainthepolicySLMandprocesspreferencemodel.

SupervisedFine-tuningthePolicySLM.