翻譯以.原文和在同一文件中前

一種基于市場(chǎng)的軟件方的提交的數(shù)量）和正確的潛力（總的修復(fù)這些的）這兩個(gè)概念。我們意圖分類(lèi) 描D.2.4軟件/程序校驗(yàn)]：可靠性D.2.10設(shè)計(jì)]：方法;J.4[社會(huì)與行為科學(xué)]：經(jīng)濟(jì)學(xué); 引其中一個(gè)這樣理想化的方式祖，EdsgerDijkstra，在他1972年建議：在本文中，我們探討用市場(chǎng)理論來(lái)驅(qū)動(dòng)軟件開(kāi)發(fā)。我們的目標(biāo)不是沒(méi)有，2 眾包方不同于傳統(tǒng)的從技術(shù)進(jìn)步上保證軟件正確性的方法。許多最近的系統(tǒng)都是“眾包”過(guò)、獎(jiǎng)品、或者其他形式的）來(lái)解決問(wèn)題或獲得信息。正如我們所見(jiàn)，這個(gè)系的奠基者到：我寧愿有布朗運(yùn)動(dòng)，在那有顯微鏡直接的證明，最終推動(dòng)科學(xué)緩慢的移動(dòng)，沒(méi)有任何人對(duì)自己所看到的有預(yù)見(jiàn)性。本文同時(shí)有很高比率在這個(gè)系2.1和缺陷報(bào)鼓勵(lì)和激勵(lì)自由市場(chǎng)的測(cè)試員和用戶(hù)報(bào)告錯(cuò)誤和是很自然的。一種使用戶(hù)表達(dá)其偏好的方式是允許他們對(duì)他們所關(guān)心的或相關(guān)問(wèn)題投票。比如，Sun公司的Java相識(shí)的，AdobeFlex的和問(wèn)題管理系統(tǒng)使用一個(gè)叫做JIRA的系統(tǒng)來(lái)追蹤Flex生成器的/SDK和ActionScript編譯器項(xiàng)目的。JIRA允許一個(gè)用戶(hù)對(duì)與其相關(guān)的一系列的票數(shù)，用戶(hù)必須認(rèn)真考慮怎么樣分配他的票數(shù)。用戶(hù)能夠取消對(duì)一個(gè)他們不在感2.1．2Mzill安全獎(jiǎng)金，Mozilla提供一個(gè)500的現(xiàn)金和一MozillaT-恤給任何一個(gè)報(bào)告了有效的，關(guān)鍵的安全的人[8]。這個(gè)錯(cuò)誤必須達(dá)到一個(gè)特定的標(biāo)準(zhǔn)，包括以下：錯(cuò)誤必須是的，他必須是存在于最近支持版的由Mozilla發(fā)布的火狐瀏覽器和Thunderbird軟件中。同時(shí)發(fā)現(xiàn)者必須寫(xiě)下寫(xiě)下錯(cuò)誤的發(fā)現(xiàn)者被激勵(lì)來(lái)和Mozilla的工程師一起解決這個(gè)錯(cuò)誤。安全被當(dāng)做一種特殊的方式對(duì)待，因?yàn)楸焕玫氖窍喈?dāng)嚴(yán)重的。安全的報(bào)告會(huì)在有限的時(shí)把公之于眾。比如，如果被忽略報(bào)告者可以選擇將公之于眾。沒(méi)有公布錯(cuò)誤信息的錯(cuò)誤報(bào)告而及時(shí)被Mozilla的工程師發(fā)現(xiàn)的報(bào)告者能夠得到根據(jù)項(xiàng)目Mozilla的和補(bǔ)償。缺陷市場(chǎng)，Schechter提議使 試者獲得一部分的。的總數(shù)會(huì)降低到最低的總值R0。只有第一個(gè)報(bào)告者能得到。自從一個(gè)測(cè)試者想要提高的總值的同時(shí)他也提高了另一個(gè)測(cè)試者第一個(gè)有趣的事情是，軟件的生產(chǎn)者和質(zhì)量上的修補(bǔ)技術(shù)是替代品。如果釋放一個(gè)補(bǔ)丁是可能的，而且軟件包含更大數(shù)量的[9]。同時(shí)還顯示了軟件的生產(chǎn)者傾向于解決社會(huì)最優(yōu)的問(wèn)題。我們注意到雖然Mozilla的報(bào)告賞金系統(tǒng)和缺陷市場(chǎng)提供報(bào)告缺陷的 自由市雖然和缺陷報(bào)告系統(tǒng)被用來(lái)驗(yàn)證軟件的問(wèn)題，他們通常沒(méi)有結(jié)合這個(gè)從平臺(tái)的來(lái)尋找和定位問(wèn)題。這個(gè)鴻溝被自由市場(chǎng)填補(bǔ)了。這是一個(gè)個(gè)人的平點(diǎn)提供活躍的工作者的歷史和質(zhì)量信息，同時(shí)一些有特點(diǎn)的工具使得支付過(guò)程被監(jiān)督。IT要公司員工工資的一個(gè)小數(shù)點(diǎn)。接下來(lái)驗(yàn)證兩個(gè)這樣的公司，RentACoder和TopCoder。雖然典型的獨(dú)立的項(xiàng)目在RentACoder，個(gè)人任務(wù)通常個(gè)上放置應(yīng)用軟件的地方。TopCoder，通過(guò)程序競(jìng)賽來(lái)建立一個(gè)從全球1000強(qiáng)企業(yè)外包的客戶(hù)TopCoderTopCoder[5]。一經(jīng)驗(yàn)證就可以盡快的獲得。這減少了不確定性，對(duì)TopCoder以及程序員都是有有效的區(qū)別任務(wù)的最好的參與者。然而，TopCoder已經(jīng)不只是一個(gè)眾包軟件任務(wù)發(fā)布平 Rent-a-也是可以的，比如“按時(shí)間”在這客戶(hù)對(duì)一個(gè)程序員安時(shí)間支付獎(jiǎng)金而不是最終的結(jié)果。Rent-a-Coder7.5%15%作為費(fèi)用。一旦一個(gè)工作完成過(guò)后客戶(hù)和程序員可以互相評(píng)價(jià)對(duì)方。這點(diǎn)與TopCoder不同，那里iTunes應(yīng)用商iPoneiTunes軟件只有$0.99。這里同時(shí)有一個(gè)系統(tǒng)。流行的應(yīng)用有上千的點(diǎn)擊率，同時(shí)從用戶(hù)個(gè)程序員投入到一個(gè)項(xiàng)目或用戶(hù)為軟件的功能投票和修復(fù)的一種組織方式。3我們現(xiàn)在提出我們的基于市場(chǎng)驅(qū)動(dòng)的理論，來(lái)驅(qū)動(dòng)軟件來(lái)增加軟件的正確性和功能。我們結(jié)合了許多之前的基于市場(chǎng)的理論，合并了錯(cuò)誤報(bào)告（作為市場(chǎng)，漏這個(gè)提議現(xiàn)在還沒(méi)有完全的完成，為了得到我們心中的一些，想象一個(gè)iTunes應(yīng)用商店和Bugzilla的組合，在那里用戶(hù)可以投票來(lái)修復(fù)或者實(shí)現(xiàn)新的特點(diǎn)，同現(xiàn)存的投票機(jī)制缺少的一個(gè)元素如錯(cuò)誤的報(bào)告并不從直接的軟件修復(fù)的額外的JWReward是由用戶(hù)u在t時(shí)刻提供的工作j。如果用戶(hù)u沒(méi)有提供則Cost是工人在t時(shí)刻開(kāi)始執(zhí)行工作j。如果一個(gè)工人w不感，那么我們認(rèn)1我們假設(shè)回報(bào)是現(xiàn)金（表為$像TopCoder完全的完成一個(gè)任務(wù)。在4.4小節(jié)我們討論如何是用市場(chǎng)機(jī)制來(lái)處理一個(gè)更加真實(shí)的案例。鑒于以上的描述，一個(gè)工作的總的為當(dāng)R=0，這里沒(méi)有對(duì)軟件的額外的需求或增加的功能。請(qǐng)注意，這并不意味著用戶(hù)滿(mǎn)0，k的需求在時(shí)間t為：提及這個(gè)概念作為正確，雖然是一個(gè)值得稱(chēng)贊的目標(biāo)，對(duì)于任何東西，但正確的要求是0,他不是要求軟件中沒(méi)有錯(cuò)誤，而是表示為沒(méi)有用戶(hù)需要完成和修復(fù)漏別的是我們可以定義一個(gè)工作jt時(shí)刻的潛力值：然而，一個(gè)真正的動(dòng)態(tài)系統(tǒng)是：一個(gè)工作的可以波動(dòng)。因此，用戶(hù)可以隨時(shí)決師估計(jì)的成本函數(shù)，銷(xiāo)售信息的參與者的系統(tǒng)。個(gè)作業(yè)要由它提供了最好的工人進(jìn)行解決方案的質(zhì)量和執(zhí)行工作，而成本之間相比更簡(jiǎn)單最廉價(jià)的工作。但是我們模型不能區(qū)別出不同質(zhì)量的工作，相反，是否給予是一的決定。另外一個(gè)衡量標(biāo)準(zhǔn)是工作的完成數(shù)量。這是很重要的，比如，目前為止我們已經(jīng)討論了來(lái)自用戶(hù)的。然而，一些用戶(hù)可能希望影響更廣泛的方向，而不僅僅是特定的工作。其他的用戶(hù)可能沒(méi)有時(shí)間或者專(zhuān)業(yè)知識(shí)來(lái)提供特定的工作。提供了法來(lái)處理這兩個(gè)問(wèn)題。用戶(hù)可以提供來(lái)在所有正確能夠平均的分配在所有的工作中，或者按比例向現(xiàn)有配套”）為了充分利用現(xiàn)存的信息在當(dāng)前的。另一個(gè)問(wèn)題是分配是否應(yīng)該現(xiàn)存的作業(yè)，設(shè)計(jì)市場(chǎng)：組件與自治所有必要的行動(dòng)帶來(lái)的就業(yè)機(jī)會(huì)應(yīng)該有市場(chǎng)的力量來(lái)驅(qū)動(dòng)。在市場(chǎng)的實(shí)體之包容性大家誰(shuí)提供信息，或工作形式，導(dǎo)致改善共享該系統(tǒng)應(yīng)當(dāng)透明，尊重包括市場(chǎng)上流動(dòng)的錢(qián)，和在任務(wù)中應(yīng)該共享的?？煽啃栽撓到y(tǒng)應(yīng)當(dāng)免疫操作，抗的魯棒性，防止“淺”的工作，使得必須在之 。從銷(xiāo)售托管。當(dāng)用戶(hù)的軟件，售價(jià)的一部分被存入托管賬戶(hù)的市場(chǎng)。然后，用戶(hù)從登記托管。即使軟件是免費(fèi)的，可能仍然有可用。許多大型開(kāi)源項(xiàng)目具有很多的請(qǐng)注意，在所有這些情況下，不管業(yè)務(wù)模型，提出了市場(chǎng)與真正的金錢(qián)，可以這筆錢(qián)是由一個(gè)可信的第來(lái)處理（而不是軟件銷(xiāo)售商我們指定為“銀行”。出一個(gè)市場(chǎng)機(jī)制（類(lèi)似于TopCoder公司）其他工人提供測(cè)試，競(jìng)爭(zhēng)的開(kāi)發(fā)者可以通過(guò)結(jié)果的正確性來(lái)從競(jìng)爭(zhēng)者那里“偷取”勝利。的RVM）都有一個(gè)“”團(tuán)隊(duì)，施加這種控制，甚至他們從大量的貢獻(xiàn)者那里接受工作，或解除外部貢獻(xiàn)者的認(rèn)證，對(duì)用戶(hù)群的。作為貢獻(xiàn)者工作，當(dāng)支持他們的聲譽(yù)增加。這個(gè)想法是，會(huì)自然的流向初始的“”小，一個(gè)心可以有的增長(zhǎng)者。如果貢者社區(qū)到臨界質(zhì)的體系，可以我調(diào)節(jié)而不需要接干預(yù)而，的以在需要時(shí)候進(jìn)者，和修復(fù)。一般來(lái)說(shuō)我們建議所有的貢獻(xiàn)者共享工作時(shí)被，雖然一些將會(huì)提前，來(lái)給予積極的鼓勵(lì)。支付可以推遲給予來(lái)確保貢獻(xiàn)者的軟件的健壯性，如果一這種在中間被分割的確切方式是仍然有待設(shè)計(jì)的，但是我們發(fā)現(xiàn)這受許多因素報(bào)告，創(chuàng)造測(cè)試樣列，和寫(xiě)文檔，能夠使得用戶(hù)能夠更好的理解代碼。作，他們的出在市場(chǎng)上積累的。實(shí)際上，市場(chǎng)上提供了一個(gè)方式為“通過(guò)隧道”一些不是0的。投標(biāo)。組織。確定當(dāng)兩個(gè)報(bào)告為同樣的的情況，或者看似為一個(gè)錯(cuò)誤，其實(shí)是兩種不同 修復(fù)。這可以是開(kāi)放的貢獻(xiàn)者具有很高的，但低 交的一個(gè)修復(fù)將提交給較高的人評(píng)價(jià)。提交。提交修正，以規(guī)范的代碼庫(kù)。這通常需要很高的市場(chǎng)價(jià)值的信息，和如何應(yīng)該被分為不同的貢獻(xiàn)者的工作流程。個(gè)緩慢增長(zhǎng)（甚至收縮）的。復(fù)的價(jià)值。在某個(gè)T1時(shí)間已經(jīng)成長(zhǎng)成了一個(gè)總量.一個(gè)開(kāi)發(fā)者，決定是否有的人發(fā)現(xiàn)，那么他們將獲得我們可以考慮一個(gè)支付調(diào)度策略，給發(fā)現(xiàn)者分發(fā)RB，同時(shí)的解決者，和的測(cè)試者在不同的比率上反應(yīng)解決錯(cuò)誤的貢獻(xiàn)。比如比率RB/(T1-T0)是計(jì)算在初重復(fù)報(bào)告。然而，某個(gè)時(shí)間過(guò)后，一些用戶(hù)執(zhí)行了一個(gè)不同的工作流，可能再現(xiàn)了2：深與淺的修復(fù)。由一引出的一個(gè)自然的問(wèn)題是，為了掌握的健壯性怎么樣設(shè)計(jì)激勵(lì)來(lái)修復(fù)，這個(gè)系統(tǒng)必須給予激勵(lì)來(lái)“深”修復(fù)，或者“淺”修復(fù)。一個(gè)貼現(xiàn)付款計(jì)劃可能是解決這個(gè)問(wèn)題的式。因此，后續(xù)的修復(fù)要比之前的修復(fù)有的，同時(shí)要為“深”修復(fù)。3：修復(fù)造成的額外效果。一個(gè)或多個(gè) 沒(méi)有意料到的系列錯(cuò)報(bào)告。我考慮一情況：用戶(hù)已在據(jù)庫(kù)中寫(xiě)了許對(duì)3的XY和Z的D，F(xiàn)，G，MQ的3方案是同一時(shí)間允許有個(gè)人修復(fù)。一修復(fù)需要間，同驗(yàn)證也需時(shí)場(chǎng)為引導(dǎo)的方法來(lái)驗(yàn)證一個(gè)的修復(fù)效果？比如，可以畫(huà)一張錯(cuò)誤的依賴(lài)關(guān)系圖，允許信用？4：錯(cuò)誤報(bào)告的誘達(dá)。這個(gè)系統(tǒng)必須引出和需要大量的總表現(xiàn) 告同時(shí)減少重復(fù)的數(shù)量。倘若一個(gè)合理的解決方法是找到用戶(hù)看重的并修復(fù)討這一;例如，它的效果怎么最小化，或者可以跳躍的影響社會(huì)福利？5致謝KarimLakhaniBenLubinRobO’CallahanEconCS組有AMarket-BasedApproachtoSoftwareDavidF. IBMResearchandHarvardUniversity HarvardUniversityDavid MalvikaHarvard Harvard Softwarecorrectnesshasbedeviledthe?eldofcomputersciencesinceitsinception.Softwarecomplexityhasincreasedfarmorequicklythanourabilitytocontrolit,reachingsizesthataremanyordersofmagnitudebeyondthereachofformalorautomatedveri?cationWeproposeanewparadigmforevaluating“correctness”basedonarichmarketecosysteminwhichcoalitionsofusersbidforfeaturesand?xes.Developers,testers,bugreporters,andystsshareintherewardsforrespondingtothosebids.Infact,wesuggestthattheentire disintermediatedmarket-mechanismdrivenbythedesiresofusersandthecapabilitiesof ,unspeci?able,andunknowablenotionofabsolutecorrectnessisthenrecedby?ablenotionsofcorrectnessdemand(thesumofbidsforbugs)andcorrectnesspotential(thesumoftheavailablepro?tfor?xingthosebugs).Wethensketchthecomponentsofamarketdesignintendedtoidentifybugs,elicitdemandfor?xingbugs,andsourceworkersfor?xingbugs.Theultimategoalistoachieveamoreappropriatenotionofcorrectness,inwhichmarketsdrivesoftwaretowardsacorrectnessequilibriuminwhichallbugsforwhichthereisenoughvalue,andwithlowenoughcostto?x,are?xed.CategoriesandSubjectD.2.4[Software/ProgramVeri?cation]:Reliability;D.2.10[Design]:Methodologies;J.4[SocialandBehavioralSciences]:Economics;K.6.3[SoftwareManagement]:SoftwaredevelopmentGeneralTermsDesign,Economics,Reliability,Speci?cation.Implementation.Veri?cation.Testing.Correctness.Thesearetheconceptsuponwhichreststhehumanenterpriseofsoftwarecreation,andthedisciplineofsoftwareengineering.Andattheirinlectualcore,evenifrarelyobtainedinpractice:Proof—thenotionthatwecan,inprinciple,createaspeci?cationforasoftwareartifact,implementit,andproveitcorrect.Thisinlectualfoundationforsoftwareengineering,thetonicidealofthebug-program,hasdominatedbothpracticalmethodologyandtheoreticalstudyincomputerOneoftheprogenitorsofthisidealizedapproach,EdsgerDijkstra,suggestedinhis1972TuringAwardlecturethatwellbeforetheseventieshaveruntocompletion,weshallbeabletodesignandimplementthekindofsystemsthatarenowstrainingourprogrammingability,attheexpenseofonlyafewpercentinman-yearsofwhattheycostusnow,andthatbesidesthat,thesesystemswillbevirtuallyofbugs.Todaythissuggestionseemsremarkablyna?ve,andifanythingthesituationhasgottenworseratherthanbetter.Variousapproachestocorrectnesshavebeenused:formalproofs(withsigni?cantprogressinrecentyearsusingmechanicalsystemsliketheCoqproofassistant)andmodelcheckingarebothseverelylimitedinthescaleofsoftwareartifacttowhichtheycanbeapplied.Someresearchershaveacknowledgedthatabsolutecorrectnessisinachievableandsuggestedalternateapproaches,notablyRinard’sworkin“failureoblivious”computing[13].Ourbeliefisthatweneedafundamentalchangeinourapproachtolarge-scalesoftwaresystemswhichreliesonorganic,self-regulatingmechanismsratherthanattemptingtoachievesomeabsolute,centralizednotionofcorrectness.Inthispaper,weexplorethepossibilityofusingamarketmechanismtodrivetheevolutionofsoftware.Thegoalisnottobebug,butrathertobeofbugsthatpeoplecareabout,andthatcanbe?xedeconomically.Webeginbysummarizingsomeapproachestousingmarketsystemsinvariouspartsofthesoftwaredevelopmentprocess,andthendescribeourproposedapproach.CROWD-SOURCINGUnliketraditionalapproachesthatseektechnicaladvancementstoensurethecorrectnessofsoftware,manyrecentsystemsare“crowdsourcing"varioussoftwaredevelopmentandimprovementtaskstotheparticipantsofthesoftwareecosystem.Crowd-sourcingisaprocessthatinvolvestheuseofacompetitiveprocess(thoughpayments,prizesorotherformsofreward)forthesourcingofworkorinformation,usuallyinvolvingproblem positionintosmall,modularchunks.Inthissectionwereviewexistingbugtrackingsystems,vulnerabilitymarkets,andonlinemarketcesthattakeacrowd-sourcingapproach.Aswewillsee,thesesystems,someofwhicharemarket-based,typicallyonlyfocusononeparticularaspectofthesoftwareecosystem.Intheirpaper"Theprinciplesofdistributedinnovation,"LakhaniandPanettadescribethreedifferentindustrieswheredistributedinnovationsystemshavebeenimplementedsuccessfully[7].Theauthorsdiscussthemotivationforpeopletoparticipateinsuchsystemsandtheorganizingprinciplesofproduction.ThepaperconsiderstheLinuxoperatingsystemasanexamplethathighlightsthebene?tsofworkthatisorganizedsuchthatmanyindividualscanself-selectandleadelementsofdevelopmentwithoutmuchexanteguidanceandcontrol.InfactLinusTorvalds,thefounderofLinux,isquotedasfollows:"IwouldmuchratherhaveBrownianmotionwherealotofmicroscopicdirectedimprovementsenduppushingthesystemslowlyinadirectionthatnoneoftheindividualdevelopersreallyhadthevisiontoseeontheirown."Thepaperalsopointsoutthatthereisarelativelyhighfailurerateinthesesystems.Moreoverdistributedinnovationsystemsdonotseemtobeef?cientintone-basedinnovationdevelopmentwhichre-quiresstrictnninganddeliveryondemand.BugandVulnerabilityItisnaturaltoengageandincentivizeusersand-markettesterstoreportbugsandidentifyvulnerabilityofsoftware.Voting-basedapproacheshavebeenusedtoallowuserstoexpresstheirpreferencesovertheimportanceofthebugs.Market-basedapproacheshavebeenproposedtoprovideincentivesforreportingcriticalsecurityorvulnerabilityBugreportingsystemsoftentrytogetusers’preferencesoverdifferentreportedbugs.Onewayforuserstoexpresssuchpreferencesistoallowthemtovoteonbugsorrelatedissuesthattheycareabout.Forexample,SunusesBugParadetotrackbugsintheJavaVirtualMachine.InBugParade,everyuniqueaddressisgivensometokensthatcouldbeusedtovoteontheimportanceofparticularbugs.Similarly,theAdobeFlexbugandissuemanagementsystemusesasystemcalledJIRAtoreportandtrackbugsfortheFlexBuilder/SDKandActionScriptCompilerprojects.JIRAallowsausertocastvotesforvariousissuesthatmaybeofrelevancetotheuser.Italsoallowsausertotrackaparticularissueandbenotifiedofanyupdatesregardingthatissue[6].Theaggregatedvotesallocatedtobugsarethoughttoreflectthelevelofinterestfromtheusercommunity.Aseachuserisallocatedalimitednumberofvotes,usersmustconsiderhowtocasttheirvotescarefully.Userscouldalsounvoteonissuesthatnolongerinterestthem–thisensuresthatmoreimportantissuesaredealtwithfirst[1].Whilevoting-basedsystemsofferexpressivenessinordertounderstanduserpreferencestheydonotprovidedirectincentivesforreporting,otherthanindirectlythroughthepotentialtoinfuencewhichbugsreceiveattention.Forsecurity-relatedissues,bugreportsareessentialtoensuretheintegrityofthesoftware.Hence,severalmarket-basedvulnerabilityreportingsystemshavebeenintroducedwiththegoalofincentivizinguserstoreportbugsandvulnerabilities.WereviewtwosuchsystemsThefirstoneiscurrentlyusedinpractice,whilethesecondoneisaproposedtheoreticalframework.MozillaSecurityBugBounty.TheMozillaFoundationoffersacashawardof$500andaMozillaT-shirttoanyonewhoreportsavalid,critical,securitybug[8].Thebugmustmeetcertaincriteriaincludingthefollowing:thebugmustbenew,itmustexistinthelatestsupportedversionofFirefoxorThunderbirdreleasedbyMozilla,andthebugfindermustnothavewrittenthebuggycodeorreviewedthatcodeorcontributedinanywaytowardsthatcode.BugreportersareencouragedtoworkwithMozillaengineersinresolvingthebug.Securityvulnerabilitiesaretreatedinaspecialwaybecausetheconsequencesofavulnerabilitybeingexploitedcanbeextremelyserious.SecuritybugreportsmaybekeptprivateforalimitedamountoftimetoenableMozillaengineerstofixthebugbeforeitismadepublic.Howeverthebugreporterisallowedtodecidewhentodisclosethebugtothepublic.Thebugreportermaychoosetodiscloseearlierifthebugisbeingignored,forexample.BynotpublicizinginformationaboutthebugimmediayupondiscoveryandinsteadreportingittoMozilla,bugrepor ctsintheinterestsoftheMozillaprojectandiscompensatedaccordingly.VulnerabilityMarkets.Schechter[14]proposestouseavulnerabilitymarkettoincentivizetesterstoidentifyvulnerabilitiesinsoftware.Inavulnerabilitymarket,softwareproducersoffertimevariantrewardstothefirsttesterswhoidentifyvulnerabilitiesinthesoftware.AminimumrewardvalueR0isinitiallyoffered,whichthengrowsovertimeataratedecided bytheproducer.Whenanewvulnerabilityisreporteditisfirstverifiedtobegenuine.Onpassingverification,thetesterresponsibleforthereportisawardedsomeportionofthereward.TherewardamountisthenresettotheminimumamountR0.Onlythefirstreporterreceivesthereward.Henceatesterthatwaitstoreportavulnerabilityincreasestheamountoftherewardbutalsoincreasesthelikelihoodthatanothertestermayreportthevulnerabilityfirsttherebycuttinghimoutofthereward.Clearlyinsuchaschemethemostfrequentlyoccurringvulnerabilitieswillbefoundfirst.Iftherewardremainsunclaimedtheproductisconsideredtobesafeenoughtoprotectinformationwhosetotalvalueisatmosttheamountofthereward.Firmshaveanincentivetocooperatewiththevulnerabilitymarketasitprovidesameasureofthequalityoftheirproductsandhencetheirreputation.Ozment[10]exinsthatthevulnerabilitymarketisessentiallyanauction–intheofanopenfirst-priceascendingauction,orreverseDutchauction.Thereisonebuyerofvulnerabilityintheauction,theproducer,andpotentiallyunlimitednumberofsellers,thetesters.Theinitialpriceissettoberelativelylowandthepricerisescontinuouslyuntilacceptedbyaseller.Thevaluationsofthesellersortestersareprivatebecausetheydependontheamountofworkorcostincurredinlocatingavulnerability.TheminimumrewardvalueR0istheauction’sreserveprice.Interestinglyitappearsthatthequalityofsoftwareandthesoftwareproducer’sinvestmentinpatchingtechnologyarestrategicsubstitutes.Ifitispossibletoreleaseapatchthensoftwareproducersenterthemarketsoonerandwithsoftwarecontainingagreaternumberofbugs[9].Ithasalsobeenshownthatsoftwareproducerstendtoissuepatcheslaterthanissociallyoptimal[9].WenotethatwhileboththeMozillaBugBountyandthevulnerabilitymarketprovideincentivesforreportingvulnerabilities,thevaluationofusersforapatchforaparticularvulnerabilityisnotcaptured.OnlinelanceMarketWhereasbugandvulnerabilityreportingsystemsareusedtoidentifyissuesinsoftware,theyoftendonotcouplethisidentificationofissueswithasourcingprocesstoperformworkinaddressingtheseissues.Thisgapisfilledwithonlinelancemarketces,whicharetformsthatconnectindividuals,small-businessowners,andevenFortune500companies lancetechnologyspecialiststosatisfytheirtechnologicalneeds.Thesitesprovidevividdetailsaboutworkers’historiesandqualifications,andsomeevenfeaturetoolsthatletthebusinessesmonitortheworktheyarepayingfor[4].Webdevelopers,softwareprogrammersandotherITspecialistsfromdifferentcountriesarereadilyavailableandoftenchargeafractionofthepriceoflocalworkers.Wewillexaminetwosuchonlinecompanies,RentACoderandTopCoder,below.WhiletaskstypicallyareindependentprojectsinRentACoder,individualtasksoftenarecomponentsofalargerprojectinTopCoder.WewillalsobrieflyreviewtheiTuneAppStore,whichisamarketceforsoftwareapplicationsforthe.TopCoderInc.usesprogrammingcompetitionstobuildprofessionalgradesoftwareoutsourcedbysrangingfromindividualentrepreneurstoglobalFortune1000companies.Foreverytask,participantscompeteagainsteachotherforcashaward.Thetoponeortwocontestantswintheaward.TopCodermemberscanworkforavarietyofoutsourcedsoftwareprojectsincludingthe"BugRaces,"whichisacompetitionforfixingbugs.Memberscanfixpostedbugsandthefirstsubmissionthatisverifiedtobefullyfunctional(accordingtotestsanddesigncriteria)isawardedaprize.Othertypesofsoftwaredevelopementprojects(softwaredesign,softwarearchitecture,etc.)useasimilarcompetitionformat.Boudreauetal.[3]arguethatthecompetitiontformofTopCodercanincreasethequalityofthebestsolutionbybroadeningthesearchforinnovation.WenotethatTopCoderseemstobeusedmainlyfordenovodesignratherthanforimprovementsorfixestoexistingsoftwaresystems.CentraltoTopCoder’smethodologyappearstobethemodularizationofsoftwaredevelopmentwork[5].Eachprojectisbrokendowntothemostgranularlevelpossible.Asaresultmostpiecesofworkcanbecompletedinafewhours[2].Inadditionthemodularizationofprojectsallowsforsimpleevaluationcriteriawhendeterminingifasubmittedsoftwaresolutionorbugfixisindeedcomplete.Italsoreducesthelikelihoodthatnewlysubmittedcodemayexertunintendedeffectsontheexistingcodebase.Hencethewinningprogrammermayberewardedforhisworkassoonasitisvalidated.ThishasthebenefitofreducinguncertaintyforTopCoderaswellasfortheprogrammers.Interestingly,thismodularization(andarchitecting,morebroadly)isitselfperformedwithinthesamecompetitiontform.WhileTopCoderprovidessomemonetaryincentivesforitsparticipants,itsfixedpricingfortasksmaynotbeefficientinidentifyingthebestparticipantstoperformatask.Moreover,TopCoderisalsomorethanasoftwarecrowd-sourcingtform,inthatitalsoprovidesvalueinestablishingacommunityoftalentedcodersandhelcompaniestoidentifytalent.TopCoderisalsoaneasilyaccessiblepracticinggroundforprogrammerslookingtohonetheirskills.Allsubmissionstoataskreceivefeedbackfromapeerreviewprocessinvolvingmultiplemetricsonascorecard,whichprovidesvaluableinformationonhowasubmissioncouldhavebeenimproved[2].Firmslikeandoftensponsorscreeningcontests[3],andseemtohiretalentedRent-a-Rent-a-Coderisanotheronlinemarketcethatconnectsbuyerstocoders.Allprojectsareprotectedbyescrowandthrougharbitration.Buyerspostnewprojectsonthesite.Sellers(coders)postquestionsandsubmitbidsontheprojects.Thebuyerthenselectsthesellerthatheorshewantstoawardtheprojecttoandputsthefundsintoescrowasapaymentguarantee.Whentheworkiscompleted,thebuyerreleasesfundsfromtheescrowaccounttotheseller.Ifthesellercompletestheworkbutthebuyerwithholdsthefunds,anarbitratorwillstepin,testthesoftwareifnecessary,andreleasethefundstothesellerasappropriate[2].DifferentauctiontypesarealllowedwithinRent-a-Coder.InanopenauctionallmembersofRent-a-Codercanbidforaproject.Inaprivateauctiononlythosecodersthatthebuyerinvitesmayparticipate.Otherarrangementsarealsopossiblesuchas"PayforTime"wherethebuyerpaysacoderforthetimespentonaprojectratherthanforanendresult.Rent-a-Coderchargescodersafeerangingfrom7.5%to15%ontheprofitfromworkdone[12].Oncetheworkiscompletedthebuyersandsellersmayrateeachother.ThisisdifferentfromTopCoder,wheremostofthereputationinformationisaggregatedfromdirectlymeasurableperformancemetricssuchasthefractionoftestspassedbydevelopedcode.iTunesAppAnothersystemthatprovidesamarket-basedapproachtoadifferentpartoftheapplicationdevelopmentecosystemistheiTunesAppStoreforthe.TheAppStorehasabuilt-inmicropaymentsystemwheredeveloperscanmakesignificantprofitsevenwithapplicationspricedat$0.99.Thereisalsoaratingsystem.Popularapplicationshavethousandsofratings,anditisinterestingthatalargeproportionofwhichincludecommentsfromusersaboutoutstandingbugsanddesiredfeatures.Developersoftenaddresstheseissuesexplicitlyintheircommentswhenreleasingupdates.Inthisway,theAppStoreiseffectiveinbetterconnectingendusersanddevelopers.TheAppStoreshowsthatamicropayment-basedmodelforsoftwaredeliveryiseminentlypracticalandwell-receivedbyconsumers,butlacksawayformultipleprogrammerstocontributetoaprojectorforuserstovoteorbidforfeaturesandfixesinanorganizedfashion.MARKET-DRIVENWenowdescribeourproposalforusingamarket-basedmechanismtodrivetheevolutionofsoftwaretoincreaseitscorrectnessanditsfunctionality.Weunifymanyofthepreviouspartialmarketbasedmechanisms,incorporatingbugreporters(asinvulnerabilitymarkets),bugvoters(asinBugParade)anddevelopersandvalidators(asinTopCoderandThisproposalisnotyetcomple yfleshedout(oratallimplemented).Togetsomeintuitivesenseofwhatwehaveinmind,imagineacombinationoftheiTunesAppStoreandBugzillawhereuserscanbidmicropaymentsforthefixingofbugsorimplementationofnewfeatures,andanyqualifieddevelopercanobtainaccesstothecodeandperformtherequestedwork.Userscouldofferaslittleasapennyforafixorfeature,andaggregatedemandcouldstillbesufficienttomakeitworthwhileforadevelopertosatisfythem.Furthermore,thedemandcanbeusedbythedevelopertodecidehowtoconcentrateefforttoincreasequalityandattractmoreusers.Onemissingelementofexistingvoting-basedsystemssuchasBugParadeisthattheydonotdirectlyelicitadditionalworkinfixingbugsinreleasedsoftware.Rather,theyareusedtofocusexistingprogrammerresourcesonproblemsidentifiedasimportantbyausercommunity.Ourapproachisdesignedtoallowuserstoelicitadditionalwork(andencouragenewdeveloperstoworkwiththesoftware)byofferingrewardsforthework.Morespecifically,weconsideramarketecosystemaroundaparticularpieceofTheevolutionofsoftwareisadynamicprocess.Howeverwewillfirstpresentastaticsnapshotofthesoftwaremarketecosystem.TheecosystemcomprisesthefollowingbasicUUsersofthesoftware.Usersmightbeindividuals,corporations,orotherJJobswhichtheuserswouldliketohaveperformedontheWWorkerswhomayperformjobs.Notethatworkersmayalsobeusersandvice KKindswhichare(optionally)usedtocategorizejobs.Examplesarecorrectness,security,mac,LabelLkjis1ifjobjhaskindk,0Rewardofferedbyuseruforjobjattimet.Ifuserudoesnotofferarewardforjobjtimet,then toworkerwforperformingjobjattimet.Ifworkerwisincapableoforuninterestedinperformingjobjattimet,thenweconsiderThelabelsexistforcategorizationandtoallowthecalculationofaggregatestatisticsaboutdifferentcategoriesofworkitems.Themarketisagnosticastothesecategories.However,usersmayexpresspreferencesbyofferingrewardsforparticularkindsofjobs;wewillexpandonthisbelow.Weassumethatrewardsareinanactualcurrency(denoted“$”)andconsiderforthemostpartanopenmarketinwhichtheonlybarriertoentryforaparticularworkeristheirabilitytocompleteaparticularjob(asopposedtotheavailabilityofthesourcecode).Akeyfeatureofourmarketisthatitincorporatesboththeaggregationofuserbids(likebugvotingsystems)andmultiplecompetingworkers(likeTopCoder).Thisexchangestructure,withinformationandpreferences(e.g.,costsfordifferentkindsofwork,valuesfordifferentkindsoffixes)onbothsides,isdesignedtoprovideforamoreefficientmarketce.Forthetimebeing,wemakeanumberofsimplifyingassumptions:thatusers’bidsarecorrectlyaggregated(thatis,wecandeterminewhentwousersofferarewardforfixingthe“same”bug)andthatasingleworkercanperformajobinitsentirety(whichrequiresjobstobemodularizedappropriay,asisthecaseinTopCoderforexample).InSection4.4wediscusshowtousemarketmechanismstohandleamorerealisticUserGiventheecosystemdescribedabove,thetotalrewardforajobjattimetandwedefinetheuserdemandonapieceofsoftwareastheaggregateoftheavailableWhenR=0,thereisnodemandforchangesoradditionstothesoftware.Notethatthisdoesnotimplythatusersaresatisfied:thedemandmightbe0becauseusersdon’tlikethesoftwareandnooneisusingit.WedenotethedemandforaparticularkindkofjobattimetAsdiscussedintheintroduction,softwarehastraditionallybeenconsideredtohavesomeideal“correct”stateinwhichthereare“nomorebugs."Henceforthwewillrefertothisnotionasabsolutecorrectness.Whileitisalaudablegoal,foranythingbutsmallmodulesabsolutecorrectnessisunachievable,andinmanylargesystems,cannotevenbecompleyspecified.However,inourmarket,wecanexpressthecorrectnessdemandinperiodtassimplyRt[correctness].Ifthecorrectnessdemandis0,itdoesnotmeanthattherearenobugsinthesoftware–butitdoesmeanthattherearenobugstowhichanyusersattachvalueforfixing.ogously,wecanfythedemandforsecurity,newfeatures,supportforaparticulartform,andsoon.MarketPotentialandIntuitively,thejobsthatare“worthng”forworkersarethosewherethecostofperformingtheworkislessthantheexpectedreward.Morespecifically,wecandefinethepotentialvalueofajobjattimetasThatis,thepotentialvalueofajobistherewardnetcostthatcanbeobtainedbytheworkerwhocanperformitforthelowestcost(providedthattherewardisatleasttheircost).Wedefinethepotentialvalueoftheentirejobmarketas:ThecorrectnesspotentialofasysteminperiodtisthenPt[correctness].NotethatPt[correctness]Rt[correctness],sothatifthereisnocorrectnessdemandthenthereisnecessarilynocorrectnesspotential.TheuserdemandRtjcanbethoughtasthepriceatwhichusersdemandfortheworkonjobj(orthebidprice).Ctwjthenrepresentsthepriceatwhichworkerwiswillingtosupplytheworkforjobj(ortheaskprice).Ideally,wheneverthebidpriceisgreaterthanorequaltotheaskprice,themarketshoulddrivetheworktohappen.Hence,wedefinethatasystemisincorrectnessequilibriumwhenPt[correctness]=0.Asystemincorrectnessequilibriumisonewhereallofthebugsthatare“worthfixing”havebeenfixed.Theremaystillbeplentyoflatentbugs,orevensignificantcorrectnessdemand,buttherearenolongeranybugswhichaworkercanfixwithoutlosingmoney.DynamicsintheThenotionsofcorrectnessdemand,potentiaandequilibriumhavesofarbeendiscussedinthecontextofan(implicitly)staticsystem,whereagroupofusersbidrewardsforajobtobeperformed,andworkersdecidewhethertoperformthejobsbasedontheircostsandtheeconomicrewards.However,arealsystemisdynamic:therewardforajobmayfluctuate(upwhenuserspostadditionalrewardsforthejobordowniftheuserhasspecifiedan“expirationdate”forthereward).Furthermore,workersmaydecidetoattackaproblematdifferenttimesdependingonhowprofitableajobisandwhethertherewardforajobmightincreasefurther,aswellasconsiderationsaboutthelevelofcompetitionwithotherworksandotheropportunities.Thusapieceofsoftwarewithalargeandvibrantusercommunitymayhaveauserdemandandamarketpotentialthatremainhigh: