《計算機視覺-王偉強》Course9

ComputerVisionII

BuildingRomeinaDay

Recall

?Fundamentalmatrixsong

?http:〃/fmrtrix/

?RANSACsong

How?

MCOHOcomoM

MultipleViewReading:theMVGbible

Geometry

incomputervision(need~2years)

Crying:NotWorkingAtAll

II

+

+

UOD

-O-U

1。①-

IO①P

IA①」IO

一OJOA

uS-uI

ojmdP

jdB

j①OJUOBS

>EI

①IAe

」q

n①—

-iz①B

d①-」

-uAPf

rn-OqbeJO

lulIxu

sAA①-J

sII一

d

①:s

t

s一os:s:

-以--

sd①①

l①pP

u①|(zhì)一|oO

o」S/I

d0①|(zhì)\IA

2ai

個

個i個

個

s個ss

①ss①①

luI一

bjeouo8ODe

loAIE

_uddI_

+

U

-O

1。OUD

IO①-

A」①

I①P

ES一OuJOIO

omd-IA

^①ODUBdP

IB

s>EAS

n①Ie

-iz①q

d-」

R①r—①

-ru-Pn

unIOx

slAlA①bjeo

sII一l

d_u

①s:

l

s一u

一o

sd以±①5±①i

l①PP

u①|(zhì)一OO

o」SAII

d0①IIA

5ai

個

個

i個

個

s個ss

①ss①①

luq

bjeousbjo

loo①e

工E

_udd_

+

OU

-。OUD

O1①-

」①

AI①P

I一O

ESOuJOAI

omd-I

①dP

^ODUBB

>EIS

sIAe

n①q

d-iz-①—

-R①r」①

-runOPn

ulIxbjeo

slIAA①l

sI一

d_u

①s:

l

一u

so

以

s一d±①5±①i

l①①PP

一OO

u」|I

o0SAIA

d①II

5ai

個

i個

個

個

個ss

ss①①

①lsq

us

bjouobjeo

leo①E

dd工_

-u

+

U

-O

。OD

1-U

IO①-

」①

IA①P

一O

ESOuJOAI

omd-I

^①ODdP

UIBB

s>EAS

Ie

n①q

-iz-①

d」—

R①r①

-run-OPn

ulIxbjo

slAA①le

sII一

d_u

①s:

l+

一u

so

以

s一d±①5±①i

l①PP

u①一OO

o」S|II

d0①IAIA+

5ai

個

個

個i

個

s個ss

①ss①①

luq

bjousbjeo+

leoo①E

-udd工_

+

U

-O

1。OUD

IO①-

A」①

I①P

U一OJOO

JSuAI

Omd-I

J」①ODUBdP

IB

s>EAS

n①Ie

-iz①q

d-」

R①r—①

-ru-Pn

unIOx

slAlA①bjeo

sII一l

d_u

①s:

l

s一u

一o

sd以±①5±①i

l①PP

u①|(zhì)一OO

o」SAII

d0①IIA

5ai

個

個

i個

個

s個ss

①ss①①

luq

bjeousbjo

loo①e

工E

_udd_

+

U

-O

1。OUD

IO①-

A」①

I①P

ES一OuJOIO

omd-IA

^①ODUBdP

IB

s>EAS

n①Ie

-iz①q

d-」

R①r—①

-ru-Pn

unIOx

slAlA①OJeO

sII一l

d-u

①s:

l

s一u

一o

sd以±①5:

l①P-s

u①|(zhì)一O①

o」SIP

d0①IAO

5aiAI

個

i個I

個

個

ss個

①ss①

luqs

bjeous①

loo①

udd工bjeo

_E

_

+

U

OOD

--U

1。-

IO①①

IA①」OP

U一OJOAI

JS-uI

OmdP

J」①ODUBdB

IS

s>EAe

n①Iq

-iz①—

d-」B

-uR①r

ur-OPn

llnIxOeJO

sAA①ul

sII一-

d

①s:

l

s一u

一o

以:

sd±①5s

l①P-①

u①|(zhì)一OP

o」SAIO

d0①IAI

5aiI

個

i個

個

個

s個s

①ss①s

luq①

bjeous

loo①bjeo

udd工E

__

+

Image-basedModeling

InputImages

Image-basedModeling

U

-O

1。OUD

IO①-

A」①

I①P

ES一OuJOIO

omd-IA

^①ODUBdP

IB

s>EAS

n①Ie

-iz①q

d-」

R①r—①

-ru-Pn

unIOx

slAlA①bjeo

sII一l

d_u

①s:

l

s一u

一o

sd以±①5±①i

l①PP

u①|(zhì)一OO

o」SAII

d0①IIA

5ai

個

個

i個

個

s個ss

①ss①①

luq

bjeousbjo

loo①e

工E

_udd_

+

U

OODU

-。-

1①-

O①

I」

A①P

O

I一

SI

lOuJOA

umd-I

oP

①OJuOd

」BB

J-

>IS

①HA

Ie

」①

nq

-iz-①—

d①

A」①

-Pr

-runOn

ulIxOJeO

lAAu

sII①J

s一

d-

①s:

l

s一u

o4s:

-d-s:

s①①-

①iP①

l|OP

u」SIO

o0①AI

i

d2aIA

iI

個

個

個

個

個

s

sss①s

①l①

l一

uuI

bjeoo8

loIODe

uddAE

_I_

+

U

-O

1。OUD

IO①-

A」①

I①P

ES一OuJOIO

omd-IA

^①ODUBdP

IB

s>EAS

n①Ie

-iz①q

d-」

R①r—①

-ru-Pn

unIOx

slAlA①OJeO

sII一l

d-u

①s:

l

s一u

一o

sd以±①5:

l①P-s

u①|(zhì)一O①

o」SIP

d0①IAO

5aiAI

個

i個I

個

個

ss個

①ss①

luqs

bjeous①

loo①

udd工bjeo

_E

_

+

StructureFromMotion

?Structure=3DPointCloudoftheScene

?Motion=CameraLocationandOrientation

?SFM=GetthePointCloudfromMovingCameras

?StructureandMotion:JointProblemstoSolve

Pipeline

StructurefromMotion(SFM)Multi-viewStereo(MVS)

Pipeline

StructurefromMotion(SFM)Multi-viewStereo(MVS)

Two-viewReconstruction

Two-viewReconstruction

Two-viewReconstruction

keypoints

fundamentalessential

match[R|t]triangulation

matrixmatrix

keypoints

KeypointsDetection

keypoints

fundamentalessential

match[R|t]triangulation

matrixmatrix

keypoints

Descriptorforeachpoint

SIFT

descriptor

SIFT

descriptor

keypoints

4,fundamentalessential

match?[R|t]triangulation

matrixmatrix

keypoints

Samefortheotherimages

SIFT

descriptor

SIFT

descriptor

keypoints

-,fundamentalessential-4...

match[Rt]triangulation

matrixmatrix

keypoints

PointMatchforcorrespondences

keypoints

fundamentalessential-4...

match[Rt]triangulation

matrixmatrix

keypoints

PointMatchforcorrespondences

keypoints

fundamentalessential-4...

match[Rt]triangulation

matrixmatrix

keypoints

FundamentalMatrix

X]ox2

X^FX2=0

Exercise

WhatisthedifferencebetweenFundamentalMatrixandHomography?

(Bothofthemaretoexplain2Dpointto2Dpointcorrespondences.)

EstimatingFundamentalMatrix

?Givenacorrespondence

X]—x,

?Thebasicincidencerelationis「

X；RX,二O卜/2，不必，再，必工2，必必，如%2，％1]f?2=。

Need8points,

EstimatingFundamentalMatrix

x；Rx,=0for8pointcorrespondences:

fn

]

x\x\“1%X；必“2y\y\乂x\y21

fl2

小;片泡諭xl1

.713

X：y渴y'ylx|1Ax=b

fl\

九:血1

,22=0Af=0

y"x2y"1

工1工2七％X：，23

X%X泗X；y\xi1

77

Xx

xxx2；1y"2x\1

Ky.力2

8X8

Xx\1

人1人2X：工3

DirectLinearTransformation(DLT)

AlgebraicErrorvs.GeometricError

?AlgebraicError

min||Af||

?GeometricError(better)Unit:pixel

Solvedby(non-linear)leastsquaresolver(e.g.Ceres)

RANSACtoEstimateFundamentalMatrix

?Formanytimes

一Pick8points

一Computeasolutionforpausingthese8points

一Countnumberofinliersthatwithcloseto0

12

?Picktheonewiththelargestnumberofinliers

MinimalproblemsinComputerVision

CenterforMachinePerception

MinimalproblemsinComputerVision

cmp.felk.cvut.cz/minimal/

Overview

?P4P?unknownfocallength

?P4P?focal?radialdistortionMinimalproblemsincomputervisionarisewhencomputinggeometricalmodelsfromimagedata.

Theyoftenleadtosolvingsystemsofalgebraicequations.

?uP2P.knownverticaldirection

?uP3P+foal?radialdtotortionThispageprovideslinkstopublications,software,data,andevaluationofminimalproblems.

?P2PlLprobtam

NEW:

?PlP2Lproblem

ACCV2010

?5-ptrelativeposeproblem

BujnakM”KukelovaZ.,PajdlaT.,Newefficientsolutiontotheabsoluteposeproblemforcamerawithunknownfocallengthandradial

?6-ptfocaltengthproblemdistortion,ACCV2010,Queenstown,NZ,November8-12,2010.FodH

?caNbrated-uncaRbratedCODE:

4~oolrnabsoluteposeorobtemwithunknownfocallenathandradialdistortion(P4Pfr)

?4-pt3-view*ralathmpoMprobtem

?2-ptpanoramastitching(1focal)KukelovaZ.tBujnakM.,PajdlaT.,Gosed-formsolutionstotheminimalabsoluteposeproblemswithknownverticaldirection,ACCV2010,

Queenstown,NZ,November8-12,2010.(',]

?3-ptpanoramastitching(2locate)CODE:

?3>ptpanoramastitching(1focal*2jxMntabsoluteooseprobiemwithknownverticaldirection(up2p)

rod*dtotortfcxO3~pointabsolutegseproblemwithknownverticaldirectionandunknownfocallengthandradialdistortion(uD3pfr)

?Aptgenerattzedcameraproblem

*6>ptcaNbratedradtoldtotortionSOURCECODESTOSEVERALMINIMALPROBLEMS

4PoimabsoluteooseDroblemwithunknownfocallenqS(P4Pf)(newfastM^tlabversion)

?8*ptuncaHbratedradialdistortion8-point"uncatibrated"retiDoseDroblemwithradialdistortion

?9-ptdWtrentdistortionprobtam4~oointabsoluteposeDroblemwithunknownfocallenathandradialdistortion(P4Pfr)

2-pointabsoluteDoseDroblcmwithknownvertiTdirection(UD2D)

?P2flregistrationproblem3-oolntabsoluteooseDroblemwithknownverticaldirectionyidunknownfocallenathandradialdistortion(uo3Dfr)

?3-vtewtrlangulatton

?9-ptcatadioptric

3oolntlute?orobiem(P3P)

?Hand-?ye4?pointobsoluteposewigunknownfocallend由(P4P「)NEWFASTMATLABSOURCECODE

?Automaticgenerator4-pointabsoluteDoseoroblemwithunknownfocallenathandrad@distortion3口/NEW.

2pomiabsoluteposeproblemwithknownverticaldirection(UD2D)NEW

?Grobenerbasissolver

3~oointluteooseDroblOTiwithknownverticaldirectionandunknownfocallength3ndradialdistortion(uo3Dfr)NEW

http://cmp.felk.cvut.cz/minimal/

FundamentalMatrix->EssentialMatrix

X^FX2=0

E=K?FK?

區(qū)232

EssentialMatrixfRt

E=K?FK?

區(qū)2肯2

EssentialMatrix|_Rt_

Result9.19.Foragivenessentialmatrix

r

E=Udiag(l5l90)V?

andthefirstcameramatrixtherearefour

possiblechoicesforthesecondcameramatrixP2：

r

P2=[uWV|+u3

P=[lJWVr|-u

230-10

rr

P2=[uWV|+u3W=100

001

rr

P2=[uWV|-u3

Page259ofthebible(MultipleViewGeometry,2ndEd)

FourPossibleSolutions

(d)

Fig.9.12.ThefourpossiblesolutionsfbrcalibratedreconstructionfromE.Betweentheleftand

rightsidesthereisabaselinereversal.BetweenthetopandbottomrowscameraBrotates180°about

thebaseline.Note,onlyin(a)isthereconstructedpointinfrontofbothcameras.

Page260ofthebible(MultipleViewGeometry,2ndEd)

Triangulation

Infrontofthecamera?

CameraExtrinsic[R|t

?CameraCenter

—R『t二—R't

?ViewDirection

-(C)=(R(3,:)r-Rrt)-(-Rrt)=R(3,:)r

CameraCoordinateSystemWorldCoordinateSystem

Infrontofthecamera?

?Apointx

?DirectionfromcameracentertopointX-C

?AngleBetweenTwoVectors

A?B=ABcos。

?AngleBetweenx-CandViewDirection

?Justneedtotest

（X-C）?R（3,：y〉0?

PicktheSolution

Withmaximalnumberofpointsinfrontofbothcameras.

Fig.9.12.ThefourpossiblesolutionsforcalibratedreconstructionfromE.Betweentheleftand

rightsidesthereisabaselinereversal.BetweenthetopandbottomrowscameraBrotates1800about

thebaseline.Note,onlyin(a)isthereconstructedpointinfrontofbothcameras.

Page260ofthebible(MultipleViewGeometry,2ndEd)

Two-viewReconstruction

keypoints

fundamentalessential

match[R|t]triangulation

matrixmatrix

keypoints

Pipeline

StructurefromMotion(SFM)Multi-viewStereo(MVS)

Pipeline

TaughtNext

MergeTwoPointCloud

*3

If

??**

TherecanbeonlyoneR2t2

MergeTwoPointCloud

?Fromthe1stand2ndimages,wehave

[Riand[R2|t2_

?Fromthe2ndand3rdimages,wehave

R2tandFR3L

?Exercise:Howtotransformthecoordinate

systemofthesecondpointcloudtoalignwith

thefirstpointcloudsothatthereisonlyone

?*

22

Oops

SeeFromaDifferentAngle

BundleAdjustment

“Iamveryverysexy"

Point1Point2Point3

1

Image1x；=Itjx襄；=也弭h

Si:'1jSSL-

Image23

寸=吆電芍=X^A=KTIL遙.遇t三Jx

k[R21tzL

Image3—ITD11Y1KRtlx3

Y=T.賽'__

V?一有]矍_.

RethinkingtheSFMproblem

?Input:Observed2Dimageposition

~1~2

&X]

支1X2克3

八2八2八2

?Output:X3X3

UnknownCameraParameters(withsomeguess)

[此4區(qū)也口憶

UnknownPoint3Dcoordinate(withsomeguess)

丫1丫2丫3

￡》*造￡Y

777

T叢【'uoneAJSsqo

裊￥

￡X11|=；xMW、［x=：x］

￡x［，『小=""|X「i|\］y=；xuoii39［ojd-9y

Mmr］x=；x|X"|，］x=；x

問isniu

-\x\x\xpue\'口『\］『1|、］U°!W|OSP!|BAV

luaiuisnrpva|pung

BundleAdjustment

3

Avalidsolution[K|tJC%12」，W3handX\X^X,

mustlettheRe-projectionclosetothe

Observation,i.e.tominimizethereprojection

error

72

minZZ("-K[Rz.|tz.]x)

ij

BundleAdjustment

3

Avalidsolution[K|tJC%12」，W3handX\X^X,

mustlettheRe-projectionclosetothe

Observation,i.e.tominimizethereprojection

error

72

minZZ("-K[Rz.|tz.]x)

ij

Question:Whatistheunitofthisobjectivefunction?

BundleAdjustment

Avalidsolution[K|tJC%周，恢3handX\X^X3,

mustlettheRe-projectionclosetothe

Observation,i.e.tominimizethereprojection

error

72

minZZ("-K[Rz.|tz.]x)

ij

Linking

SIFTMatchingSIFTMatching

SolvingThisOptimizationProblem

?Theory:

TheLevenberg-Marquardtalgorithm

http://en.wikipedia.or4/wiki/Levenberg-Marc1uvrdtalgorithm

?Practice:

TheCeres-SolverfromGoogle

http:〃code.及。。由/p/ceres-soker/

ParameterizingRotationMatrix

-sin(6>)x

cos(e)y

R7=R\detR=l

x

3DRotation

Yaw,pitchandrollare%0/

100

&(。)=0cos(^)-sin(0

0sin(6)cos(。)也伊((QK(a)

cos(6)0sin(。)

Ry(e)=0io

-sin(O)0cos(6)Euleranglesare

cos(0-sin⑹0

⑻

Rz=sin(0cos(^)0Rz仍R")R,(a)

001

/wiki/Rotationmatrix

3DRotation

Axis-anglerepresentationQuaternions

.(e\（州

vsin—,cos

<27\2J)

AvoidGimbalLock!

Recommendation!

TripletRepresentationv6(3dof)GNotover-parameterized

Rodrigues'rotationformula

kmz=kcos8+(vxk)sine+v(v-k)(l—cos。)

http:〃/wiki/Axis-anglerepresentation

http://en.wikipedi).orH/wiki/Quaternionsandspatialrotation

http://en.wikipedia.ore/wiki/Rodrigues%27rotationformula

InitializationMatters

~1~2

X]X]

(

?Input:Observed2Dimageposition-i-23

?*2X?A2

?Output:ji女;

UnknownCameraParameters(withsomeguess)

RltjRHJAh

UnknownPoint3Dcoordinate(withsomeguess)

Pipeline

TaughtNext

MultipleViewStereo

State-of-the-art:

PMVS:http:〃vrail.cs.washinR/softwar~e/pmvs/

Accurate,Dense,andRobustMulti-ViewStereopsis,YFurukawaandJPonce,2007.

Benchmark:

http:〃/mview/

AComparisonandEvaluationofMulti-ViewStereoReconstructionAlgorithms.

SMSeitz,BCurless,JDiebel,DScharstein,RSzeliski.2006.

Baseline:

Multi-viewstereorevisited.MGoesele.BCurless,SMSeitz.2006.

Keyidea:MatchingPropagation

[1]LearningTwo-viewStereoMatching,JXiao,JChen,DYYeung,andLQuan,2008.

[2]Accurate,Dense,andRobustMulti-ViewStereopsis,YFurukawaandJPonce,2007.

[3]Multi-viewstereorevisited.MGoesele,BCurless,SMSeitz.2006.

[4]RobustDenseMatchingUsingLocalandGlobalGeometricConstraints,MLhuillier&L

Quan,2000.

Inanothercontext:

[5]PatchMatch:ARandomizedCorrespondenceAlgorithmforStructuralImageEditing,

CBarnes,EShechtman,AFinkelstein,andDBGoldman,2009.

SimplestMatchingPropagation

Wearegoingtolearnaverysimplealgorithm

Descriptor:ZNCC(Zero-meanNormalizedCross-Correlation)

Invarianttolinearradiometricchanges

Moreconservativethanotherssuchassumofabsoluteorsquaredifferencesinuniformregions

Moretolerantintexturedareaswherenoisemightbeimportant

2^(/(x+i)-7(x))(/(x+i)-7(x))

*1122

ZNCC(X1,X2)

Descriptor:ZNCC(Zero-meanNormalizedCross-Correlation)

,Invarianttolinearradiometricchanges

,Moreconservativethanotherssuchassumofabsoluteorsquaredifferencesinuniformregions

,Moretolerantintexturedareaswherenoisemightbeimportant

2^(/(x1+i)-7(x1))(/(x2+i)-7(x2))

si

o

UOABuBdoJdpas

c

MatchingPropagation

?MaintainapriorityqueueQ

?Initialize:PutallseedsintoQwiththeirZNCC

valuesasscores

?Foreachiteration:

一PopthematchwithbestZNCCscorefromQ

—Addnewpotentialmatchesintheirimmediate

spatialneighborhoodintoQ

?Safety:handleuniqueness,andpropagate

onlyonmatchablearea

MatchableArea

theareawithmaximalgradience>threshold

l-ns①工

+

+

#

+++

++

+++

#++

孕>+