Table Of Content

Optimal Radiometric Calibration for Camera-Display Communication WenjiaYuan EricWengrowski KristinJ.Dana AshwinAshok MarcoGruteser NarayanMandayam DepartmentofElectricalandComputerEngineering,and WINLAB(WirelessInformationNetworkLaboratory)RutgersUniversity 5 Tel.: +732-445-5253 1 ContactAuthor: [email protected] 0 2 n a Abstract Online J Radiometric Calibration 8 Unknown We present a novel method for communicating between Camera Pose acameraanddisplaybyembeddingandrecoveringhidden and ] Photometry V anddynamicinformationwithinadisplayedimage.Ahand- Displayed C heldcamerapointedatthedisplaycanreceivenotonlythe Intensity Histogram displayimage,butalsotheunderlyingmessage. Theseac- Camera-Display Transfer Function (CDTF) Preserved . s tivescenesarefundamentallydifferentfromtraditionalpas- c [ sivesceneslikeQRcodesbecauseimageformationisbased ondisplayemittance,notsurfacereflectance.Detectingand 1 Message Embedding v decodingthemessagerequirescarefulphotometricmodel- Intensity Histogram Warped by CDTF Message Recovery 4 ing for computational message recovery. Unlike standard 4 watermarking and steganography methods that lie outside Figure 1. The above flowchart illustrates the process by which 7 the domain of computer vision, our message recovery al- 1 gorithmusesilluminationtoopticallycommunicatehidden Online Radiometric Calibration is used to estimate and negate 0 the light-altering effects of the Camera-Display Transfer Func- messages in real world scenes. The key innovation of our . tion (CDTF) in camera-display communication. Variables such 1 approach is an algorithm that performs simultaneous ra- as camera pose, photometry, and hardware all have a significant 0 diometric calibration and message recovery in one convex effectonlightsignalspassingfromelectronicdisplaytocamera. 5 optimization problem. By modeling the photometry of the Ineachpairofintensityhistogramsshownabove, theleftrepre- 1 : system using a camera-display transfer function (CDTF), sentsanimage’shistogrambeforepassingthroughtheCDTF,and v wederiveaphysics-basedkernelfunctionforsupportvec- therightrepresentsthehistogramaftertheCDTF.OnlineRadio- i X tormachineclassification.Wedemonstratethatourmethod metricCalibrationmitigatesthedistortingeffectsoftheCDTFto ofoptimalonlineradiometriccalibration(OORC)leadsto preservetheimage’shistogram,enablingmoreaccurateimagere- r a anefficientandrobustalgorithmforcomputationalmessag- covery. ingbetweenninecommercialcamerasanddisplays. simpleexamplesbecausethebold,staticpatternmakesde- 1.Introduction tectionsomewhattrivial. Theproblemismorechallenging from a computer vision point of view when the codes are While traditional computer vision concentrates on ob- notvisiblemarkers,butratherarehiddenwithinadisplayed jectsthatreflectenvironmentlighting(passivescenes),ob- image. The displayed image is a light field, and decoding jects which emit light, such as electronic displays, are themessageisaninterestingprobleminphotometricmod- increasingly common in modern scenes. Unlike passive eling and computational photography. The paradigm has scenes, activescenescanhaveintentionalinformationthat numerous applications because the electronic display and mustbedetectedandrecovered.Forexample,displayswith the camera can act as a communication channel where the QRcodes[13]canbefoundinnumerouslocationssuchas displaypixelsaretransmittersandthecamerapixelsarere- shopwindowsandbillboards. However,QR-codesarevery ceivers [2][1][31]. Unlike hidden messaging in the digi- 1 Thecharacterizationofthisfunctionisdonebymeasuring I Display Light in Free Camera I d Space c a range of scene radiances and the corresponding capture images pixels. Our problem in camera-display messaging Emmitance Sensitivity Radiometric is similar but has important key differences. The CDTF Function Function Response e((cid:79),(cid:84)) s((cid:79)) f(i) is more complex than standard radiometric calibration be- cause the system consists of both a display and a camera, Figure2.ImageFormationPipeline:TheimageI isdisplayedby d each device adding its own nonlinearities. We can exploit anelectronicdisplaywithanemittancefunctione. Thedisplayis observedbyacamerawithsensitivitysandradiometricresponse thecontrolofpixelintensitiesonthedisplayandeasilycap- functionf. ture the full range of input intensities. However, the dis- play emittance function is typically dependent on the dis- play viewing angle. Therefore, the CDTF is dependent on tal domain, prior work in real-world camera-display mes- camerapose. Inamovingcamerasystem, theCDTFmust saging is very limited. In this paper, we develop an opti- beestimatedperframe;thatis,anonlineCDTFestimation malmethodforsendingandretrievinghiddentime-varying is needed. Furthermore, this function varies spatially over messages using electronic displays and cameras which ac- theelectronicdisplaysurface. countsforthethecharacteristicsoflightemittancefromthe We show that the two-part problem of online radiomet- display. We assume the electronic display has two simul- riccalibrationandaccuratemessageretrievalcanbestruc- taneous purposes: 1) the original display function such as tured as an optimization problem. This leads to the pri- advertising,maps,slides,orartwork;2)thetransmissionof marycontributionofthepaper. Wepresentanelegantprob- hiddentime-varyingmessages. lem formulation where the photometric modeling leads to When light is emitted from a display, the resultant 3D physically-motivated kernel functions that are used with a lightfieldhasanintensitythatdependsontheangleofob- supportvectormachineclassifier. Weshowthatcalibration servation as well as the pixel value controlled by the dis- and message bit classification can be done simultaneously play. The emittance function of the electronic display is andtheresultingoptimizationalgorithmoperatesinfourdi- analogous to the BRDF (bidirectional reflectance distribu- mensional space and is convex. The algorithm is a novel tionfunction)ofasurface. Thisfunctioncharacterizesthe methodforonlineoptimalradiometriccalibration(OORC) lightradiatingfromadisplaypixel. Ithasaparticularspec- thatenablesaccuratecamera-displaymessaging. Anexam- tralshapethatdoesnotmatchthespectralsensitivitycurve ple message recovery result is shown in Figure 3. Our ex- ofthecamera.Theeffectsofthedisplayemittancefunction, perimentalresultsshowthataccuracylevelsformessagere- thespectralsensitivityofthecameraandtheeffectofcam- covery can improve from as low as 40-60% to higher than era viewing angle are all components of our photometric 90% using our approach when compared to either no cali- model for image formation as shown in Figure 2. Our ap- bration, or sequential calibration followed message recov- proach does not require measurement or knowledge of the ery. For evaluation of results, 9 different combinations of exact display emittance function. Instead, we measure the displays and cameras are used with 15 different image se- entire system transfer function, as a camera-display trans- quences, formultipleembeddedintensityvalues, andmul- fer function (CDTF), which determines the captured pixel tiplecamera-displayviewangles. value as a function of the displayed pixel value. By using The standard problem of radiometric calibration is frame-to-framecharacterizationoftheCDTF,themethodis solvedbyvaryingexposuresothatarangeofsceneradiance independentoftheparticularchoiceofdisplayandcamera. can be measured. For CDTF estimation, textured patches Interestingly,whileouroverallgoalhasverystrongsim- areplacedwithinthedisplayimagethathaveintensityvari- ilarities to the field of watermarking and steganography, ationoverthefullrangeofdisplaybrightnessvalues. These we present results that are novel and are aligned with the patches can be placed in inconspicuous regions of the dis- goalsofcomputationalphotography. Althoughwatermark- play image or in corners. We use the term ratex patch to ingliteraturehasmanyhiddenmessagingmethods,thisarea refer to these radiometric calibration texture patches. The largelyignoresthephysicsofillumination. Display-camera ratex patches are not used as part of the hidden message. messaging is fundamentally different from watermarking Multipleratexpatchescanbeusedtofindaspatiallyvary- becauseeachpixeloftheimageisalightsourcethatpropa- ing CDTF. The ratex patches have the advantage that they gatesinfreespace. Therefore,representationsandmethods areperceptuallyacceptable,theyrepresenttheentirerange thatactonlyinthedigitaldomainarenotsufficient. ofgray-scaleintensityvariation,andtheycanbedistributed The problem of understanding the relationship between spatially. Furthermore, these patches are used for support thedisplayedpixelandthecapturedpixeliscloselyrelated vectormachinetrainingasdescribedinSection4. to the area of radiometric calibration [22][6][24]. In these Additionally,weintroduceamethodofradiometriccali- methods,abrightnesstransferfunctioncharacterizesthere- brationthatemploysvisuallynon-disruptive“hiddenratex” lationship between scene radiance and image pixel values. mapping. Ratherthandirectlymeasuringtheeffectthatthe (a) Differenceimage (b) Thresholding (c) Ourmethod Figure3.Comparisonofmessagerecoverywithanaivemethodandtheproposedoptimalmethod(a)Differenceoftwoconsecutiveframes inthecapturedsequencetorevealthetransmittedmessage. (b)Naivemethod: Thresholdthedifferenceimagebyaconstant(threshold T = 5forthisexample). (c)OptimalMethod: Bitsareclassifiedbyasimultaneousradiometriccalibrationandsupportvectormachine classifier. CDTFhasonknownintensityvalues,weareabletomodel robustness to geometric changes during the imaging pro- theCDTFbasedonchangestoaknownfrequencydistribu- cesssuchasscaling,rotations,translationsandgeneralho- tion of intensity values. Radiometric calibration with hid- mography transformations [7] [29] [8] [34] [19] [28] [30]. den ratex produces a distribution-driven intensity mapping However, the photometry of imaging has largely been ig- thatmitigatesthephotometriceffectsoftheCDTFforsim- nored. Therarementionofphotometriceffects[40][37]in plemessagerecovery. thewatermarkingliteraturedoesn’tdefinephotometrywith Thecontributionsofthepapercanbesummarizedasfol- respecttoillumination; insteadphotometriceffectsarede- lows: 1)Anewoptimalonlineradiometriccalibrationwith finedas“lossycompression, denoising, noiseadditionand simultaneousmessagerecovery,castasaconvexoptimiza- lowpass filtering”. In fact, photometric attacks are some- tion problem; 2) photometric model of the camera display timesdefinedasjpegcompression[8]. transferfunction;3)theuseofratexpatchestoprovidecon- tinualcalibrationinformationasapracticalmethodforon- line calibration; 4) the use of distribution-driven intensity RadiometricCalibration Ideally,weconsiderthepixel- mapping as a practical method for visually non-disruptive values in a camera image to be a measurement of light in- onlinecalibration. cidentontheimageplanesensor. Itiswellknownthatthe relationshipistypicallynonlinear. Radiometriccalibration methods have been developed to estimate the camera re- 2.RelatedWork sponsefunctionthatconvertsirradiancetopixelvalues. In Watermarking In developing a system where cameras measuringacameraresponse,aseriesofknownbrightness anddisplayscancommunicateunderrealworldconditions, valuesaremeasuredalongwiththecorrespondingpixelval- theinitialexpectationwasthatexistingwatermarkingtech- ues.Ingeneral,havingsuchgroundtruthbrightnessisquite niques could be used directly. Certainly the work in this difficult.Theclassicmethod[6]usesmultipleexposureval- fieldisextensiveandhasalonghistorywithnumeroussur- ues instead. The light intensity on the sensor is a linear veyscompiled[4][35][28][5][14][27]. Surprisingly,ex- functionofthetimeofexposure,soknownexposuretimes isting methods are not directly applicable to our problem. enables ground truth light intensity. This exposure-based Inthefieldofwatermarking, afixedimageormarkisem- method is used in several radiometric calibration methods beddedinanimageoftenwiththegoalofidentifyingfraud- [22] [24] [6] [21] [17]. Our goal for the display-camera ulentcopiesofavideo,imageordocument. Existingwork systemisrelatedtoradiometriccalibration,yetdifferentin emphasizesalmostexclusivelythedigitaldomainanddoes significantways. Weareinterestednotjustinasystemthat not account for the effect of illumination in the image for- convertssceneradiancetopixels(thecamera),butalsocon- mationprocessinrealworldscenes. Inthedigitaldomain, verts from pixel to scene radiance (the display) so that the neglecting the physics of illumination is quite reasonable; wholecamera-displaysystemisafunctionthatmapsacolor however,forcamera-displaymessaging,illuminationplays valueatthedisplaytoacolorvalueatthecamera. acentralrole. Thecameraresponseinradiometriccalibrationiseither Fromacomputervisionpointofview,theimagingpro- estimated as a full mapping where i is specified for ev- out cesscanbedividedintotwomaincomponents: photometry ery i or as an analytic function g(i ). Several authors in in and geometry. The geometric aspects of image formation [22] [3] [18] use polynomials to model the radiometric re- have been addressed to some extent in the watermarking sponsefunction. Similarly,wehavefoundthatfourthorder community, andmanytechniqueshavebeendevelopedfor polynomialscanbeusedformodelingtheinversedisplay- camera transfer function. The dependence on color is typ- analyzingfactorsthatcommonlyinfluencetheCDTF. icallymodeledbyconsideringeachchannelindependently 3.1.DisplayEmittanceVariation [22] [24] [6] [9] . Interestingly, although more complex colormodelshavebeendeveloped[16][20][36], wehave Displays vary widely in brightness, hue, white balance, found the independent channel approach suitable for the contrast and many other parameters that will influence the display-camera representation where the optimality crite- appearanceoflight.Toaffirmthishypothesis,anSLRcam- rionisaccuratemessagerecovery. era with fixed parameters observes 3 displays and models Existing radiometric calibration methods are developed the CDTF for each one. See Samsung in Fig. 4(a), LG in for cameras, not camera-display systems. Therefore, dis- Fig. 4(b), and iMac 4(c). Although each display is tuned playemittancefunctionisnotpartofthesystemtobecal- tothesameparameters,includingcontrastandRGBvalues, ibrated. However,forthecamera-displaytransferfunction, eachdisplayproducesauniqueCDTF. thiscomponentplaysanimportantrole. Wedonotusethe measured display emittance function explicitly, but since theCDTFisviewdependentandthecameracanmove,our approach is to perform radiometric calibration per frame, by the insertion of radiometric calibration patches (ratex patches). OCcfrooatmhmmeperuoratue-MrdrpiesvrtpiohslpaioooydnssecdcofooammrpmpmCruuonanaimcicthaye.t,rioaFbn-ouDsrtiesehxpxaalivasmetyipnlpgCer,eomcrmeeesdmteheaounrdtncshiciendarsitfitfoohenner Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBGr5a0y 100 150 200 250 Displayed Intensities Displayed Intensities Displayed Intensities theBokodeproject[23]presentedasystemusinganinvisi- (a) Samsung (b) LG (c) iMac blemessage,howeverthemessageisafixedsymbol,nota Figure4.VarianceofLightOutputamongDisplays. AnSLR time-varyingmessage. InvisibleQRcodeswereaddressed cameracapturedarangeofgrayscale[0,255]intensityvaluespro- in[15],buttheseQR-codesarefixed. Similarly,traditional ducedby3differentLCDs. These3CDTFcurveshighlightthe watermark approaches typically contained fixed messages. dramatic difference in the light emmitance function for different LCD-camera communications is presented in [25] with a displays,particularlytheLG. time-varyingmessage,butthecameraisinafixedposition with respect to the display. Consequently, the electronic displayisnotdetected,trackedorsegmentedfromtheback- 3.2.ObservationAngles ground.Furthermore,thetransmittedsignalisnothiddenin Displaysdonotemitlightinalldirectionswiththesame this work. Recent work has been done in high speed visi- power level. Therefore CDTF is also sensitive to observa- ble light communications [32], but this work does not uti- tion angles. To verify this hypothesis, an experiment was lizeexistingdisplaysandcamerasandrequiresspecialized performed where an SLR camera captured the light inten- hardwareandLEDdevices.Time-of-flightcamerashavere- sityproducedbyacomputerdisplayfrommultipleangles. centlybeenusedforphase-basedcommunication[39], but TheresultsinFig.5showthatmoreobliqueobservationan- thesemethodsrequirespecialhardware. Interestincamera- glesyieldlowercapturedpixelsintensities.Moreover,there display messaging is also shared in the mobile communi- isanonlinearrelationshipbetweencapturedlightintensity cations domain. COBRA, RDCode, and Strata have de- andviewingangle. veloped2Dbarcodeschemesdesignedtoaddressthechal- lenges of low-resolution and slow shutter speeds typically 4.Methods present in smartphone cameras [10] [33] [12]. Likewise, Lightsynchastargetedsynchronizationchallengeswithlow 4.1.PhotometryofDisplay-Camerasystems frequencycameras. [11]. ThecapturedimageI fromthecameraviewingtheelec- c 3.SystemProperties tronic display image Id can be modeled using the image formation pipeline in Figure 2. First, consider a particular Inourproposedcamera-displaycommunicationsystem, pixelwithinthedisplayimageI withred, blueandgreen d pixelvaluesfromthedisplayareinputs,whilecapturedin- componentsgivenbyρ=(ρ ,ρ ,ρ ). Thecapturedimage r g b tensities from the camera are output. We denote the map- I at the camera has three color components (Ir,Ig,Ib), c c c c pingfromdisplayedintensitiestocapturedonesasCamera- howeverthereisnoone-to-onecorrespondencebetweenthe DisplayTransferFunction(CDTF).Inthissection,wemo- color channels of the camera sensitivity function and the tivatetheneedforonlineradiometriccalibrationbybriefly electronic display emittance function. When the monitor sensitivityfunctionfortheredcomponent,thentheredpixel valueIr canbeexpressedas c (cid:90) Ir ∝ [ρ·e(λ,θ))]s (λ)dλ. (3) c r λ Captured Intensities11225050500 00000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Npeomefon“tiditrcteeaednn”ctcheeianftoutnthnhecewtmisaoeovnnensoliietftonitvrhgieittshymdftoihufnafneitctrtoeiisron.ntTlifokhrofeamlttyhiset,dhctiahfatfemeorifenernttahetrehptcahreasatmnaateidtroheane-. Displayed Intensities Displayed Intensities Displayed Intensities Notice that a sign of proportionality is used in Equation 3 (a) 30◦ (b) 45◦ (c) 60◦ tospecifythatthepixelvalueisalinearfunctionofthein- Figure 5. Influence of observation angles. Using the Nikon- tensityatthesensor,assumingalinearcameraanddisplay. Samsungpair,arangeofgrayscale[0,255]valuesweredisplayed ThisassumptionwillberemovedinSection4.3. and captured from a set of different observation angles. As ob- Equation 3 can be written to consider all color compo- servationanglebecamemoreoblique,thecapturedlightintensity nentsinthecapturedimageI as c sharply decreased. Therefore, observation angle has a dramatic, nonlineareffectonCDTF. (cid:90) I ∝ [ρ·e(λ,θ)]s(λ)dλ. (4) c λ wheres=(s ,s ,s ). r g b 4.2.MessageStructure The pixel value ρ is controllable by the electronic dis- 6x104 6x104 6x104 playdriver,andsoitprovidesamechanismforembedding 5 5 5 4 4 4 information.Weusetwosequentialframesinourapproach. 3 3 3 We modify the monitor intensity by adding the value κ 2 2 2 1 1 1 and transmit two consecutive images, one with the added 0 0 0 0 50Inte1n0s0ity 1V5a0lue200 250 0 50Inte1n0s0ity 1V5a0lue200 250 0 50Inte1n0s0ity 1V5a0lue200 250 value Ie and one image of original intensity Io. The re- coveredmessagedependsonthedisplayemittancefunction (a) 30◦ (b) 45◦ (c) 60◦ andcamerasensitivityfunctioniftheembeddedmessageis Figure6.Histogramsofintensitiesacrossthedisplay.Noticeas donebyaddingκasfollows: observation angle changes, so does the frequency distribution of capturedintensities.Iftheintensitydistribution(histogram)ofthe (cid:90) displayedimagewasknown,anobservercanestimatetheCDTF. Ie ∝ [(κ+ρ)·e(λ,θ)]s(λ)dλ. (5) λ Recoveryoftheembeddedsignalleadstoadifferenceequa- displaysthevalue(ρ ,ρ ,ρ )atapixel,itisemittinglight r g b tion in a manner governed by its emittance function and mod- (cid:90) ulatedbyρ. Themonitoremittancefunctioneistypically Ie−Io ∝ [(κ)·e(λ,θ)]s(λ)dλ. (6) λ afunctionoftheviewingangleθ = (θ ,φ )comprisedof v v The dependence on the properties of the display e and a polar and azimuthal component. For example, the emit- the spectral sensitivity of the camera s remains. We use tance function of an LCD monitor has a large decrease in additive-based messaging, instead of ratio-based methods, intensitywithpolarangle(seeFigure6). becausethisstructureisconvenientforconvexityoftheal- The emittance function has three components, i.e. e = gorithmasdescribedinSection4.3. (e ,e ,e ). Therefore the emitted light I as a function of r g b The main concept for message embedding is illustrated wavelengthλforagivenpixel(x,y)ontheelectronicdis- inFigure7. Inordertoconveymany“bits”perimage, we playisgivenby dividetheimageregionintoaseriesofblockcomponents. I(x,y,λ)=ρ e (λ,θ)+ρ e (λ,θ)+ρ e (λ,θ), (1) Each block can convey a bit “1” or “0”. The blocks cor- r r g g b b respondingtoa“1”containtheaddedvalueκtypicallyset or to 10 gray levels, while the zero blocks have no additive component(κ = 0). Themessageisrecoveredbysending I(x,y,λ)=ρ·e(λ,θ). (2) theoriginalframefollowedbyaframewiththeembedded Nowconsidertheintensityofthelightreceivedbyonepixel messageandusingthedifferenceformessagerecovery.The elementatthecamerasensor. Lets (λ)denotethecamera messagecanalsobeaddedtothecoarserscalesofaimage r Captured embedded image ⎡0 0 1 1 1 0 1 0⎤ Message Id(t) Ic(t) ⎢⎢1 1 0 1 0 0 1 1⎥⎥ ⎢0 1 1 0 1 0 0 1⎥ Message ⎢ ⎥ retrieval ⎢1 0 1 0 0 1 1 1⎥ Hidden Image using ⎢0 1 1 1 0 1 0 0⎥ OORC ⎢ ⎥ ⎢1 0 0 1 0 1 1 0⎥ I (t−1) I (t−1) d c ⎢0 1 1 0 0 0 1 0⎥ ⎢ ⎥ ⎢⎣1 0 0 1 0 1 1 0⎥⎦ Display Image Captured original image Figure7.MessageEmbeddingandRetrieval. Twosequentialframesaresent,anoriginalframeandaframewithanembeddedmessage image.Simpledifferencingisnotsufficientformessageretrieval.Ourmethod(OORC)isusedtorecovermessagesaccurately. pyramiddecomposition[26],inordertobetterhidethemes- Let g(i) be the inverse function that is modeled with a sagewithinthedisplayimagecontent. Thedisplaycanbe fourthorderpolynomialasfollows trackedwithexistingmethods[38]. Thismessagestructure g(i)=a i4+a i3+a i2+a i+a . (10) isdecidedlyverysimple,sothemethodspresentedherecan 4 3 2 1 0 beappliedtomanymessagecodingschemes. Consider two images frames i , where i is the original o o Whenaccountingforthenonlinearityinthecameraand frameandi theimageframewiththeembeddedmessage. e display, we rewrite Equation 4 to include the radiometric Note the use of i instead of I for notional compactness. o o responsefunctionf, Since we are using an additive message embedding, we (cid:18)(cid:90) (cid:19) wish to classify the message bits as either ones or zeros Ic =f [ρ·e(λ,θ)]s(λ)dλ . (7) basedonthedifferenceimageio −ie. Inordertoclassify λ the message bits, the ratex patches are also used for train- ing. Consecutive frames of ratex patches toggle between Moreconcisely, message bit “1” (κ = 10) and message bit “0” (κ = 0). I =f(I ), (8) c d Thistrainingdatacanbeusedforasupportvectormachine andtherecovereddisplayintensityis (SVM)classifer. Takingintoaccounttheradiometriccalibration,wewant I =f−1(I )=g(I ). (9) to classify on the recovered data g(i )−g(i ). Assuming d d d o e that the inverse function can be modeled by a fourth order We use polynomials to represent the radiometric inverse polynomial,thefunctiontobeclassifiedis function g(i). The same inverse function g is used for all g(i )−g(i )= colorchannelsandgray-scaleratexpatches. Thissimplifi- o e a (i4−i4)+a (i3−i3)+a (i2−i2)+a1(i −i ). cation of the color problem is justified by the accuracy of 4 o e 3 o e 2 o e o e (11) theempiricalresults. In Equation 11, we see that the calibration problem has 4.3. Simultaneous Radiometric Calibration and a physically motivated nonlinear mapping function. That MessageRecoveryviaConvexOptimization is, we see that the original data (i ,i ) can be placed into o e a higher dimensional space using the nonlinear mapping The two goals of message recovery and calibration can function Φ which maps from a two dimensional space to be combined to a single problem. While ideal radiometric afourdimensionalspaceasfollows calibrationwouldprovideacapturedimagethatisalinear function of the displayed image, we show that calibrating Φ(i ,i )=(cid:2) (i4−i4) (i3−i3) (i2−i2) (i −i ) (cid:3). o e o e o e o e o e followedbymessagerecoveryonlygivesarelativelysmall (12) increaseinmessageaccuracy.However,ifthetwogoalsare Inthisfourdimensionalspaceweseekaseparatinghyper- combined into a simultaneous problem we have two ben- planebetweenthetwoclasses(one-bitsandzero-bits). Our efits: 1) the problem formulation can be done in a convex experimentalresultsindicatethatthesearenotseparablein optimizationparadigmwithasingleglobalsolutionand2) lowerdimensionalspace,butthemovementtoahigherdi- theaccuracyincreasessignificantly. mensional space enables the separation. Also, the form of thathigherdimensionalspaceisphysicallymotivatedbythe foredistortionbytheCDTF.Inotherwords, histogramin- needforradiometriccalibration.Thereforeourproblembe- tensitymappingactsastheinverseCDTF.Althoughthereis comesasupportvectormachineclassifierwherethesupport notone-to-onecorrespondence,intensitymappingisanef- vectorweightsandthecalibrationparametersaresimultane- fectivemethodforhiddenratexasavisiblynon-disruptive ouslyestimated. Thatis,weestimate methodforradiometriccalibration. Because histogram-driven intensity mapping serves as wTu+b, (13) an effective inverse-CDTF mapping, embedded messages bits can be labeled with simple thresholding. For each where,w ∈R4,baretheseparatinghyperplaneparameters, pairofcorrectedimages(originalandembedded),intensity anduistheinputfeaturevector. Sincewewanttoperform mappingisappliedtotheoriginalimage. Thatsamemap- radiometriccalibration,thefour-dimensionalinputisgiven pingisthenappliedtotheembeddedmessage. Thediffer- ence between the original and carrier image are then com- u=(cid:2) a4(i4o−i4e) a3(i3o−i3e) a2(i2o−i2e) a(io−ie) (cid:3)T . puted. Theembeddedblocksarenowseparablebyasimple (14) constant threshold, because, undisrupted by the photomet- Notice that the wTu + b is still linear in the coefficients ric effects of the CDTF, message blocks are nothing more oftheinverseradiometricfunction. Thesecoefficientsand thanaknownaddedconstant. Inotherwords,i andi are e o thescalefactorw areestimatedsimultaneously. Wearrive remapped via the same intensity mapping. The remapped attheimportantobservationthataccountingfortheCDTF differencei −i isusedtorecoverthemessagebit. e o preserves the convexity of the overall classification prob- lem. The coefficients of the function g are scaled by w, 5.Results sothatcalibrationandclassificationcanbedonesimultane- ously,andconvexityoftheSVMispreserved. Forempiricalvalidation,9differentcombinationofdis- playsandcamerasareusedcomprisedof3displays: 1)LG 4.4.RadiometricCalibrationwithHiddenRatex M3204CCBA 32 inch, 2) Samsung SyncMaster 2494SW, The main disadvantage of OORC is the requirement 3) iMac (21.5 inch 2009); and 3 cameras: 1) Canon EOS that visible ratex patches must be placed on screen. Ra- Rebel XSi, 2) Nikon D70, 3) Sony DSC-RX100. Fifteen tex patches are somewhat visually obtrusive and unattrac- 8-bit display images are used. From each display image, tiveforcertainapplications. However, theyareconvenient we create a display video of 10 frames: 5 frames with the for modeling the CDTF. Instead of directly observing the originaldisplayimagesinterleavedwith5imagesofembed- effects of the CDTF on the full intensity gamut, we can ded time-varying messages. An embedded message frame observe how the CDTF modifies the intensity histogram. isfollowedbyanoriginalimageframetoprovidethetem- Forthistowork, weneedtoknowtheinitialintensitydis- poral image pair ie and io. The display image does not tribution of an image before it passes through the CDTF. change in the video, only the bits of the message frames. Weperformanintensitymappingoneveryimageentering Eachmessageframehas8×8 = 64blocksusedformes- the camera-display transfer function so the intensity his- sagebits(with5bitsusedfor ratexpatchesforcalibration togramisknown.Wecanthinkoftheknownintensitymap- andclassificationtrainingdata). Considering5displayim- pingoftheseimagesas“hiddenratex.” Oncetheimageis ages, with 5 message frames and 59 bits per frame results camera-captured, the new, modified distribution of the im- inapproximately1500messagebits. Theaccuracyforeach age’s intensities are observed. Since the intensity distribu- video is defined as the number of correctly classified bits tion is predetermined, we are able to measure the effects dividedbythetotalbitsembeddedandisaveragedoverall of the CDTF by observing the differences in the camera- testing videos. The entire test set over all display-camera capturedintensityhistogram. Forexample,wemaywishto combinationsisapproximately18,000testbits. chooseauniform,ornearuniformintensitydistributionfor There are 4 methods for embedded message recovery. camera-displaytransferimages. Byhistogramequalizinga Method 1 has no radiometric calibration, only the differ- displayed image, a receiver can infer that the distribution ence i −i is used to recover the message bit. Method 2 e o of this image’s intensities are near uniform. An intensity is calibration followed by differencing for message recov- mapping is applied to an image before it is displayed. Al- ery. Method 3 (OORC) is the optimal calibration where thoughthiswillhaveaneffectontheappearanceofthecar- both radiometric calibration and classification are done si- rierimage,werefertothismethodashiddenratexbecause multaneously. Method4iscalibrationviahiddenratexfol- it does not require markers to be displayed on screen for lowed by simple differencing for message recovery. For calibration. Once the image is captured, the photometric the first three methods, training data from pixels in the ra- effects of the CDTF has altered the image. The captured tex patches are used to train an SVM classifier. For each image is then intensity mapped again, so that its intensity of the 9 display-camera combinations, the accuracy of the histogramismoresimilartothedisplayeddistribution,be- 4 message recovery methods was tested with 2 sets of ex- perimental variables: 1) 0◦frontal camera-display view; 2) Accuracy Naive Two- OORC Hidden 45◦oblique camera-display view; and: 1) embedded mes- (%) Threshold step Ratex sageintensitydifferenceof5;2)embeddedmessageinten- Canon- 97.06 94.50 99.83 95.37 sity difference of 3. The results of these tests are can be iMac foundinTables1,2,3,and4. Canon- 87.89 99.00 99.39 99.44 LG Accuracy Naive Two- OORC Hidden Canon- 71.67 88.11 100.00 95.37 (%) Threshold step Ratex Samsung Canon- 72.94 75.67 99.17 89.63 Nikon- 91.89 93.67 96.00 96.11 iMac iMac Canon- 58.94 84.94 98.44 95.74 Nikon-LG 81.56 95.11 99.94 98.88 LG Nikon- 58.78 92.22 99.39 97.41 Canon- 48.44 64.89 99.39 89.91 Samsung Samsung Sony- 92.28 92.00 99.72 80.37 Nikon- 60.17 75.50 95.17 90.00 iMac iMac Sony-LG 77.06 96.22 100.00 91.13 Nikon-LG 49.72 73.39 99.33 94.81 Sony- 63.28 94.17 99.89 81.67 Nikon- 47.22 72.89 95.00 89.54 Samsung Samsung Average 80.16 93.89 99.35 93.71 Sony- 64.44 76.00 99.06 71.11 Table 3. Accuracy of embedded message recovery and label- iMac ing with additive difference +5 on [0,255] and captured with Sony-LG 56.11 75.61 98.56 90.93 45◦obliqueperspective. Sony- 47.50 79.11 98.89 87.80 Samsung Accuracy Naive Two- OORC Hidden Average 56.17 75.33 98.11 88.83 (%) Threshold step Ratex Canon- 95.28 96.61 99.00 95.74 Table 1. Accuracy of embedded message recovery and label- ing with additive difference +3 on [0,255] and captured with iMac 45◦obliqueperspective. Canon- 97.11 99.72 97.17 97.59 LG Canon- 97.39 97.33 98.94 94.35 Accuracy Naive Two- OORC Hidden Samsung (%) Threshold step Ratex Nikon- 98.39 99.17 99.22 96.11 Canon- 85.56 83.06 96.44 91.57 iMac iMac Nikon-LG 99.83 100.00 99.83 97.31 Canon- 86.39 90.94 98.67 94.07 Nikon- 96.33 97.44 98.56 95.74 LG Samsung Canon- 87.94 87.78 98.94 91.30 Sony- 97.72 97.00 99.94 81.67 Samsung iMac Nikon- 84.06 84.00 96.50 90.27 Sony-LG 99.39 100.00 100.00 90.74 iMac Sony- 92.50 92.33 98.06 90.28 Nikon-LG 74.67 81.44 99.94 90.09 Samsung Nikon- 77.33 86.06 98.00 91.57 Average 97.10 97.73 98.97 93.28 Samsung Table4. Accuracyofembeddedmessagerecoveryandlabeling Sony- 89.33 84.22 99.44 70.00 with additive difference +5 on [0,255] and captured at 0◦frontal iMac view. Sony-LG 87.61 95.39 99.72 80.74 Sony- 80.00 83.78 96.26 84.54 6.DiscussionandConclusion Samsung Average 83.56 86.30 98.22 87.13 The results indicate a substantial improvement of bit Table2. Accuracyofembeddedmessagerecoveryandlabeling classification in a camera-display messaging system with with additive difference +3 on [0,255] and captured at 0◦frontal our methods. We demonstrate experimental results for view. nine different camera-display combinations at frontal and obliqueviewingdirections. Weshowthatnaivethreshold- ing is a poor choice because the variation of display in- [9] M.D.GrossbergandS.K.Nayar.Whatcanbeknown tensity with camera position is ignored. Any method that abouttheradiometricresponsefromimages? InPro- embeds a message without accounting for the variation of ceedings of the 7th European Conference on Com- display intensity will degrade for non-frontal views. We puterVision-PartIV,ECCV’02,pages189–205,Lon- presenttwowaystoperformonlineradiometriccalibration. don,UK,UK,2002.Springer-Verlag. 4 The first method uses calibration information in the image [10] T.Hao, R.Zhou, andG.Xing. Cobra: colorbarcode calledratexpatches.Inthesecondapproach,thecalibration streamingforsmartphonesystems. InProceedingsof information is hidden and no patches appear in the image. the 10th international conference on Mobile systems, Our experimental results show that hidden, dynamic mes- applications,andservices,pages85–98.ACM,2012. sages can be embedded in a display image and recovered 4 robustly. We show that naive methods of message embed- ding without photometric modeling lead to failed message [11] W.Hu,H.Gu,andQ.Pu. Lightsync:Unsynchronized recovery, especially for oblique views (45◦) and small in- visual communication over screen-camera links. In Proceedings of the 19th annual international confer- tensitymessages(+3). Wepresentavisuallynon-disruptive method for radiometric calibration in the form of hidden ence on Mobile computing & networking, pages 15– ratex intensity mapping. Although the CDTF is spatially 26.ACM,2013. 4 dependent,asinglesetofcalibrationcoefficientsperframe [12] W.Hu,J.Mao,Z.Huang,Y.Xue,J.She,K.Bian,and weresufficientforhighmessageaccuracy. Theapproachis G. Shen. Strata: layered coding for scalable visual well-justifiedbytheoryandbyempiricalevaluation. communication.InProceedingsofthe20thannualin- ternationalconferenceonMobilecomputingandnet- References working,pages79–90.ACM,2014. 4 [1] A.Ashok, M.Gruteser, N.Mandayam, andK.Dana. [13] ISO/IEC18004. Inf.technology,Automaticidentifica- Characterizing multiplexing and diversity in visual tion and data capture techniques, Bar code symbol- mimo. Information Sciences and Systems (CISS), ogy,QRCode,2000. 1 pagespp.1–6,March2011. 1 [14] N.Johnson,Z.Duric,andS.Jajodia.InformationHid- [2] A. Ashok, M. Gruteser, N. Mandayam, J. Silva, ing: Steganography and Watermarking - Attacks and M. Varga, and K. Dana. Challenge: mobile optical Countermeasures. Springer,1edition,2000. 3 networks through visual mimo. MobiCom: Proceed- [15] K. Kamijo, N. Kamijo, and G. Zhang. Invisible bar- ings of the sixteenth annual international conference code with optimized error correction. In Image Pro- onMobilecomputingandnetworking,pagespp.105– cessing, 2008. ICIP 2008. 15th IEEE International 112,2010. 1 Conferenceon,pages2036–2039,oct.2008. 4 [3] A.Chakrabarti,D.Scharstein,andT.Zickler. Anem- [16] S.J.Kim, H.T.Lin, Z.Lu, S.Susstrunk, S.Lin, and piricalcameramodelforinternetcolorvision. British M. S. BrownHai. A new in-camera imaging model MachineVisionConference,2009. 3 forcolorcomputervisionanditsapplication. InIEEE [4] A.Cheddad, J.Condell, K.Curran, andP.M.Kevitt. TransactionsonPatternAnalysisandMachineIntelli- Digital image steganography: Survey and analysis of gence(TPAMI),2012. 4 currentmethods. SignalProcessing,90(3):727–752, 2010. 3 [17] S. J. Kim and M. Pollefeys. Robust radiometric cal- ibration and vignetting correction. IEEE Transac- [5] I.Cox,M.Miller,J.Bloom,andC.Honsinger.Digital tions on Pattern Analysis and Machine Intelligence, watermarking. Journal of Electronic Imaging, 2002. 30(4):562–576,april2008. 3 3 [18] J.-Y. Lee, Y. Matsushita, B. Shi, I. S. Kweon, and [6] P. Debevec and J. Malik. Recovering high dynamic K.Ikeuchi.Radiometriccalibrationbyrankminimiza- range radiance maps from photographs. ACM SIG- tion.PatternAnalysisandMachineIntelligence,IEEE GRAPH,pagespp.369–378,1997. 2,3,4 Transactionson,35(1):144–156,jan.2013. 3 [7] P.Dong,J.G.Brankov,N.P.Galatsanos,Y.Yang,and F.Davoine. Digitalwatermarkingrobusttogeometric [19] C.Y.Lin,M.Wu,J.A.Bloom,I.J.Cox,M.L.Miller, distortions. ImageProcessing,IEEETransactionson, andY.M.Lui.Rotation,scale,andtranslationresilient 14(12):2140–2150,2005. 3 watermarking for images. Image Processing, IEEE Transactionson,10(5):767–782,2001. 3 [8] J.L.Dugelay, S.Roche, C.Rey, andG.Doerr. Still- image watermarking robust to local geometric dis- [20] H.T.Lin, S.J.Kim, S.Susstrunk, andM.S.Brown. tortions. Image Processing, IEEE Transactions on, Revisiting radiometric calibration for color computer 15(9):2831–2842,2006. 3 vision. ICCV,2011. 4 [21] S.MannandR.Picard. Onbeingundigitalwithdig- screen-camera links. In Proceedings of the 20th an- italcameras: Extendingdynamicrangebycombining nual international conference on Mobile computing differently exposed pictures. Proc. IST 46th annual andnetworking,pages41–52.ACM,2014. 4 conference,pages422–428,1995. 3 [34] X.Wang, L.Hou, andJ.Wu. Afeature-basedrobust [22] T.MitsunagaandS.Nayar. Radiometricselfcalibra- digitalimagewatermarkingagainstgeometricattacks. tion. In Computer Vision and Pattern Recognition, ImageVisionComput.,26(7),2008. 3 1999. IEEE Computer Society Conference on., vol- [35] P.Wayner. DisappearingCryptography: Information ume1,pages2vol.(xxiii+637+663),1999. 2,3,4 Hiding: Steganography & Watermarking. Morgan [23] A. Mohan, G. Woo, S. Hiura, Q. Smithwick, and KaufmannPublishersInc.,3edition,2009. 3 R.Raskar.Bokode:imperceptiblevisualtagsforcam- [36] Y.Xiong,K.Saenko,T.Darrell,andT.Zickler. From erabasedinteractionfromadistance. InSIGGRAPH. pixels to physics: Probabilistic color de-rendering. ACM,2009. 4 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages pp. 358–365, [24] S. K. Nayar and T. Mitsunaga. High dynamic range 2012. 4 imaging: spatiallyvaryingpixelexposures. Proceed- ingsoftheIEEEConferenceonComputerVisionand [37] L. Yang and Z. Guo. A robust video watermarking PatternRecognition,pagespp.472–479,2000. 2,3,4 schemeresilienttospatialdesynchronizationandpho- tometric distortion. In Signal Processing, 2006 8th [25] S. D. Perli, N. Ahmed, and D. Katabi. InternationalConferenceon,volume4,16-202006.3 Pixnet:designing interference-free wireless links using lcd-camera pairs. ACM Int. Conf. on Mobile [38] W.Yuan,K.Dana,A.Ashok,M.Varga,M.Gruteser, ComputingandNetworking,2010. 4 and N. Mandayam. Photographic steganography for visual mimo: A computer vision approach. IEEE [26] J. B. Peter and E. H. Adelson. The laplacian pyra- Workshop on the Applications of Computer Vision midasacompactimagecode. IEEETransactionson (WACV),pages345–352,2012. 6 Communications,31:532–540,1983. 6 [39] W. Yuan, R. E. Howard, K. J. Dana, R. Raskar, [27] F. A. P. Petitcolas, R. J. Anderson, and M. G. Kuhn. A. Ashok, M. Gruteser, and N. Mandayam. Phase Informationhiding-asurvey.ProceedingsoftheIEEE, messagingmethodfortime-of-flightcameras.InCom- 87(7):1062–1078,jul1999. 3 putational Photography (ICCP), 2014 IEEE Interna- [28] V. M. Potdar, S. Han, and E. Chang. A survey of tionalConferenceon,pages1–8.IEEE,2014. 4 digital image watermarking techniques. IEEE Inter- [40] F.Zou,H.Ling,X.Li,Z.Xu,andP.Li. Robustimage national Conference on Industrial Informatics, pages copy detection using local invariant feature. In Mul- 709–716,2005. 3 timedia Information Networking and Security, 2009. [29] A.Sangeetha,B.Gomathy,andK.Anusudha. Awa- MINES ’09. International Conference on, volume 1, termarkingapproachtocombatgeometricattacks. In pages57–61,2009. 3 DigitalImageProcessing,2009InternationalConfer- enceon,pages381–385,2009. 3 [30] J.S..SeoandC.D.Yoo. Imagewatermarkingbased oninvariantregionsofscale-spacerepresentation.Sig- nal Processing, IEEE Transactions on, 54(4):1537 – 1549,2006. 3 [31] M. Varga, A. Ashok, M. Gruteser, N. Mandayam, W. Yuan, and K. Dana. Demo: Visual mimo- based led-camera communication applied to automo- bile safety. Proceedings of ACM/USENIX Interna- tional Conference on Mobile Systems, Applications, andServices(MobiSys),pagespp.383–384,2011. 1 [32] J. Vucic, C. Kottke, S. Nerreter, K. Langer, and J.Walewski. 513mbit/svisiblelightcommunications linkbasedondmt-modulationofawhiteled. Journal ofLightwaveTechnology,28(24):3512–3518,2010. 4 [33] A.Wang,S.Ma,C.Hu,J.Huai,C.Peng,andG.Shen. Enhancing reliability to boost the throughput over

ebook img

Optimal Radiometric Calibration for Camera-Display Communication PDF

3.8 MB·English
by  Wenjia Yuan
#journals #arxiv
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Optimal Radiometric Calibration for Camera-Display Communication

Optimal Radiometric Calibration for Camera-Display Communication WenjiaYuan EricWengrowski KristinJ.Dana AshwinAshok MarcoGruteser NarayanMandayam DepartmentofElectricalandComputerEngineering,and WINLAB(WirelessInformationNetworkLaboratory)RutgersUniversity 5 Tel.: +732-445-5253 1 ContactAuthor: [email protected] 0 2 n a Abstract Online J Radiometric Calibration 8 Unknown We present a novel method for communicating between Camera Pose acameraanddisplaybyembeddingandrecoveringhidden and ] Photometry V anddynamicinformationwithinadisplayedimage.Ahand- Displayed C heldcamerapointedatthedisplaycanreceivenotonlythe Intensity Histogram displayimage,butalsotheunderlyingmessage. Theseac- Camera-Display Transfer Function (CDTF) Preserved . s tivescenesarefundamentallydifferentfromtraditionalpas- c [ sivesceneslikeQRcodesbecauseimageformationisbased ondisplayemittance,notsurfacereflectance.Detectingand 1 Message Embedding v decodingthemessagerequirescarefulphotometricmodel- Intensity Histogram Warped by CDTF Message Recovery 4 ing for computational message recovery. Unlike standard 4 watermarking and steganography methods that lie outside Figure 1. The above flowchart illustrates the process by which 7 the domain of computer vision, our message recovery al- 1 gorithmusesilluminationtoopticallycommunicatehidden Online Radiometric Calibration is used to estimate and negate 0 the light-altering effects of the Camera-Display Transfer Func- messages in real world scenes. The key innovation of our . tion (CDTF) in camera-display communication. Variables such 1 approach is an algorithm that performs simultaneous ra- as camera pose, photometry, and hardware all have a significant 0 diometric calibration and message recovery in one convex effectonlightsignalspassingfromelectronicdisplaytocamera. 5 optimization problem. By modeling the photometry of the Ineachpairofintensityhistogramsshownabove, theleftrepre- 1 : system using a camera-display transfer function (CDTF), sentsanimage’shistogrambeforepassingthroughtheCDTF,and v wederiveaphysics-basedkernelfunctionforsupportvec- therightrepresentsthehistogramaftertheCDTF.OnlineRadio- i X tormachineclassification.Wedemonstratethatourmethod metricCalibrationmitigatesthedistortingeffectsoftheCDTFto ofoptimalonlineradiometriccalibration(OORC)leadsto preservetheimage’shistogram,enablingmoreaccurateimagere- r a anefficientandrobustalgorithmforcomputationalmessag- covery. ingbetweenninecommercialcamerasanddisplays. simpleexamplesbecausethebold,staticpatternmakesde- 1.Introduction tectionsomewhattrivial. Theproblemismorechallenging from a computer vision point of view when the codes are While traditional computer vision concentrates on ob- notvisiblemarkers,butratherarehiddenwithinadisplayed jectsthatreflectenvironmentlighting(passivescenes),ob- image. The displayed image is a light field, and decoding jects which emit light, such as electronic displays, are themessageisaninterestingprobleminphotometricmod- increasingly common in modern scenes. Unlike passive eling and computational photography. The paradigm has scenes, activescenescanhaveintentionalinformationthat numerous applications because the electronic display and mustbedetectedandrecovered.Forexample,displayswith the camera can act as a communication channel where the QRcodes[13]canbefoundinnumerouslocationssuchas displaypixelsaretransmittersandthecamerapixelsarere- shopwindowsandbillboards. However,QR-codesarevery ceivers [2][1][31]. Unlike hidden messaging in the digi- 1 Thecharacterizationofthisfunctionisdonebymeasuring I Display Light in Free Camera I d Space c a range of scene radiances and the corresponding capture images pixels. Our problem in camera-display messaging Emmitance Sensitivity Radiometric is similar but has important key differences. The CDTF Function Function Response e((cid:79),(cid:84)) s((cid:79)) f(i) is more complex than standard radiometric calibration be- cause the system consists of both a display and a camera, Figure2.ImageFormationPipeline:TheimageI isdisplayedby d each device adding its own nonlinearities. We can exploit anelectronicdisplaywithanemittancefunctione. Thedisplayis observedbyacamerawithsensitivitysandradiometricresponse thecontrolofpixelintensitiesonthedisplayandeasilycap- functionf. ture the full range of input intensities. However, the dis- play emittance function is typically dependent on the dis- play viewing angle. Therefore, the CDTF is dependent on tal domain, prior work in real-world camera-display mes- camerapose. Inamovingcamerasystem, theCDTFmust saging is very limited. In this paper, we develop an opti- beestimatedperframe;thatis,anonlineCDTFestimation malmethodforsendingandretrievinghiddentime-varying is needed. Furthermore, this function varies spatially over messages using electronic displays and cameras which ac- theelectronicdisplaysurface. countsforthethecharacteristicsoflightemittancefromthe We show that the two-part problem of online radiomet- display. We assume the electronic display has two simul- riccalibrationandaccuratemessageretrievalcanbestruc- taneous purposes: 1) the original display function such as tured as an optimization problem. This leads to the pri- advertising,maps,slides,orartwork;2)thetransmissionof marycontributionofthepaper. Wepresentanelegantprob- hiddentime-varyingmessages. lem formulation where the photometric modeling leads to When light is emitted from a display, the resultant 3D physically-motivated kernel functions that are used with a lightfieldhasanintensitythatdependsontheangleofob- supportvectormachineclassifier. Weshowthatcalibration servation as well as the pixel value controlled by the dis- and message bit classification can be done simultaneously play. The emittance function of the electronic display is andtheresultingoptimizationalgorithmoperatesinfourdi- analogous to the BRDF (bidirectional reflectance distribu- mensional space and is convex. The algorithm is a novel tionfunction)ofasurface. Thisfunctioncharacterizesthe methodforonlineoptimalradiometriccalibration(OORC) lightradiatingfromadisplaypixel. Ithasaparticularspec- thatenablesaccuratecamera-displaymessaging. Anexam- tralshapethatdoesnotmatchthespectralsensitivitycurve ple message recovery result is shown in Figure 3. Our ex- ofthecamera.Theeffectsofthedisplayemittancefunction, perimentalresultsshowthataccuracylevelsformessagere- thespectralsensitivityofthecameraandtheeffectofcam- covery can improve from as low as 40-60% to higher than era viewing angle are all components of our photometric 90% using our approach when compared to either no cali- model for image formation as shown in Figure 2. Our ap- bration, or sequential calibration followed message recov- proach does not require measurement or knowledge of the ery. For evaluation of results, 9 different combinations of exact display emittance function. Instead, we measure the displays and cameras are used with 15 different image se- entire system transfer function, as a camera-display trans- quences, formultipleembeddedintensityvalues, andmul- fer function (CDTF), which determines the captured pixel tiplecamera-displayviewangles. value as a function of the displayed pixel value. By using The standard problem of radiometric calibration is frame-to-framecharacterizationoftheCDTF,themethodis solvedbyvaryingexposuresothatarangeofsceneradiance independentoftheparticularchoiceofdisplayandcamera. can be measured. For CDTF estimation, textured patches Interestingly,whileouroverallgoalhasverystrongsim- areplacedwithinthedisplayimagethathaveintensityvari- ilarities to the field of watermarking and steganography, ationoverthefullrangeofdisplaybrightnessvalues. These we present results that are novel and are aligned with the patches can be placed in inconspicuous regions of the dis- goalsofcomputationalphotography. Althoughwatermark- play image or in corners. We use the term ratex patch to ingliteraturehasmanyhiddenmessagingmethods,thisarea refer to these radiometric calibration texture patches. The largelyignoresthephysicsofillumination. Display-camera ratex patches are not used as part of the hidden message. messaging is fundamentally different from watermarking Multipleratexpatchescanbeusedtofindaspatiallyvary- becauseeachpixeloftheimageisalightsourcethatpropa- ing CDTF. The ratex patches have the advantage that they gatesinfreespace. Therefore,representationsandmethods areperceptuallyacceptable,theyrepresenttheentirerange thatactonlyinthedigitaldomainarenotsufficient. ofgray-scaleintensityvariation,andtheycanbedistributed The problem of understanding the relationship between spatially. Furthermore, these patches are used for support thedisplayedpixelandthecapturedpixeliscloselyrelated vectormachinetrainingasdescribedinSection4. to the area of radiometric calibration [22][6][24]. In these Additionally,weintroduceamethodofradiometriccali- methods,abrightnesstransferfunctioncharacterizesthere- brationthatemploysvisuallynon-disruptive“hiddenratex” lationship between scene radiance and image pixel values. mapping. Ratherthandirectlymeasuringtheeffectthatthe (a) Differenceimage (b) Thresholding (c) Ourmethod Figure3.Comparisonofmessagerecoverywithanaivemethodandtheproposedoptimalmethod(a)Differenceoftwoconsecutiveframes inthecapturedsequencetorevealthetransmittedmessage. (b)Naivemethod: Thresholdthedifferenceimagebyaconstant(threshold T = 5forthisexample). (c)OptimalMethod: Bitsareclassifiedbyasimultaneousradiometriccalibrationandsupportvectormachine classifier. CDTFhasonknownintensityvalues,weareabletomodel robustness to geometric changes during the imaging pro- theCDTFbasedonchangestoaknownfrequencydistribu- cesssuchasscaling,rotations,translationsandgeneralho- tion of intensity values. Radiometric calibration with hid- mography transformations [7] [29] [8] [34] [19] [28] [30]. den ratex produces a distribution-driven intensity mapping However, the photometry of imaging has largely been ig- thatmitigatesthephotometriceffectsoftheCDTFforsim- nored. Therarementionofphotometriceffects[40][37]in plemessagerecovery. thewatermarkingliteraturedoesn’tdefinephotometrywith Thecontributionsofthepapercanbesummarizedasfol- respecttoillumination; insteadphotometriceffectsarede- lows: 1)Anewoptimalonlineradiometriccalibrationwith finedas“lossycompression, denoising, noiseadditionand simultaneousmessagerecovery,castasaconvexoptimiza- lowpass filtering”. In fact, photometric attacks are some- tion problem; 2) photometric model of the camera display timesdefinedasjpegcompression[8]. transferfunction;3)theuseofratexpatchestoprovidecon- tinualcalibrationinformationasapracticalmethodforon- line calibration; 4) the use of distribution-driven intensity RadiometricCalibration Ideally,weconsiderthepixel- mapping as a practical method for visually non-disruptive values in a camera image to be a measurement of light in- onlinecalibration. cidentontheimageplanesensor. Itiswellknownthatthe relationshipistypicallynonlinear. Radiometriccalibration methods have been developed to estimate the camera re- 2.RelatedWork sponsefunctionthatconvertsirradiancetopixelvalues. In Watermarking In developing a system where cameras measuringacameraresponse,aseriesofknownbrightness anddisplayscancommunicateunderrealworldconditions, valuesaremeasuredalongwiththecorrespondingpixelval- theinitialexpectationwasthatexistingwatermarkingtech- ues.Ingeneral,havingsuchgroundtruthbrightnessisquite niques could be used directly. Certainly the work in this difficult.Theclassicmethod[6]usesmultipleexposureval- fieldisextensiveandhasalonghistorywithnumeroussur- ues instead. The light intensity on the sensor is a linear veyscompiled[4][35][28][5][14][27]. Surprisingly,ex- functionofthetimeofexposure,soknownexposuretimes isting methods are not directly applicable to our problem. enables ground truth light intensity. This exposure-based Inthefieldofwatermarking, afixedimageormarkisem- method is used in several radiometric calibration methods beddedinanimageoftenwiththegoalofidentifyingfraud- [22] [24] [6] [21] [17]. Our goal for the display-camera ulentcopiesofavideo,imageordocument. Existingwork systemisrelatedtoradiometriccalibration,yetdifferentin emphasizesalmostexclusivelythedigitaldomainanddoes significantways. Weareinterestednotjustinasystemthat not account for the effect of illumination in the image for- convertssceneradiancetopixels(thecamera),butalsocon- mationprocessinrealworldscenes. Inthedigitaldomain, verts from pixel to scene radiance (the display) so that the neglecting the physics of illumination is quite reasonable; wholecamera-displaysystemisafunctionthatmapsacolor however,forcamera-displaymessaging,illuminationplays valueatthedisplaytoacolorvalueatthecamera. acentralrole. Thecameraresponseinradiometriccalibrationiseither Fromacomputervisionpointofview,theimagingpro- estimated as a full mapping where i is specified for ev- out cesscanbedividedintotwomaincomponents: photometry ery i or as an analytic function g(i ). Several authors in in and geometry. The geometric aspects of image formation [22] [3] [18] use polynomials to model the radiometric re- have been addressed to some extent in the watermarking sponsefunction. Similarly,wehavefoundthatfourthorder community, andmanytechniqueshavebeendevelopedfor polynomialscanbeusedformodelingtheinversedisplay- camera transfer function. The dependence on color is typ- analyzingfactorsthatcommonlyinfluencetheCDTF. icallymodeledbyconsideringeachchannelindependently 3.1.DisplayEmittanceVariation [22] [24] [6] [9] . Interestingly, although more complex colormodelshavebeendeveloped[16][20][36], wehave Displays vary widely in brightness, hue, white balance, found the independent channel approach suitable for the contrast and many other parameters that will influence the display-camera representation where the optimality crite- appearanceoflight.Toaffirmthishypothesis,anSLRcam- rionisaccuratemessagerecovery. era with fixed parameters observes 3 displays and models Existing radiometric calibration methods are developed the CDTF for each one. See Samsung in Fig. 4(a), LG in for cameras, not camera-display systems. Therefore, dis- Fig. 4(b), and iMac 4(c). Although each display is tuned playemittancefunctionisnotpartofthesystemtobecal- tothesameparameters,includingcontrastandRGBvalues, ibrated. However,forthecamera-displaytransferfunction, eachdisplayproducesauniqueCDTF. thiscomponentplaysanimportantrole. Wedonotusethe measured display emittance function explicitly, but since theCDTFisviewdependentandthecameracanmove,our approach is to perform radiometric calibration per frame, by the insertion of radiometric calibration patches (ratex patches). OCcfrooatmhmmeperuoratue-MrdrpiesvrtpiohslpaioooydnssecdcofooammrpmpmCruuonanaimcicthaye.t,rioaFbn-ouDsrtiesehxpxaalivasmetyipnlpgCer,eomcrmeeesdmteheaounrdtncshiciendarsitfitfoohenner Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBGr5a0y 100 150 200 250 Displayed Intensities Displayed Intensities Displayed Intensities theBokodeproject[23]presentedasystemusinganinvisi- (a) Samsung (b) LG (c) iMac blemessage,howeverthemessageisafixedsymbol,nota Figure4.VarianceofLightOutputamongDisplays. AnSLR time-varyingmessage. InvisibleQRcodeswereaddressed cameracapturedarangeofgrayscale[0,255]intensityvaluespro- in[15],buttheseQR-codesarefixed. Similarly,traditional ducedby3differentLCDs. These3CDTFcurveshighlightthe watermark approaches typically contained fixed messages. dramatic difference in the light emmitance function for different LCD-camera communications is presented in [25] with a displays,particularlytheLG. time-varyingmessage,butthecameraisinafixedposition with respect to the display. Consequently, the electronic displayisnotdetected,trackedorsegmentedfromtheback- 3.2.ObservationAngles ground.Furthermore,thetransmittedsignalisnothiddenin Displaysdonotemitlightinalldirectionswiththesame this work. Recent work has been done in high speed visi- power level. Therefore CDTF is also sensitive to observa- ble light communications [32], but this work does not uti- tion angles. To verify this hypothesis, an experiment was lizeexistingdisplaysandcamerasandrequiresspecialized performed where an SLR camera captured the light inten- hardwareandLEDdevices.Time-of-flightcamerashavere- sityproducedbyacomputerdisplayfrommultipleangles. centlybeenusedforphase-basedcommunication[39], but TheresultsinFig.5showthatmoreobliqueobservationan- thesemethodsrequirespecialhardware. Interestincamera- glesyieldlowercapturedpixelsintensities.Moreover,there display messaging is also shared in the mobile communi- isanonlinearrelationshipbetweencapturedlightintensity cations domain. COBRA, RDCode, and Strata have de- andviewingangle. veloped2Dbarcodeschemesdesignedtoaddressthechal- lenges of low-resolution and slow shutter speeds typically 4.Methods present in smartphone cameras [10] [33] [12]. Likewise, Lightsynchastargetedsynchronizationchallengeswithlow 4.1.PhotometryofDisplay-Camerasystems frequencycameras. [11]. ThecapturedimageI fromthecameraviewingtheelec- c 3.SystemProperties tronic display image Id can be modeled using the image formation pipeline in Figure 2. First, consider a particular Inourproposedcamera-displaycommunicationsystem, pixelwithinthedisplayimageI withred, blueandgreen d pixelvaluesfromthedisplayareinputs,whilecapturedin- componentsgivenbyρ=(ρ ,ρ ,ρ ). Thecapturedimage r g b tensities from the camera are output. We denote the map- I at the camera has three color components (Ir,Ig,Ib), c c c c pingfromdisplayedintensitiestocapturedonesasCamera- howeverthereisnoone-to-onecorrespondencebetweenthe DisplayTransferFunction(CDTF).Inthissection,wemo- color channels of the camera sensitivity function and the tivatetheneedforonlineradiometriccalibrationbybriefly electronic display emittance function. When the monitor sensitivityfunctionfortheredcomponent,thentheredpixel valueIr canbeexpressedas c (cid:90) Ir ∝ [ρ·e(λ,θ))]s (λ)dλ. (3) c r λ Captured Intensities11225050500 00000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Captured Intensities1122505050000000 RGBG5ra0y 100 150 200 250 Npeomefon“tiditrcteeaednn”ctcheeianftoutnthnhecewtmisaoeovnnensoliietftonitvrhgieittshymdftoihufnafneitctrtoeiisron.ntTlifokhrofeamlttyhiset,dhctiahfatfemeorifenernttahetrehptcahreasatmnaateidtroheane-. Displayed Intensities Displayed Intensities Displayed Intensities Notice that a sign of proportionality is used in Equation 3 (a) 30◦ (b) 45◦ (c) 60◦ tospecifythatthepixelvalueisalinearfunctionofthein- Figure 5. Influence of observation angles. Using the Nikon- tensityatthesensor,assumingalinearcameraanddisplay. Samsungpair,arangeofgrayscale[0,255]valuesweredisplayed ThisassumptionwillberemovedinSection4.3. and captured from a set of different observation angles. As ob- Equation 3 can be written to consider all color compo- servationanglebecamemoreoblique,thecapturedlightintensity nentsinthecapturedimageI as c sharply decreased. Therefore, observation angle has a dramatic, nonlineareffectonCDTF. (cid:90) I ∝ [ρ·e(λ,θ)]s(λ)dλ. (4) c λ wheres=(s ,s ,s ). r g b 4.2.MessageStructure The pixel value ρ is controllable by the electronic dis- 6x104 6x104 6x104 playdriver,andsoitprovidesamechanismforembedding 5 5 5 4 4 4 information.Weusetwosequentialframesinourapproach. 3 3 3 We modify the monitor intensity by adding the value κ 2 2 2 1 1 1 and transmit two consecutive images, one with the added 0 0 0 0 50Inte1n0s0ity 1V5a0lue200 250 0 50Inte1n0s0ity 1V5a0lue200 250 0 50Inte1n0s0ity 1V5a0lue200 250 value Ie and one image of original intensity Io. The re- coveredmessagedependsonthedisplayemittancefunction (a) 30◦ (b) 45◦ (c) 60◦ andcamerasensitivityfunctioniftheembeddedmessageis Figure6.Histogramsofintensitiesacrossthedisplay.Noticeas donebyaddingκasfollows: observation angle changes, so does the frequency distribution of capturedintensities.Iftheintensitydistribution(histogram)ofthe (cid:90) displayedimagewasknown,anobservercanestimatetheCDTF. Ie ∝ [(κ+ρ)·e(λ,θ)]s(λ)dλ. (5) λ Recoveryoftheembeddedsignalleadstoadifferenceequa- displaysthevalue(ρ ,ρ ,ρ )atapixel,itisemittinglight r g b tion in a manner governed by its emittance function and mod- (cid:90) ulatedbyρ. Themonitoremittancefunctioneistypically Ie−Io ∝ [(κ)·e(λ,θ)]s(λ)dλ. (6) λ afunctionoftheviewingangleθ = (θ ,φ )comprisedof v v The dependence on the properties of the display e and a polar and azimuthal component. For example, the emit- the spectral sensitivity of the camera s remains. We use tance function of an LCD monitor has a large decrease in additive-based messaging, instead of ratio-based methods, intensitywithpolarangle(seeFigure6). becausethisstructureisconvenientforconvexityoftheal- The emittance function has three components, i.e. e = gorithmasdescribedinSection4.3. (e ,e ,e ). Therefore the emitted light I as a function of r g b The main concept for message embedding is illustrated wavelengthλforagivenpixel(x,y)ontheelectronicdis- inFigure7. Inordertoconveymany“bits”perimage, we playisgivenby dividetheimageregionintoaseriesofblockcomponents. I(x,y,λ)=ρ e (λ,θ)+ρ e (λ,θ)+ρ e (λ,θ), (1) Each block can convey a bit “1” or “0”. The blocks cor- r r g g b b respondingtoa“1”containtheaddedvalueκtypicallyset or to 10 gray levels, while the zero blocks have no additive component(κ = 0). Themessageisrecoveredbysending I(x,y,λ)=ρ·e(λ,θ). (2) theoriginalframefollowedbyaframewiththeembedded Nowconsidertheintensityofthelightreceivedbyonepixel messageandusingthedifferenceformessagerecovery.The elementatthecamerasensor. Lets (λ)denotethecamera messagecanalsobeaddedtothecoarserscalesofaimage r Captured embedded image ⎡0 0 1 1 1 0 1 0⎤ Message Id(t) Ic(t) ⎢⎢1 1 0 1 0 0 1 1⎥⎥ ⎢0 1 1 0 1 0 0 1⎥ Message ⎢ ⎥ retrieval ⎢1 0 1 0 0 1 1 1⎥ Hidden Image using ⎢0 1 1 1 0 1 0 0⎥ OORC ⎢ ⎥ ⎢1 0 0 1 0 1 1 0⎥ I (t−1) I (t−1) d c ⎢0 1 1 0 0 0 1 0⎥ ⎢ ⎥ ⎢⎣1 0 0 1 0 1 1 0⎥⎦ Display Image Captured original image Figure7.MessageEmbeddingandRetrieval. Twosequentialframesaresent,anoriginalframeandaframewithanembeddedmessage image.Simpledifferencingisnotsufficientformessageretrieval.Ourmethod(OORC)isusedtorecovermessagesaccurately. pyramiddecomposition[26],inordertobetterhidethemes- Let g(i) be the inverse function that is modeled with a sagewithinthedisplayimagecontent. Thedisplaycanbe fourthorderpolynomialasfollows trackedwithexistingmethods[38]. Thismessagestructure g(i)=a i4+a i3+a i2+a i+a . (10) isdecidedlyverysimple,sothemethodspresentedherecan 4 3 2 1 0 beappliedtomanymessagecodingschemes. Consider two images frames i , where i is the original o o Whenaccountingforthenonlinearityinthecameraand frameandi theimageframewiththeembeddedmessage. e display, we rewrite Equation 4 to include the radiometric Note the use of i instead of I for notional compactness. o o responsefunctionf, Since we are using an additive message embedding, we (cid:18)(cid:90) (cid:19) wish to classify the message bits as either ones or zeros Ic =f [ρ·e(λ,θ)]s(λ)dλ . (7) basedonthedifferenceimageio −ie. Inordertoclassify λ the message bits, the ratex patches are also used for train- ing. Consecutive frames of ratex patches toggle between Moreconcisely, message bit “1” (κ = 10) and message bit “0” (κ = 0). I =f(I ), (8) c d Thistrainingdatacanbeusedforasupportvectormachine andtherecovereddisplayintensityis (SVM)classifer. Takingintoaccounttheradiometriccalibration,wewant I =f−1(I )=g(I ). (9) to classify on the recovered data g(i )−g(i ). Assuming d d d o e that the inverse function can be modeled by a fourth order We use polynomials to represent the radiometric inverse polynomial,thefunctiontobeclassifiedis function g(i). The same inverse function g is used for all g(i )−g(i )= colorchannelsandgray-scaleratexpatches. Thissimplifi- o e a (i4−i4)+a (i3−i3)+a (i2−i2)+a1(i −i ). cation of the color problem is justified by the accuracy of 4 o e 3 o e 2 o e o e (11) theempiricalresults. In Equation 11, we see that the calibration problem has 4.3. Simultaneous Radiometric Calibration and a physically motivated nonlinear mapping function. That MessageRecoveryviaConvexOptimization is, we see that the original data (i ,i ) can be placed into o e a higher dimensional space using the nonlinear mapping The two goals of message recovery and calibration can function Φ which maps from a two dimensional space to be combined to a single problem. While ideal radiometric afourdimensionalspaceasfollows calibrationwouldprovideacapturedimagethatisalinear function of the displayed image, we show that calibrating Φ(i ,i )=(cid:2) (i4−i4) (i3−i3) (i2−i2) (i −i ) (cid:3). o e o e o e o e o e followedbymessagerecoveryonlygivesarelativelysmall (12) increaseinmessageaccuracy.However,ifthetwogoalsare Inthisfourdimensionalspaceweseekaseparatinghyper- combined into a simultaneous problem we have two ben- planebetweenthetwoclasses(one-bitsandzero-bits). Our efits: 1) the problem formulation can be done in a convex experimentalresultsindicatethatthesearenotseparablein optimizationparadigmwithasingleglobalsolutionand2) lowerdimensionalspace,butthemovementtoahigherdi- theaccuracyincreasessignificantly. mensional space enables the separation. Also, the form of thathigherdimensionalspaceisphysicallymotivatedbythe foredistortionbytheCDTF.Inotherwords, histogramin- needforradiometriccalibration.Thereforeourproblembe- tensitymappingactsastheinverseCDTF.Althoughthereis comesasupportvectormachineclassifierwherethesupport notone-to-onecorrespondence,intensitymappingisanef- vectorweightsandthecalibrationparametersaresimultane- fectivemethodforhiddenratexasavisiblynon-disruptive ouslyestimated. Thatis,weestimate methodforradiometriccalibration. Because histogram-driven intensity mapping serves as wTu+b, (13) an effective inverse-CDTF mapping, embedded messages bits can be labeled with simple thresholding. For each where,w ∈R4,baretheseparatinghyperplaneparameters, pairofcorrectedimages(originalandembedded),intensity anduistheinputfeaturevector. Sincewewanttoperform mappingisappliedtotheoriginalimage. Thatsamemap- radiometriccalibration,thefour-dimensionalinputisgiven pingisthenappliedtotheembeddedmessage. Thediffer- ence between the original and carrier image are then com- u=(cid:2) a4(i4o−i4e) a3(i3o−i3e) a2(i2o−i2e) a(io−ie) (cid:3)T . puted. Theembeddedblocksarenowseparablebyasimple (14) constant threshold, because, undisrupted by the photomet- Notice that the wTu + b is still linear in the coefficients ric effects of the CDTF, message blocks are nothing more oftheinverseradiometricfunction. Thesecoefficientsand thanaknownaddedconstant. Inotherwords,i andi are e o thescalefactorw areestimatedsimultaneously. Wearrive remapped via the same intensity mapping. The remapped attheimportantobservationthataccountingfortheCDTF differencei −i isusedtorecoverthemessagebit. e o preserves the convexity of the overall classification prob- lem. The coefficients of the function g are scaled by w, 5.Results sothatcalibrationandclassificationcanbedonesimultane- ously,andconvexityoftheSVMispreserved. Forempiricalvalidation,9differentcombinationofdis- playsandcamerasareusedcomprisedof3displays: 1)LG 4.4.RadiometricCalibrationwithHiddenRatex M3204CCBA 32 inch, 2) Samsung SyncMaster 2494SW, The main disadvantage of OORC is the requirement 3) iMac (21.5 inch 2009); and 3 cameras: 1) Canon EOS that visible ratex patches must be placed on screen. Ra- Rebel XSi, 2) Nikon D70, 3) Sony DSC-RX100. Fifteen tex patches are somewhat visually obtrusive and unattrac- 8-bit display images are used. From each display image, tiveforcertainapplications. However, theyareconvenient we create a display video of 10 frames: 5 frames with the for modeling the CDTF. Instead of directly observing the originaldisplayimagesinterleavedwith5imagesofembed- effects of the CDTF on the full intensity gamut, we can ded time-varying messages. An embedded message frame observe how the CDTF modifies the intensity histogram. isfollowedbyanoriginalimageframetoprovidethetem- Forthistowork, weneedtoknowtheinitialintensitydis- poral image pair ie and io. The display image does not tribution of an image before it passes through the CDTF. change in the video, only the bits of the message frames. Weperformanintensitymappingoneveryimageentering Eachmessageframehas8×8 = 64blocksusedformes- the camera-display transfer function so the intensity his- sagebits(with5bitsusedfor ratexpatchesforcalibration togramisknown.Wecanthinkoftheknownintensitymap- andclassificationtrainingdata). Considering5displayim- pingoftheseimagesas“hiddenratex.” Oncetheimageis ages, with 5 message frames and 59 bits per frame results camera-captured, the new, modified distribution of the im- inapproximately1500messagebits. Theaccuracyforeach age’s intensities are observed. Since the intensity distribu- video is defined as the number of correctly classified bits tion is predetermined, we are able to measure the effects dividedbythetotalbitsembeddedandisaveragedoverall of the CDTF by observing the differences in the camera- testing videos. The entire test set over all display-camera capturedintensityhistogram. Forexample,wemaywishto combinationsisapproximately18,000testbits. chooseauniform,ornearuniformintensitydistributionfor There are 4 methods for embedded message recovery. camera-displaytransferimages. Byhistogramequalizinga Method 1 has no radiometric calibration, only the differ- displayed image, a receiver can infer that the distribution ence i −i is used to recover the message bit. Method 2 e o of this image’s intensities are near uniform. An intensity is calibration followed by differencing for message recov- mapping is applied to an image before it is displayed. Al- ery. Method 3 (OORC) is the optimal calibration where thoughthiswillhaveaneffectontheappearanceofthecar- both radiometric calibration and classification are done si- rierimage,werefertothismethodashiddenratexbecause multaneously. Method4iscalibrationviahiddenratexfol- it does not require markers to be displayed on screen for lowed by simple differencing for message recovery. For calibration. Once the image is captured, the photometric the first three methods, training data from pixels in the ra- effects of the CDTF has altered the image. The captured tex patches are used to train an SVM classifier. For each image is then intensity mapped again, so that its intensity of the 9 display-camera combinations, the accuracy of the histogramismoresimilartothedisplayeddistribution,be- 4 message recovery methods was tested with 2 sets of ex- perimental variables: 1) 0◦frontal camera-display view; 2) Accuracy Naive Two- OORC Hidden 45◦oblique camera-display view; and: 1) embedded mes- (%) Threshold step Ratex sageintensitydifferenceof5;2)embeddedmessageinten- Canon- 97.06 94.50 99.83 95.37 sity difference of 3. The results of these tests are can be iMac foundinTables1,2,3,and4. Canon- 87.89 99.00 99.39 99.44 LG Accuracy Naive Two- OORC Hidden Canon- 71.67 88.11 100.00 95.37 (%) Threshold step Ratex Samsung Canon- 72.94 75.67 99.17 89.63 Nikon- 91.89 93.67 96.00 96.11 iMac iMac Canon- 58.94 84.94 98.44 95.74 Nikon-LG 81.56 95.11 99.94 98.88 LG Nikon- 58.78 92.22 99.39 97.41 Canon- 48.44 64.89 99.39 89.91 Samsung Samsung Sony- 92.28 92.00 99.72 80.37 Nikon- 60.17 75.50 95.17 90.00 iMac iMac Sony-LG 77.06 96.22 100.00 91.13 Nikon-LG 49.72 73.39 99.33 94.81 Sony- 63.28 94.17 99.89 81.67 Nikon- 47.22 72.89 95.00 89.54 Samsung Samsung Average 80.16 93.89 99.35 93.71 Sony- 64.44 76.00 99.06 71.11 Table 3. Accuracy of embedded message recovery and label- iMac ing with additive difference +5 on [0,255] and captured with Sony-LG 56.11 75.61 98.56 90.93 45◦obliqueperspective. Sony- 47.50 79.11 98.89 87.80 Samsung Accuracy Naive Two- OORC Hidden Average 56.17 75.33 98.11 88.83 (%) Threshold step Ratex Canon- 95.28 96.61 99.00 95.74 Table 1. Accuracy of embedded message recovery and label- ing with additive difference +3 on [0,255] and captured with iMac 45◦obliqueperspective. Canon- 97.11 99.72 97.17 97.59 LG Canon- 97.39 97.33 98.94 94.35 Accuracy Naive Two- OORC Hidden Samsung (%) Threshold step Ratex Nikon- 98.39 99.17 99.22 96.11 Canon- 85.56 83.06 96.44 91.57 iMac iMac Nikon-LG 99.83 100.00 99.83 97.31 Canon- 86.39 90.94 98.67 94.07 Nikon- 96.33 97.44 98.56 95.74 LG Samsung Canon- 87.94 87.78 98.94 91.30 Sony- 97.72 97.00 99.94 81.67 Samsung iMac Nikon- 84.06 84.00 96.50 90.27 Sony-LG 99.39 100.00 100.00 90.74 iMac Sony- 92.50 92.33 98.06 90.28 Nikon-LG 74.67 81.44 99.94 90.09 Samsung Nikon- 77.33 86.06 98.00 91.57 Average 97.10 97.73 98.97 93.28 Samsung Table4. Accuracyofembeddedmessagerecoveryandlabeling Sony- 89.33 84.22 99.44 70.00 with additive difference +5 on [0,255] and captured at 0◦frontal iMac view. Sony-LG 87.61 95.39 99.72 80.74 Sony- 80.00 83.78 96.26 84.54 6.DiscussionandConclusion Samsung Average 83.56 86.30 98.22 87.13 The results indicate a substantial improvement of bit Table2. Accuracyofembeddedmessagerecoveryandlabeling classification in a camera-display messaging system with with additive difference +3 on [0,255] and captured at 0◦frontal our methods. We demonstrate experimental results for view. nine different camera-display combinations at frontal and obliqueviewingdirections. Weshowthatnaivethreshold- ing is a poor choice because the variation of display in- [9] M.D.GrossbergandS.K.Nayar.Whatcanbeknown tensity with camera position is ignored. Any method that abouttheradiometricresponsefromimages? InPro- embeds a message without accounting for the variation of ceedings of the 7th European Conference on Com- display intensity will degrade for non-frontal views. We puterVision-PartIV,ECCV’02,pages189–205,Lon- presenttwowaystoperformonlineradiometriccalibration. don,UK,UK,2002.Springer-Verlag. 4 The first method uses calibration information in the image [10] T.Hao, R.Zhou, andG.Xing. Cobra: colorbarcode calledratexpatches.Inthesecondapproach,thecalibration streamingforsmartphonesystems. InProceedingsof information is hidden and no patches appear in the image. the 10th international conference on Mobile systems, Our experimental results show that hidden, dynamic mes- applications,andservices,pages85–98.ACM,2012. sages can be embedded in a display image and recovered 4 robustly. We show that naive methods of message embed- ding without photometric modeling lead to failed message [11] W.Hu,H.Gu,andQ.Pu. Lightsync:Unsynchronized recovery, especially for oblique views (45◦) and small in- visual communication over screen-camera links. In Proceedings of the 19th annual international confer- tensitymessages(+3). Wepresentavisuallynon-disruptive method for radiometric calibration in the form of hidden ence on Mobile computing & networking, pages 15– ratex intensity mapping. Although the CDTF is spatially 26.ACM,2013. 4 dependent,asinglesetofcalibrationcoefficientsperframe [12] W.Hu,J.Mao,Z.Huang,Y.Xue,J.She,K.Bian,and weresufficientforhighmessageaccuracy. Theapproachis G. Shen. Strata: layered coding for scalable visual well-justifiedbytheoryandbyempiricalevaluation. communication.InProceedingsofthe20thannualin- ternationalconferenceonMobilecomputingandnet- References working,pages79–90.ACM,2014. 4 [1] A.Ashok, M.Gruteser, N.Mandayam, andK.Dana. [13] ISO/IEC18004. Inf.technology,Automaticidentifica- Characterizing multiplexing and diversity in visual tion and data capture techniques, Bar code symbol- mimo. Information Sciences and Systems (CISS), ogy,QRCode,2000. 1 pagespp.1–6,March2011. 1 [14] N.Johnson,Z.Duric,andS.Jajodia.InformationHid- [2] A. Ashok, M. Gruteser, N. Mandayam, J. Silva, ing: Steganography and Watermarking - Attacks and M. Varga, and K. Dana. Challenge: mobile optical Countermeasures. Springer,1edition,2000. 3 networks through visual mimo. MobiCom: Proceed- [15] K. Kamijo, N. Kamijo, and G. Zhang. Invisible bar- ings of the sixteenth annual international conference code with optimized error correction. In Image Pro- onMobilecomputingandnetworking,pagespp.105– cessing, 2008. ICIP 2008. 15th IEEE International 112,2010. 1 Conferenceon,pages2036–2039,oct.2008. 4 [3] A.Chakrabarti,D.Scharstein,andT.Zickler. Anem- [16] S.J.Kim, H.T.Lin, Z.Lu, S.Susstrunk, S.Lin, and piricalcameramodelforinternetcolorvision. British M. S. BrownHai. A new in-camera imaging model MachineVisionConference,2009. 3 forcolorcomputervisionanditsapplication. InIEEE [4] A.Cheddad, J.Condell, K.Curran, andP.M.Kevitt. TransactionsonPatternAnalysisandMachineIntelli- Digital image steganography: Survey and analysis of gence(TPAMI),2012. 4 currentmethods. SignalProcessing,90(3):727–752, 2010. 3 [17] S. J. Kim and M. Pollefeys. Robust radiometric cal- ibration and vignetting correction. IEEE Transac- [5] I.Cox,M.Miller,J.Bloom,andC.Honsinger.Digital tions on Pattern Analysis and Machine Intelligence, watermarking. Journal of Electronic Imaging, 2002. 30(4):562–576,april2008. 3 3 [18] J.-Y. Lee, Y. Matsushita, B. Shi, I. S. Kweon, and [6] P. Debevec and J. Malik. Recovering high dynamic K.Ikeuchi.Radiometriccalibrationbyrankminimiza- range radiance maps from photographs. ACM SIG- tion.PatternAnalysisandMachineIntelligence,IEEE GRAPH,pagespp.369–378,1997. 2,3,4 Transactionson,35(1):144–156,jan.2013. 3 [7] P.Dong,J.G.Brankov,N.P.Galatsanos,Y.Yang,and F.Davoine. Digitalwatermarkingrobusttogeometric [19] C.Y.Lin,M.Wu,J.A.Bloom,I.J.Cox,M.L.Miller, distortions. ImageProcessing,IEEETransactionson, andY.M.Lui.Rotation,scale,andtranslationresilient 14(12):2140–2150,2005. 3 watermarking for images. Image Processing, IEEE Transactionson,10(5):767–782,2001. 3 [8] J.L.Dugelay, S.Roche, C.Rey, andG.Doerr. Still- image watermarking robust to local geometric dis- [20] H.T.Lin, S.J.Kim, S.Susstrunk, andM.S.Brown. tortions. Image Processing, IEEE Transactions on, Revisiting radiometric calibration for color computer 15(9):2831–2842,2006. 3 vision. ICCV,2011. 4 [21] S.MannandR.Picard. Onbeingundigitalwithdig- screen-camera links. In Proceedings of the 20th an- italcameras: Extendingdynamicrangebycombining nual international conference on Mobile computing differently exposed pictures. Proc. IST 46th annual andnetworking,pages41–52.ACM,2014. 4 conference,pages422–428,1995. 3 [34] X.Wang, L.Hou, andJ.Wu. Afeature-basedrobust [22] T.MitsunagaandS.Nayar. Radiometricselfcalibra- digitalimagewatermarkingagainstgeometricattacks. tion. In Computer Vision and Pattern Recognition, ImageVisionComput.,26(7),2008. 3 1999. IEEE Computer Society Conference on., vol- [35] P.Wayner. DisappearingCryptography: Information ume1,pages2vol.(xxiii+637+663),1999. 2,3,4 Hiding: Steganography & Watermarking. Morgan [23] A. Mohan, G. Woo, S. Hiura, Q. Smithwick, and KaufmannPublishersInc.,3edition,2009. 3 R.Raskar.Bokode:imperceptiblevisualtagsforcam- [36] Y.Xiong,K.Saenko,T.Darrell,andT.Zickler. From erabasedinteractionfromadistance. InSIGGRAPH. pixels to physics: Probabilistic color de-rendering. ACM,2009. 4 Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages pp. 358–365, [24] S. K. Nayar and T. Mitsunaga. High dynamic range 2012. 4 imaging: spatiallyvaryingpixelexposures. Proceed- ingsoftheIEEEConferenceonComputerVisionand [37] L. Yang and Z. Guo. A robust video watermarking PatternRecognition,pagespp.472–479,2000. 2,3,4 schemeresilienttospatialdesynchronizationandpho- tometric distortion. In Signal Processing, 2006 8th [25] S. D. Perli, N. Ahmed, and D. Katabi. InternationalConferenceon,volume4,16-202006.3 Pixnet:designing interference-free wireless links using lcd-camera pairs. ACM Int. Conf. on Mobile [38] W.Yuan,K.Dana,A.Ashok,M.Varga,M.Gruteser, ComputingandNetworking,2010. 4 and N. Mandayam. Photographic steganography for visual mimo: A computer vision approach. IEEE [26] J. B. Peter and E. H. Adelson. The laplacian pyra- Workshop on the Applications of Computer Vision midasacompactimagecode. IEEETransactionson (WACV),pages345–352,2012. 6 Communications,31:532–540,1983. 6 [39] W. Yuan, R. E. Howard, K. J. Dana, R. Raskar, [27] F. A. P. Petitcolas, R. J. Anderson, and M. G. Kuhn. A. Ashok, M. Gruteser, and N. Mandayam. Phase Informationhiding-asurvey.ProceedingsoftheIEEE, messagingmethodfortime-of-flightcameras.InCom- 87(7):1062–1078,jul1999. 3 putational Photography (ICCP), 2014 IEEE Interna- [28] V. M. Potdar, S. Han, and E. Chang. A survey of tionalConferenceon,pages1–8.IEEE,2014. 4 digital image watermarking techniques. IEEE Inter- [40] F.Zou,H.Ling,X.Li,Z.Xu,andP.Li. Robustimage national Conference on Industrial Informatics, pages copy detection using local invariant feature. In Mul- 709–716,2005. 3 timedia Information Networking and Security, 2009. [29] A.Sangeetha,B.Gomathy,andK.Anusudha. Awa- MINES ’09. International Conference on, volume 1, termarkingapproachtocombatgeometricattacks. In pages57–61,2009. 3 DigitalImageProcessing,2009InternationalConfer- enceon,pages381–385,2009. 3 [30] J.S..SeoandC.D.Yoo. Imagewatermarkingbased oninvariantregionsofscale-spacerepresentation.Sig- nal Processing, IEEE Transactions on, 54(4):1537 – 1549,2006. 3 [31] M. Varga, A. Ashok, M. Gruteser, N. Mandayam, W. Yuan, and K. Dana. Demo: Visual mimo- based led-camera communication applied to automo- bile safety. Proceedings of ACM/USENIX Interna- tional Conference on Mobile Systems, Applications, andServices(MobiSys),pagespp.383–384,2011. 1 [32] J. Vucic, C. Kottke, S. Nerreter, K. Langer, and J.Walewski. 513mbit/svisiblelightcommunications linkbasedondmt-modulationofawhiteled. Journal ofLightwaveTechnology,28(24):3512–3518,2010. 4 [33] A.Wang,S.Ma,C.Hu,J.Huai,C.Peng,andG.Shen. Enhancing reliability to boost the throughput over

See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users