36-examining the impact of see-through cockpits on driving perance in a mixed reality prototype

1、Adjunct Proceedings of the 9th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI 17), September 2427, 2017, Oldenburg, Germany.Examining the Impact of See-Through Cockpits on Driving Performance in a Mixed Reality PrototypeAbstractWe buil

2、t and evaluated a see-through cockpit prototype for a driving simulation in a mixed reality environment, simulat- ing an HMD-based interface. Advantages of such a system may include better driving performance, collision avoidance and situation awareness. Early results from driving line data indicate

3、 potential for improving lateral control by driving with transparent cockpits and show no difference regarding dif- ferent levels of transparency. We extended our prototype based on the results and abandoned the head-registered interface in favor of a simulation of a projection-based sys- tem target

4、ting specific car parts. We present the current prototype and discuss how it relates to existing proof of concepts and potential future real-world implementations. We plan to evaluate the latest prototype in a larger-scale study to determine its impact on lane-keeping performance. We also want to co

5、nsider impairments of the real world and plan to evaluate the system with artificially induced head tracking errors.Patrick Lindemann Technical University of Munich Munich, Germany patrick.lindemanntum.deGerhard RigollTechnical University of Munich Munich, Germany rigolltum.dePermission to make digi

6、tal or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work own

7、ed by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from .ACM.AutomotiveUI 17 Adjunct, September 2427, 2017

8、, Oldenburg, Germany ACM 978-1-4503-5151-5/17/09./10.1145/3131726.3131754Author KeywordsAugmented reality; diminished reality; mixed reality; virtual reality; AR simulation; lane keeping; driving performance.CCS Concepts Human-centered computing Mixed / augmented re- ality; Computing m

9、ethodologies Virtual reality;Work-in-Progress Monday SessionAutomotiveUI Adjunct Proceedings 17, Oldenburg, GermanyIntroduction and Related WorkConveying information to the driver fast and intuitively is an important aspect of reducing distraction, especially when engaged in complex tertiary tasks.

10、This holds true even with the ongoing progress in automated driving, since tech- nical and especially legal developments still need time be- fore full automation may become pervasive. Effective han- dling of handover situations poses an additional research problem in the meantime. In existing work,

11、communicating information to the driver has commonly been achieved via head-down displays (HDDs), head-up displays (HUDs), the instrument cluster (IC) or mobile devices like smartphones and tablets. HUDs created the prerequisites for applying augmented reality (AR) to the domain of automotive user i

12、nterfaces, but they are still restricted to a relatively small part of the drivers field-of-view (FOV). Windshield displays (WSDs) offer a somewhat larger coverage. The most flexi- bility could be provided by head-mounted displays (HMDs) which would not restrict information display to the front view

13、. However, automotive HMD interfaces remain an underex- plored field for now due to still inconvenient hardware and technical restrictions. We anticipate increased capabilities for AR devices in the future and want to evaluate potential interfaces ahead of time by creating AR simulations in a mixed

14、reality (MR) environment.Potential BenefitsExamples & ScenariosImprove driving performance Lane-keeping Narrow-space parking Side barriers, curbs Road holes, piercing items Blind spot obstaclesCollision avoidance in close quartersHeighten situation awareness During eyes-off-road phases Handover situ

15、ationsTable 1: Possible advantages for drivers of see-through cockpits.Figure 1: The first transparent cockpit prototype running in a4-sided CAVE-likeenvironment. The see-through circle is head- aligned (here: full transparency; small diameter). Tires remain visible as a point of reference.performan

16、ce and general situation awareness and decision making (see Table 1). We discuss early results from a first prototype, consequential changes to our simulation and our plans for upcoming evaluations.Although DR has been applied in the automotive domain, previous research is usually focusing on matter

17、s of tech- nical realization, not actual interaction. The most common application is diminishing preceding cars in order to give the driver an unobstructed front view and benefit foresighted driving. Existing work here tries to realize this with the help of multiple video sources in inter-car commun

18、ication and via inter-video mapping 1, color markers 2 or by inter-car pose estimation 6. Beside other traffic participants, it may also be advantageous to diminish the surrounding environ- ment. 10 used rendered videos to evaluate a wall see- through system for drivers for the scenario of approachi

19、ng obstructed crossings with danger of side collisions. Results showed that a simplified visualization of obstructed cars as a moving circle is sufficient for collision estimation.In this work, we present an application of diminished real- ity (DR) in the form of a transparent cockpit simulation. It

20、 allows the driver to see otherwise occluded parts of the sur- roundings by diminishing specific parts of the own car. DR can be regarded as the opposite of AR in that information is taken away from reality rather than added to it. Both em- ployed together form a mediated reality 4, which can be put

21、 at the center of the Reality-Virtuality Continuum 5. In our opinion, the transparent cockpit could provide the driver with multiple benefits regarding collision avoidance, drivingFigure 2: Current version ofour prototype on a test track for upcoming user studies. The left guardrail is visible throu

22、gh the simulated projection on the door and A-pillar, giving the driver an additional cue for lane keeping and collision avoidance.Our proposed prototype aims at diminishing the drivers own car, giving him the impression of a see-through cock-84Work-in-Progress Monday SessionAutomotiveUI Adjunct Pro

23、ceedings 17, Oldenburg, Germanypit. We assume that advancements in AR technology may lead to such an interface becoming feasible in the future, either via HMDs, projection or tailored OLED screens. We do not focus on potential real-world implementations but on ways to evaluate how drivers interact w

24、ith the system and its impact on driving. However, we also try to consider pos- sible impairments of actual real-world prototypes. The basic requirements would be outside-mounted cameras provid- ing a combinable image of the environment, tracking of the drivers head for perspectively correct project

25、ion and a suit- able output device. Actual realizations have been shown with head-mounted projection 11 and with stationary pro- jectors 8, mainly to show the capabilities of retro-reflective materials. Sasai et al. superimposed wheel trajectory vi- sualizations on a see-through dashboard projection

26、 7 for autonomous car passengers to evaluate anxiety and stress. Entirely diminishing the cockpit would currently only be fea- sible via video see-through and telexistence 9.delimited by guardrails as is visible in figure 1. It was 500m long and consisted of straight and curve segments, which should

27、 resemble the situation in freeway construction site areas (3 gives a full description of the setup and subjective results). Participants preferred the large-sized see-through over the small one in terms of comfort and helpfulness in navigating the track securely. The more rapid movements of the sma

28、ll-sized see-through area due to the head-alignment proved to be irritating. There was no clear preference be- tween semi and full transparency 3.Lateral Deviation Resultsm0.40.3Despite the small group of participants, we recorded and later analyzed driving line data in order to get a general pic- t

29、ure of the prototypes impact on driving performance from descriptive statistics. For the reference line, we used a sim- plified ideal line defined by the center point between the two track delimiters. For every run, we determined the mean de- viation from the ideal line and the corresponding standar

30、d deviation. Each participant executed 12 runs consisting of 4 cycles as follows: a baseline run without transparent cockpit followed by 2 runs with different see-through conditions. To reduce the impact of learning effects in our results, we then regarded only the difference of the ideal line devia

31、tion val- ues between a transparent cockpit run and the users most recent baseline run. The resulting values are aggregated in figure 0.0-0.1First Prototype and Preliminary ResultsOur prototype is part of a custom driving simulation devel- oped for deployment in a CAVE-like virtual environme

32、nt (VE) with head tracking, enabling off-center stereoscopic projection. Apart from some real objects in the environ- ment (seat, wheel, pedals and wheel stand), the VE is fully visible which makes it a suitable setting for testing AR/DR simulations affecting the entire cockpit. Our first prototype

33、(see Figure 1) was showing a circular see-through patch aligning with the drivers head pose. We were using two lev- els of transparency (semi resp. 50% and fully transparent) and two sizes of the see-through effect covering roughly the area of one front tire (small) or both front tires (large). To g

34、ain first insights, we conducted an informal pre-studywith 8 participants (3 female, aged 19-39, M = 27.38, = 5.83) driving with constant 40kph (ca. 25mph) under the different see-through conditions along a narrow track-0.2Full TP Semi-TP Full TP Semi-TP LargeLargeSmallSmallSee-Through Effect Condit

35、ionsFigure 3: Lateral deviation values for all see-through conditions, calculated as difference to most recent individual baseline. Positive mean values indicate runs with better overall adherence to the ideal line. Positive SD values indicate a smaller spread of lateral performance in see-throughru

36、ns.In general, the majority of runs with the transparent cock- pit prototype resulted in a better overall adherence to the reference line (mean difference). However, results seemed very dependent on individual aptitude. Two drivers had an improvement over baseline in every run they did with the tran

37、sparent cockpit, while two of the others deteriorated in total. The remaining drivers had mixed results with positive total values. Although there were still too many runs with worse performance than the respective baselines, the spec-85y 0: Improvement y 0: DeteriorationLateral Deviation Mean Diffe

38、rence (Baseline - X)Lateral Deviation SD Difference (Baseline - X)Work-in-Progress Monday SessionAutomotiveUI Adjunct Proceedings 17, Oldenburg, Germanytrum of mean values appears promising. There seems to be a higher potential for improvements, which we want to explore more thoroughly in upcoming l

39、arger-scale studies with an improved prototype.The latest prototype is an in-car projection simulation, i.e. one or more virtual projectors can be placed in arbitrary positions inside the vehicle. They throw images of the sur- roundings (provided by virtual cameras) onto specific parts of the interi

40、or, which can be selected and combined arbi- trarily. For upcoming evaluations, we plan to use A-pillars and doors as projection targets as shown in figures 2, 4 and 5. Projector settings can be adjusted in terms of bright- ness, contrast and base coloring. To take more potential impairments of a re

41、al-world setup into account, we also included a simulation of head tracking errors that allows ad- justing accuracy, precision and delay. This results in visible discrepancies between projection and outside environment (see Figure 4). How this influences user experience and the transparent cockpits

42、capabilities for improving performance is an aspect we want to examine in our future work.While the sample size does not allow statistical inference, some basic observations may still influence future design decisions. Deviation data does not seem in line with par-ticipants personal preferences rega

43、rding see-through size.We believe that although the high amount of movement of a small head-aligned see-through spot irritated partici-pants on the one hand (as shown by subjective results 3), it also forced them into being more concentrated or intense in handling the driving task. Since this would

44、still be an un- desirable compromise, we abandoned head-alignment for the next version. The similarity in interquartile range be- tween the two transparency levels within each see-through size condition seems supportive of participants personal opinions. We regard this as a positive sign for future

45、real- world setups, since most realizations would be technically restricted to a semi-transparent effect anyway. For further prototype versions, we mainly work with the more realistic case of semi-transparent see-through effects.Figure 4: Screenshot of latest prototype. Projection is simulated with

46、artificial head tracking error and non-standard projector image settings, leading to visible discrepancies with surroundings. Best viewed in color.Our next efforts will at first be targeted on evaluating the prototypes impact on lane-keeping and driving alongside barriers, thus giving insight into w

47、hether the transparent cockpit can improve performance and collision avoidance. To this end, we created a track environment (see Figure 2) on which we are able to record relevant data (including lat- eral position/deviation from ideal line, lane exceedances, time to line crossing, steering data and

48、barrier collisions). We will be able to distinguish between straights and differ- ent types of corners when examining the data. Further eval- uations will focus on the impact of the previously mentioned tracking error simulation as well as the question how the prototype might help when being engaged

49、 in tertiary tasks,i.e. during eyes-off-road phases. We will also examine howaugmented overlays (e.g. visualized wheel trajectories) may support the effectiveness of the transparent cockpit. In the long run, we also want to apply the prototype in other sce- narios like narrow-space parking, for hand

50、over support or in an automated driving setting.Current Prototype and Future WorkWe take the preliminary results as an indicator that there may be good potential for the interface to improve driving performance. However, the strong differences between indi- vidual drivers suggest that usage of the t

51、ransparent cockpit needs to be more natural or intuitive and with less poten- tial irritations. Since head-aligned small-sized see-throughs (HMD simulation) proved to be too irritating, we abandoned head-alignment in favor of a vehicle-fixed version. This should also provide a setup closer to curren

52、tly conceivable real-world implementations 8, simulating projection-based interfaces rather than HMD interfaces.Figure 5: Latest prototype with driver looking to the right, using the transparent cockpit to look into a side-street ahead of time. Here, perspective is perfectly aligned to the drivers v

53、iew.86Work-in-Progress Monday SessionAutomotiveUI Adjunct Proceedings 17, Oldenburg, GermanyREFERENCES7.Shota Sasai, Itaru Kitahara, Yoshinari Kameda, Yuichi Ohta, Masayuki Kanbara, Yoichi Morales, Norimichi Ukita, Norihiro Hagita, Tetsushi Ikeda, and Kazuhiko Shinozawa. 2015. MR Visualization of Wh

54、eel Trajectories of Driving Vehicle by Seeing-Through Dashboard. In 2015 IEEE International Symposium on Mixed and Augmented Reality workshops (ISMARW), Takeshi Oishi (Ed.). IEEE, Piscataway, NJ, 4046.S. Tachi, M. Inami, and Y. Uema. 2014. The transparent cockpit. IEEE Spectrum 51, 11 (2014), 5256.Takura Yanagi, Charith Lasantha Fernando, M. H.D. Yamen Saraiji, Kouta Minamizawa, Susumu Tachi, and Norimasa Kishi. 2015. Transparent cockpit using t