智慧型家庭網(wǎng)路之技術(shù)與應(yīng)用ppt課件

1、智慧型家庭網(wǎng)路之技術(shù)與應(yīng)用智慧型家庭網(wǎng)路之技術(shù)與應(yīng)用Professor Yau-Hwang KuoDirectorCenter for Research of E-life Digital Technology(CREDIT)National Cheng Kung UniversityTainan, Taiwan OutlinelIntroductionlStructure of Smart Home NetworklRealization of Device & Network LayerslAgent-based Platform lAffective HCIlIntegrated

2、 PerceptionlCognition LayerlSmart Home ServiceslConclusionTrend of Digital HomelHouse_n (MIT)、Aware Home (Geogria Tech.)、Interactive Workspace (Stanford Univ.)、MavHome (UTA)。lDigital Home Working Group: HP, Intel, IBM,.lECHONET: Energy Conversation and Homecare Network.lCELF: Consumer Electronic Lin

3、ux Forum.lOSGi: Open Service Gateway Initiativel Easy Living: MicrosoftScenarios of Digital Lifesmart digital housekeeper.ubiquitous digital nursing agent.affective digital tutor.ubiquitous home security monitor.ubiquitous home content service.universal cyber circles.ubiquitous universal messaging s

4、ervice.personal knowledge warehouse/navigation.nomadic personal digital secretary.secure traffic navigator.Microsofts View for Digital Home SolutionlTotal connectivitylNo more islands of functionalitylPersonalized experienceslCustomized entertainment, communications, and controllUbiquitous accesslYo

5、ur PCs, devices, and content,securely accessible everywhere Microsofts View for Digital Home SolutionlTechnology “by invitation only, not imposedlHighly personal and personalized spacelVirtually random, unmanaged “build outlComplex mix of products and servicesIssues of Digital Homel人機(jī)互動能否人性化？lrobust

6、ness、adaptability、multi-modal collaboration 人性化互動特質(zhì)。l感官、認(rèn)知、情緒、協(xié)調(diào)、協(xié)作實現(xiàn)人性化互動的技術(shù)要素。lubiquitous multi-modal affective human-machine interaction 數(shù)位家庭的人性化互動需求。 Issues of Digital Home (cont.)l人際互動能否得到提昇擴(kuò)大？l去空間限制、去時間限制、去工具限制、去平安限制。l家電間的協(xié)力協(xié)作才干能否得到提昇?lconnectivity among appliances、autonomous collaboration of

7、appliances、interoperability of appliances。Issues of Digital Home (cont.)l人在數(shù)位生活空間的自在度能否得到提昇？l可移動性、可轉(zhuǎn)移性、可調(diào)整性。lubiquitous integration home network、location-awareness、universal access、multi-modal human-machine interaction Issues of Digital Home (cont.)l人在數(shù)位生活空間的便利度能否得到提昇？l生活機(jī)能完好性、設(shè)備與網(wǎng)路無縫結(jié)合度、生活機(jī)能可獲性(ava

8、ilability)、用戶干預(yù)度、操作易度、穩(wěn)私與平安等。 l人在數(shù)位生活空間所獲得的生活輔助機(jī)能能否得到提昇? l Smart home network is necessary!Goals:Infrastructure & applicationslCreate a new life space supported by a smart home service network and attached digital appliances.lDevelop e-services over the smart home network and digital appliances

9、to realize a new life style.lDevelop a service modeling and execution environment over the smart home network to realize various e-services.Goals:technologieslDevelop nomadic HCI technologylSpeech, vision, physiology, sensors.lDevelop affective HCI technologylDevelop agent-based home service network

10、 middleware.lDevelop embedded platform & SoC for smart appliances.Applications (health care, entertainment, surveillance, etc.)Application LayerSpeechVision (face)PhysiologySmellPerception LayerHome Network (802.11, Bluetooth, HomePlug) + Mobile Internet (SIP +3G)Network LayerLayered Structure o

11、f Smart Home Service NetworkService Model Execution Platform (script translation, scheduling, QoS)Integrated PerceptionDevice LayerNetworked Physiology& EnvironmentMonitoring AppliancesNetworked Microphones;Cameras;SpeakersHome Comm. Gateway;Home Perception Server;Home Media CenterWirelessA/V St

12、reamingAppliancesMobile Agent PlatformAgent LayerEmotion / Semantics / Behavior / Intention UnderstandingCorpus of Knowledge(Ontology)InferenceEngineNatural LanguageProcessing(text, spoken)Cognition / Affection LayerVision (gesture)Device & Network Layers: types of digital applianceslClient-type

13、 devicesl802.11g-based multifunctional audio/voice adaptorl802.11g/MPEG-4-based multifunctional video adaptorl802.11g/MPEG-4-based smart IP cameralBluetooth-based ECG device lGateway-type deviceslMultimedia communication gatewaylServer-type deviceslHouse control server lHuman-machine interaction ser

14、verlContent serverlApplication serverDevice & Network Layers: relationship among server appliances用戶端設(shè)備FTTH/3G/WiMAX通訊伺服器COInternet/WWWTelephonyhouse control & housekeeping devicesdata storeA/V devicesWiFi/Home PlugWiFi/Home Plug主人應(yīng)用伺服器屋控伺服器WiFi/Home PlugWiFi/Home Plug內(nèi)容伺服器Scheduling Algorit

15、hmXMLService AgentTask agentTask agentTask agentASIBISPMSLKNFEAASI_1_1ASI_1_2BIS_1_1BIS_2_1PMS_1_2PMS_1_1LKN_1_1LKN_2_1FEA_1_1FEA_2_1Service ServerRegister SDHScenarioServer DBUser Request LocationServerScript XMLWhat to do ? How to do ? Where to do ? CommonAPI Architecture of agent platformAgent-ba

16、sed Runtime EnvironmentlExecution environment: IBM Aglets systemlCommon APIEvent_TriggergetDataFromSub(Int subsystemId,Int destSubsystemIdString function_name, parameters)Start_Service_Agent (Int subsystemId,Int deviceLocation, String text)Start_Service_Agent (Int subsystemId,Int deviceLocation, Fil

17、e file)getSDHStatus(void)getSubList (void)getFunctionList (void)getSubList (Int subsystemId )getScenarios (void)getUserLoc (Int userId)Event_TriggergetDataFromSub(Int subsystemId,Int destSubsystemIdString function_name, parameters)Start_Service_Agent (Int subsystemId,Int deviceLocation, String text)

18、Start_Service_Agent (Int subsystemId,Int deviceLocation, File file)getSDHStatus(void)getSubList (void)getFunctionList (void)getSubList (Int subsystemId )getScenarios (void)getUserLoc (Int userId)Adaptive Service Provider:architectureScheduling AlgorithmScheduling AlgorithmUser request (data, args)XM

19、LXMLService AgentService AgentTask agentTask agentTask agentTask agentTask agentTask agentASIBISPMSLKNFEAASI_1_1ASI_1_2BIS_1_1BIS_2_1PMS_1_2PMS_1_1LKN_1_1LKN_2_1FEA_1_1FEA_2_1Service ServerService ServerRegister Service AgentService AgentTask ListScheduling AlgorithmScheduling AlgorithmUser request

20、(data, args)XMLXMLService AgentService AgentTask agentTask agentTask agentTask agentTask agentTask agentASIBISPMSLKNFEAASI_1_1ASI_1_2BIS_1_1BIS_2_1PMS_1_2PMS_1_1LKN_1_1LKN_2_1FEA_1_1FEA_2_1Service ServerService ServerRegister Service AgentService AgentTask ListAdaptive Service Provider:functionaliti

21、eslFunctionalitieslRegistry mechanism for subsystem, device and functionalitieslService provider for user requestslLoad balanced service scheduling algorithm according to system resources lAgent cooperation mechanismAdaptive Service Provider:componentslService server lSubsystem and devices functiona

22、lities registration lService portal for userslMonitoring each subsystem and device lService agents lProvide service for each user requestlService composition lTask assignment and task agent dispatch according to predefined XML-based scenariosAdaptive Service Provider:components (cont.)lTask agent lE

23、xecute each functionality on each subsystemlCommon APIlService scheduling algorithmlProvide a task list for service agent according to registry and pre-defined scenarios in databaselA Petri net based & load balanced scheduling algorithm for adaptive service path in each subsystem and deviceAgent

24、-based Middleware: mobility managementlLocation detectionlDevice-followed type: mobile IP; signal analysislDevice-free type: speech interaction; vision monitoring.lSeamless handoff and transcoding for ubiquitous service following lRoaming path tracking and predictionAgent-based Middleware: appliance

25、 collaboration managementlCollaboration among homogeneous appliances: data fusion, task migration.lCollaboration among heterogeneous appliances: multi-modal HCI.lScheduling, concurrency control & synchronization of collaborative tasks.lSelf-organization for service deploymentAgent-based Middlewa

26、re: interoperability managementlDevice bridgelProtocol bridgelTranscryptionlTranscodinglContent translation & adaptationAgent-based Middleware: remote access managementlRemote service deploymentlremote service accesslremote service managementlauto-configurationlservice re-directionlservice aggre

27、gationlUI remotingAgent-based Middleware: other management functionslload management:lClient-server load partitionlServer load sharingl Load scheduling of appliance farm lavailability managementlFault tolerancelJust-in-time activation of appliances lservice quality managementAffective Speech Convers

28、ationEmotional Speech SynthesisTextAnalysisSadNeutralAngryEmotionalSpeech DatabaseDatabaseSelectionSyntacticAnalysisUnitSelectionSpeechSmoothingTextInputEmotionSelectionUsers ActionHappySpeechSegmentationBehavior Understanding by VisionlHigh-Level behavior understanding from videoslState MachinelHum

29、an Activity RecognitionlTwo-Stage recognition processlAccident/Abnormal behavior detectionlContext & domain knowledge CombinationSystem ArchitectureImageImageImageVideo StreamSegmentation &TrackingBackground(Update)TrackingForegrounddetectionFeature ExtractionPostureRecognitionMotionEstimati

30、onHistoryMapActivity RecognitionPosturesAnalysisMotionsAnalysisSizeAnalysisAccident DetectionViolent MotionsLying & StaticNormal DetectionState TransitionContextCombinationAbnormal DetectionDaily lifeinformationTemporalinformationMethod Activity RecognitionlActivity RecognitionlLevel 1 - posture

31、slPosture SequencelLevel 2 motion/historylHistory Map MatchingMethod Behavior understandinglBehaviorlNormal behaviorlState MachinelActivity + ContextslAbnormal behaviorlNormal behavior + domain knowledgelAccidentlUnreasonable activity + domain knowledgeFacial Expression AnalysisFace AcquisitionAcqui

32、sitionSegmentationFacial Feature ExtractionDeformation ExtractionMotion ExtractionRepresentationFacial ExpressionClassificationRecognitionKey frameSelectionYCbCrColor spaceEye RegionMouth RegionRegionOfInterestEyePointsMouthPointsDisplacement VectorsFuzzyNeuralNetworkInvariantMomentsOpticalFlowKey F

33、rameImage SequenceResultsIntegrated Perception:fuzzification of reference perceptual modelslManipulate all kinds of perception in a uniform process to ease the perceptual integration.lDue to high vagueness of perception, fuzzy logic based approach is a good choice to establish the reference models o

34、f perception.lThe reference models which fuzzify perceptual attributes and perceptual decision subspaces will be embedded into the integrated perception model.FL-based Acoustic Reference Model for Emotion Recognitionfeature extractionAAU1 modelSVM clusteringfor emotion 1SVM clusteringfor emotion 2SV

35、M clusteringfor emotion Vspeech corpusAAU2 modelAAUS model fuzzification of acoustic features (AFs) and construction of acoustic action units (AAUs) FL-based Acoustic Reference Model for Emotion Recognition (cont.)lAdopt SVM clustering approach in the subspace of each emotion type to gather the clus

36、ters of acoustic training patterns.lInspect all produced SVM clusters in the whole feature space and merge the highly overlapped clusters.lEach cluster is modeled as an AAU represented with its fuzzy cluster center where each feature is a fuzzy set whose membership function is determined by the leas

37、t-square curve fitting approach on the feature values of training samples included in the cluster.FL-based Acoustic Reference Model for Emotion Recognition (cont.)lThe mapping between AAUs and emotion types is dependent on the SVM clustering result of each emotion type.lEach emotion type is associat

38、ed with a set of clusters of acoustic samples. The weight of each cluster is determined by the ratio of the number of samples it contains with respect to the total amount of samples of the same emotion. FL-based Facial Reference Model for Emotion Recognition graphical head modelmorphological process

39、 to simulate AUsFACS AUs identification processfeature points (FPs) extraction processfuzzy logic based reference model for FACScorrespondenceMembership gradeMembership gradeFP1 valueFP2 valueFAU1FAU2FAUiFAUjFAUkFL-based Facial Reference Model for Emotion Recognition (cont.)lIntend to construct a co

40、mputational reference model for FACS action units based on the measurable features of facial expression.lAn approach similar to the construction of acoustic reference model is adopted.lThe training samples are generated from a generic head model with necessary morphological manipulation.FL-based Fac

41、ial Reference Model for Emotion Recognition (cont.)lThe membership functions will be determined by the least-square curve fitting approach according to the sample patterns produced from the morphological process.lEach AU may just represent a partial facial expression and relate to more than one emot

42、ion.FAU1FAU2FAUKAAU1AAU2AAUSFP1FPnAF1AFmemotion type layerrepresentative concept layerscaled feature layerprimaryfeature layerFace Features ExpressionAcoustic FeaturesFuzzy Neural Network for Integrated Emotion RecognitionFearAngerSurpriseFearAngerSurpriseFuzzy group decision process, group level of

43、 agreement Fuzzy Neural Network for Integrated Emotion Recognition (cont.)lAll kinds of perceptual information are fused by the FNN model to realize emotion recognition.lEach appliance will have an instance of the corresponding FNN to join the emotion recognition job. lA two-layered (emotion type &a

44、mp; concept layers) BP learning algorithm is adopted by using the training samples in constructing reference models. The fuzzy group decision process does not join the learning.lScaling input value to 0,1 in the second layer is realized by the membership function of the corresponding fuzzy set.Fuzzy

45、 Neural Network for Integrated Emotion Recognition (cont.)lThe links between AUs and scaled features are not fully connected.lThe FAU/AAU nodes realize normalized weighted sum for the membership grades of input features weighted by their respective link strength. lEach emotion type node determines o

46、utput value by the normalized weighted sum of its inputs from the representative concept layer.Cognition Layer:understanding and responselUnderstand the semantics of multi-modal expression.lClassify and recognize the intention/l need/emotion of semantic expression.lSummarize the semantics of multi-m

47、odal expression according to classified result.Cognition Layer:understanding and response (cont.)lPredict the user behavior sequence according to the classified result.lSchedule the response sequence according to the prediction result.lDetermine the instant response. StimulusPerceptionCognitionspokenlanguagegesturefaceexpressionphysiol