Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems

352 visualizaciones

Publicado el

16th PAAMS International Conference

Publicado en: Software
  • Inicia sesión para ver los comentarios

A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems

  1. 1. ALMA MATER STUDIORUM – UNIVERSIT`A DI BOLOGNA Computer Science and Engineering Department (DISI) – Cesena Campus, Italy A Model and Platform for Building Agent-Based Pervasive Mixed Reality Systems Angelo Croatti, Alessandro Ricci {a.croatti | a.ricci} 16th PAAMS International Conference June 22nd , 2018 – Toledo (Spain) A. Croatti (UNIBO) Augmented Worlds June 22, 2018 1 / 32
  2. 2. Mixed Reality will be Reality. . . The future of human consciousness will be a hybrid affair. We will live and work in a ubiquitous environment, where physical reality and a pervasive digital layer mix seamlessly according to the logic of software. This is Mixed Reality, and it will soon simply be reality! — John Rousseau [Rou16] A. Croatti (UNIBO) Augmented Worlds June 22, 2018 2 / 32
  3. 3. Mixed Reality Nowadays, a mainstream technology! To be exploited for: designing smart environments, blending physical and virtual objects introducing new opportunities in supporting individual and cooperative human activities Allows for: reshaping the environments where people work and live rethinking the way in which they interact and collaborate . . . A. Croatti (UNIBO) Augmented Worlds June 22, 2018 3 / 32
  4. 4. Virtual Continuum and Technologies A. Croatti (UNIBO) Augmented Worlds June 22, 2018 4 / 32
  5. 5. Toward (Agent-based) Pervasive Mixed Reality Systems Goal Investigate how MR technologies can be put in synergy with Pervasive Computing and Internet-of-Things (IoT) – as main ingredients of modern smart environments A. Croatti (UNIBO) Augmented Worlds June 22, 2018 5 / 32
  6. 6. The role of Agents in MR Agents (and MAS) can play a key role: modelling and implementing holograms with autonomous behaviour supporting the interaction of holograms with other holograms and humans featuring the observation of the physical world A“new era”of smart environments, where autonomous cognitive agents can dynamically create and control holograms and virtual objects control physical things and physical environment interact with humans in an homogeneous cyber-physical system A. Croatti (UNIBO) Augmented Worlds June 22, 2018 6 / 32
  7. 7. Related Works MiRa – Mixed Reality Agents [HCO+11] and AuRAs – Augmented Reality Agents [CSHO14] the focus is about agents having a representation in an MR environment to become perceivable by humans IVEs – Intelligent Virtual Environments [LA00, RPLJ+16] Agents for context aware applications Virtual Agents Mirror Worlds [RPTC15, RTP+15] A. Croatti (UNIBO) Augmented Worlds June 22, 2018 7 / 32
  8. 8. Augmented Worlds The Augmented World (AW) Vision Defining a single and uniform conceptual model capturing main aspects of current augmentation technologies allowing for the design of smart pervasive mixed reality software systems with an deep inspiration to agents programming paradigm Allowing the design and the development of innovative smart environments with an high level of abstraction proactive autonomous/cognitive software agents interacting with real things, physical space and its inhabitants A. Croatti (UNIBO) Augmented Worlds June 22, 2018 8 / 32
  9. 9. Augmented Worlds Augmented World A software application that enriches the functionalities of a particular physical environment exploiting augmented entities The idea of AWs is inspired by Mirror Worlds [RPTC15] and A&A meta-model [ORV08] A. Croatti (UNIBO) Augmented Worlds June 22, 2018 9 / 32
  10. 10. Augmented Worlds Conceptual Model Conceptual Model Formalization Agent Augmented World Augmented Entity AW User App Hologram Region +name +extension + name + position + orientation + extension + obs state + actions() + extension Human User run perceive, interact with render enter join, quit define, track, untrack create, do act, track, untrack spawn, kill can have User Mirror + gaze represented by Thing Mirror is-a represented by Physical Thing A. Croatti (UNIBO) Augmented Worlds June 22, 2018 10 / 32
  11. 11. Augmented Worlds Conceptual Model AW Concepts Augmented Entities and Holograms The full-fledge computational objects composing each AW Conceptually located into the physical space with a specific position, orientation and extension defined in relation to the reference system of the AW Designed with an observable state and a set of actions agents and humans can observe entities and act over them An AE exists in an AW even if it is not perceivable in terms of visual impact AWs allow to associates a geometry – an hologram – to each AEs To provide a representation for the entity in the real world Allows humans to see entities and interact whit them in Augmented Reality meaning A. Croatti (UNIBO) Augmented Worlds June 22, 2018 11 / 32
  12. 12. Augmented Worlds Conceptual Model AW Concepts Agents Cognitive Agents plays a key role in AW providing autonomous behaviour and functionalities to physical/digital entities When an agent join an AW, acquires the capabilities to create and manage augmented entities (tracking them and perceiving observable state) In cognitive agents based on BDI model, belief about the state of the AW and AEs are automatically managed, considering agents tracking requirements A. Croatti (UNIBO) Augmented Worlds June 22, 2018 12 / 32
  13. 13. Augmented Worlds Conceptual Model AW Concepts Human Users A human user starts a session inside an AW by means of an AW user app Is responsible to create an AR view, considering current position and orientation of the user Provides support to detect user inputs towards specific holograms Many users can join the AW concurrently, each of them represented in the AW by a User Mirror augmented entity The state of a user must be made available to all interested entities, using the User Mirror concept A. Croatti (UNIBO) Augmented Worlds June 22, 2018 13 / 32
  14. 14. Augmented Worlds Conceptual Model AW Concepts Physical World Coupling An AE can offers the medium to dynamically extend physical objects functionalities injecting MR benefits into the IoT (and WoT) landscape Bidirectional Augmentation AEs must be coupled with related physical object Changes in the physical object state tracked in the AE state and vice versa. Necessity of a“degree-of-freshness”to infer if and when the digital state could be no longer aligned to the real state A. Croatti (UNIBO) Augmented Worlds June 22, 2018 14 / 32
  15. 15. MiRAgE MiRAgE: a framework for Augmented Worlds Mixed Reality based Augmented Environments provides a runtime and an API to design, develop and deploy AWs allows for the integration of agents and MAS in pervasive mixed reality complex system Key Features 1. Incremental augmentation degree from flat (smartphone-based) to full hologram-based MR experiences 2. Multi-user/cooperative systems (multi)agents and (multi)humans cooperation/collaboration 3. Integrated with SW system engineering support for heterogeneous mainstream technologies integration of IoT/WoT stack and technologies A. Croatti (UNIBO) Augmented Worlds June 22, 2018 15 / 32
  16. 16. MiRAgE Architecture MiRAgE Architecture A. Croatti (UNIBO) Augmented Worlds June 22, 2018 16 / 32
  17. 17. MiRAgE Architecture MiRAgE Architecture AW Runtime and WoAT Hosts the execution of one or multiple instances of AWs Manages the creation of AEs, the execution of their actions and allows access to their observable properties Proposes an interoperable interface to access to entities called WoAT – Web of Augmented Things [CR17b, CR17a] a RESTful interface based on HTTP operations and event-oriented mechanisms full interoperability and openness at application levelfor agents technologies based on (is a superset of) the Web of Things (WoT) model [GT16] A. Croatti (UNIBO) Augmented Worlds June 22, 2018 17 / 32
  18. 18. MiRAgE Architecture MiRAgE Architecture Hologram Engine Is responsible of the management of holograms geometries definition and updates physics management allows interactions in AR-style and input by human users (e.g. gaze-based, hands/finger gestures, . . . ) . . . Supports the AR/MR rendering of holograms wraps AR/MR libraries and algorithms enables marker-based and marker-less AR techniques A. Croatti (UNIBO) Augmented Worlds June 22, 2018 18 / 32
  19. 19. MiRAgE Architecture MiRAgE Prototype Technologies AW Runtime – mainly Java based Vert.x library for implementing (part of) the WoAT layer JSON used to model data exchanged in WoAT REST APIs calls Hologram Engine – based on Unity 3D (and C# scripts ) Vuforia plugin to manage AR aspects Mapbox plugin to manage GPS localization aspects Augmented Entities – defined as templates through Java-based API exploiting ad-hoc annotations for types, fields and methods Agents – designed and developed using the BDI model proposed either by Jason/JaCaMo and ASTRA CArtAgO and EIS libraries provided A. Croatti (UNIBO) Augmented Worlds June 22, 2018 19 / 32
  20. 20. MiRAgE Examples The MiRAgE main benefit Designed with the goal to hide as much as possible implementation details and enabling mechanisms of the AR/MR level Idea: focusing only on the application logic! 1. Design AEs templates (defining observable properties and actions of each AE) 2. Design Hologram geometries (ore reuse provided ones) 3. Design application logic for agents living in the AW 4. Design application logic related to Humans interaction with the AW 5. Deploy and Run the AW A. Croatti (UNIBO) Augmented Worlds June 22, 2018 20 / 32
  21. 21. MiRAgE Examples AE template example @HOLOGRAM("Boat") public class Boat extends AE { @PROPERTY private double speedVal; @PROPERTY private Vector2D speedVersor; @PROPERTY private double windForce; @ACTION public void setSpeed(Vector2D versor , double val) { customProperty ("speedVal", val); customProperty (" speedVersor", versor); } @ACTION public void setWindForce (double windForce) { customProperty ("windForce", windForce); } } A. Croatti (UNIBO) Augmented Worlds June 22, 2018 21 / 32
  22. 22. MiRAgE Examples Agents API Primitive Actions Description joinAW(name, location): awID to join an existing augmented world, getting an id of the session quitAW(awID) to quit from working in an augmented word createAE(awID, name, tem- plate, args, config): aeID to create a new augmented entity in a specified aug- mented world, specifying its name, template, param- eters (that depend on the specific template), and initial configuration (including position, orientation, ...) disposeAE(aeID) to dispose an existing augmented entity trackAE(aeID) to start tracking an existing augmented entity stopTrackingAE(aeID) to stop tracking an existing augmented entity moveAE(aeID, pos, orienta- tion) to change the position and orientation of an aug- mented entity, if allowed defineRegion(awID, name, re- gion) to define a named region, specifying a name and the extension trackRegion(awID, name) to start tracking a region stopTrackingRegion(awID, name) to start tracking a region A. Croatti (UNIBO) Augmented Worlds June 22, 2018 22 / 32
  23. 23. MiRAgE Examples Jason Agent examples BoatPilot.asl +! init <- joinAW("museumAW"), trackAE("boat"), trackAE("wind"), !navigate (30). +! navigate(Speed) <- getRndOrientation (Versor), setSpeed(Versor , Speed), .wait (5000) , !navigate(Speed). @mng_border_collision [atomic] + borderReached <- ! divertBoatRoute (180). +windForce(V) <- setWindForce (V). LighthouseCtrl.asl !init. +! init <- joinAW("museumAW"), trackRegion ("museumAW"," red -zone"). + regionUpdate ("red -zone", " enter", AE) <- turnOnLighthouse , registerRedZoneAccess (AE) . + regionUpdate ("red -zone", " exit", AE) <- turnOffLighthouse , registerRedZoneExit (AE). A. Croatti (UNIBO) Augmented Worlds June 22, 2018 23 / 32
  24. 24. MiRAgE Examples The Unity (customized) Editor A. Croatti (UNIBO) Augmented Worlds June 22, 2018 24 / 32
  25. 25. Case Studies and Applications Case Studies Augmented Museum and Augmented Harbour Two real case studies in the cultural heritage field (maritime archaeology context) with the collaboration of the“Museo della Regina”, Cattolica Augmented Hospital In the Healthcare 4.0 vision, (a part of) an hospital can be conceived as an AW Currently we are working with the purpose to build an Augmented Emergency Room – the TraumaTracker Project to support Trauma Team in managing emergencies to assists Trauma Leader with a kind of Personal Assistant Agent with the collaboration of the Trauma Center of the Cesena Hospital A. Croatti (UNIBO) Augmented Worlds June 22, 2018 25 / 32
  26. 26. Current State and Future Work Current state and Considerations AW is a conceptual framework for investigating and exploring the use of cognitive agents and MAS for developing complex AR/MR systems MiRAgE (with the WoAT layer) allows software designers and developers to shape cyber-physical systems and smart environments abstracting from specific enabling augmentation technologies We promote an agent-oriented level of abstraction at the design level Developed case studies have provided a first positive feedback about the effectiveness of the approach A. Croatti (UNIBO) Augmented Worlds June 22, 2018 26 / 32
  27. 27. Current State and Future Work The AW Contribution to the State-of-the-Art Shared/multi-user worlds multiple human users immersed in the same AW sharing and interacting with the same virtual entities Bidirectional Augmentation proposing a notion of augmentation wider than in MR pure context about enriching physical environment with computational capabilities Open Systems where software agents and humans can dynamically join to a dynamic augmented world Enabling AR Technologies allowing for a complete reuse of existing AR/MR enabling technologies both software and hardware Generality supporting the development of AWs for different application domains A. Croatti (UNIBO) Augmented Worlds June 22, 2018 27 / 32
  28. 28. Current State and Future Work The AW Contribution in Agents and MAS Research The design of AWs is an interesting application domain for agents research challenging features it’s a real (not simulated) testbed Humans in the loop The Humans-Agents Collaboration can be studied in situated contexts A wide perspective for the research Allow to simplify the design of Personal Agents for human assistance in AWs, agents can“see what humans see”: both the real and the virtual world elements A. Croatti (UNIBO) Augmented Worlds June 22, 2018 28 / 32
  29. 29. Current State and Future Work Future Work and Directions A concrete evaluation of the AW conceptual model through (ongoing) case studies and ad-hoc experiments An improvement of the MiRAgE architecture and its implementation to make it widely available as a reference platform to build Agent-based Pervasive Mixed Reality Systems Explore the application of AWs towards building Digital Twins mainly, in the Industry 4.0 Explore agent design considering HCI techniques A. Croatti (UNIBO) Augmented Worlds June 22, 2018 29 / 32
  30. 30. References References I [CR17a] Angelo Croatti and Alessando Ricci, Towards the web of augmented things, 2017 IEEE International Conference on Software Architecture Workshops (ICSAW), IEEE, April 2017, pp. 80–87. [CR17b] Angelo Croatti and Alessandro Ricci, Mashing up the physical and augmented reality: The web of augmented things idea, Proceedings of the Eighth International Workshop on the Web of Things (New York, NY, USA), WoT 2017, ACM, 2017, pp. 4–7. [CSHO14] Abraham G. Campbell, John W. Stafford, Thomas Holz, and G. M. O’hare, Why, when and how to use augmented reality agents (auras), Virtual Real. 18 (2014), no. 2, 139–159. [GT16] Dominique Guinard and Vlad Trifa, Building the web of things: With examples in node.js and raspberry pi, 1st ed., Manning Publications Co., Greenwich, CT, USA, 2016. A. Croatti (UNIBO) Augmented Worlds June 22, 2018 30 / 32
  31. 31. References References II [HCO+ 11] Thomas Holz, Abraham G. Campbell, Gregory M. P. O’Hare, John W. Stafford, Alan Martin, and Mauro Dragone, Mira-mixed reality agents, Int. J. Hum.-Comput. Stud. 69 (2011), no. 4, 251–268. [LA00] Michael Luck and Ruth Aylett, Applying artificial intelligence to virtual reality: Intelligent virtual environments, Applied Artificial Intelligence 14 (2000), no. 1, 3–32. [ORV08] A. Omicini, A. Ricci, and M. Viroli, Artifacts in the a&a meta-model for multi-agent systems, Autonomous Agents and Multi-Agent Systems 17 (2008), no. 3, 432–456. [Rou16] John Rousseau, The laws of mixed reality, or mixed reality without the rose-colored glasses, 2016, reality-without-rose-colored-glasses/. A. Croatti (UNIBO) Augmented Worlds June 22, 2018 31 / 32
  32. 32. References References III [RPLJ+ 16] J. A. Rincon, J. L. Poza-Lujan, V. Julian, J. L. Posadas-Yague, and C. Carrascosa, Extending mam5 meta-model and JaCalIVE framework to integrate smart devices from real environments, PLOS ONE 11 (2016), no. 2, 1–27. [RPTC15] Alssandro Ricci, Michele Piunti, Luca Tummolini, and Cristiano Castelfranchi, The mirror world: Preparing for mixed-reality living, IEEE Pervasive Computing 14 (2015), no. 2, 60–63. [RTP+ 15] Alessandro Ricci, Luca Tummolini, Michele Piunti, Olivier Boissier, and Cristiano Castelfranchi, Mirror worlds as agent societies situated in mixed reality environments, Revised Selected Papers of the International Workshops on Coordination, Organizations, Institutions, and Norms in Agent Systems X - Volume 9372 (New York, NY, USA), Springer-Verlag New York, Inc., 2015, pp. 197–212. A. Croatti (UNIBO) Augmented Worlds June 22, 2018 32 / 32