Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

CMPE- 280-Research_paper

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Cargando en…3
×

Eche un vistazo a continuación

1 de 10 Anuncio
Anuncio

Más Contenido Relacionado

Presentaciones para usted (20)

Similares a CMPE- 280-Research_paper (20)

Anuncio

CMPE- 280-Research_paper

  1. 1. Diksha Jain, Meera Mali, Ojas Milind Kale, Sanjeedha Sanofer Raja, Shagun Juneja CMPE Department, San Jose State University, CA Abstract: This paper surveys the idea of augmented reality, which is the incorporation of digital data such as online information and graphics with the user's environment in real time. It makes the end-user’s world more interactive and digitally manipulable. It is proving as an extremely demanding technology these days. This paper presents the concepts and designs behind building up this technology and the characteristics of applications using augmented reality. It focuses on the related work or the different cases of Augmented Reality applications using this technology as a part of different creative ways. It demonstrates the historical viewpoint of the technology and the procedure to augment the reality. This paper summarizes the current problems with the existing systems and the future areas requiring further research. The objective of this survey paper is to provide a general layout, fact and findings and current analysis on the ongoing projects based on Augmented Reality with some future enhancements. Index Terms: 3-D Graphics with real-world, augmented reality, project Tango, augmented real world. I.INTRODUCTION Nowadays, researchers and engineers are extracting graphics out of the TV screen or computer display and integrating them into real-world environments [1]. This new innovation, called augmented reality, blurs the line between what's real and what's computer-generated by enhancing what we see, hear, feel and smell. Augmented reality is not just like a real life but better than it. In a general term, Augmented Reality is a combination of a real scene viewed by the end-user and a virtual scene generated by a computer that augments the scene with some additional information and meaning to a real object or a place. It involves AR technology i.e. addition of computer vision and object recognition which helps in making a simulated environment whose components are augmented by computer-based sensory inputs such as sound, graphics, video, or GPS data. As a result, this innovative technology functions by improvising one’s present aspect of reality. Unlike virtual reality, which makes a totally artificial environment, augmented reality uses the current existing environment and overlays new data on top of it. The idea behind this is straightforward and direct: take a real-life scene, or (better) a video of a scene, and include some kind of logical information to it. This is how augmented reality works and thus helps in identifying the things which are around you, recognizing the people in the scene and the places you want to visit. Milgram characterized a continuum of real to virtual environments, where augmented reality is a part of “Mixed Reality.” He defined the term Augmented Virtuality to recognize systems which are mostly synthetic in nature with some real world imagery included just like texture mapping video on the virtual objects. A. How Does Augmented Reality Work? The webcam connected to the computer is responsible for capturing the photos or videos which captures the images or videos in the traditional manner (by taking lots of photos successively i.e. one after the other.) When a “marker” is placed in front of the webcam, it recognizes the “marker” and collects the information or pattern encoded in it and sends this data to the computer. The computer perceives the data and overlays the marker with a picture such that it seems to the viewer that the image has appeared by magic. The computer is capable enough to track the size and continuous movement AUGMENTED REALITY: INTEGRATING THE DIGITAL WORLD WITH REAL WORLD
  2. 2. of the image which means if the marker is brought closer to the webcam, the image will get magnified. If the marker is tilted towards left, the image will also behave in the same manner and thus it appears to be tilting in the left direction. This procedure is like sports broadcasts on television, such as swimming events, a line is powerfully added over the paths to demonstrate the virtual position of the present record holder as a race continues. B. Why Did Augmented Reality Come? Augmented Reality is a live view of the world from an indirect point of access. When 2D and 3D photos can give only a static view of the live world, people were not satisfied with only a two or a three dimension. Business people wanted a methodology to lure more shoppers to their products. They wanted the products and the specs of the products to be more reachable to the consumers without actually buying their products. Medical professionals need to get more pragmatic contour of the patient when they are in remote operation. Gamers need to indulge themselves to the gaming environment and they want to be the characters themselves. The main idea for all the people is to create more realistic and tactile view of the environment and the object placed in the real world. The imaginary world of reality must be interactive and digitally manipulable. This need provokes the creation of an augmented reality environment. II. RELATED WORK Augmented Reality (AR) has rapidly turned out to be very well-known and widely used technology and thus become an expression that trips easily off technologists' lips; yet we've been seeing various versions of it for a long while. AR has come quite close to a wide range of engineers and this technology is sufficiently effective to make use of it by making large number of people possessed by it. The arrival of powerful smartphones and computers having built-in video capabilities increases the use of AR and thus people don’t have to wait for the AR effects as they do with television. They can simply be overlaid onto real life. Stepping one step ahead in the same like Apple's iPhone, and phones based on Google’s Android as an operating system, they both are capable of overlaying data or information on top of an image or video. A. What Are Some Examples? 1) Intel X-ray-like glasses: One of the project which is based on AR is Intel X- ray-like glasses which permit wearers to ‘see inside’ objects. Smart augmented reality helmet allows wearers to overlays maps, schematics and thermal pictures to viably see through walls, funnels, pipes and other solid objects. Intel has launched a set of glasses incorporated with a helmet which used RealSense 3D camera for providing x-ray-like vision such that viewer can view inside the object. This technology makes use of a 3D high-definition and infrared camera, an infrared laser projector and software that is used to customize the camera usable for a particular application. Unlike other devices, which have been developed for consumers or end-user’s use, this device is designed and developed with industrial use in mind. It will allow the wearer or the person wearing this gadget to effectively view the actual workings of the objects using real-time information, such as wiring graphs, diagrams, charts etc. It will be helpful for the viewer to find out the problem areas, which needs further maintenance or needs to get fixed. Fig 1: A wearable helmet to see the x-ray like vision . This gadget makes use of RealSense systems, which are combined with Intel’s latest Core m7 processor, a 360-degree sensor array, and Daqri’s computer vision, and tracking system. By making use of such a high tech objects, it claims to be the “most powerful augmented reality wearable device” available in the market. This device will be useful in various domains such as aerospace, oil, construction and gas industries.
  3. 3. This product of Intel will be available in the market for purchase in the first quarter of this year i.e. 2016. 2) Google’s Project Tango + Lenovo Another venture working on AR is Google’s project Tango. Project Tango consolidates 3D motion tracking with depth sensing to give mobile device an ability to know where it is and how travels through space. With project Tango, developers or engineers can develop applications that investigate physical space around the user, including exact routing without GPS, windows into virtual 3D worlds, estimation of spaces, and diversions that know where they are present in the room and what’s around them [2]. We can understand the meaning of above sentence as visualizing the actual real world with close eyes. Google’s Project Tango and Lenovo are bounding together to make the world’s first smart phone powered by Project Tango innovation. The device, will allow the users to experience the world in a more innovative manner, which was never possible through a smartphone. This device will be going to launch in summer 2016 and will be available in the market for end- user for purchasing. Fig 2: Sensors used in Project Tango This technology works by adjoining various inputs from the pre-installed range of sensors and processing these inputs into usable information very rapidly. These sensors consist of an infrared camera and infrared emitter, which emits radars and helps in absorbing the reflected light. A wide-angle camera adds visualization clues about the current location. This system also consists of varieties of highly functional accelerometers, barometers and gyroscopes. For this technology, Google offers three APIs- first one for game developers, second one for using Java to integrate Tango into various mobile apps, and third one for the apps, which have their own visualization centric engine to provide compatibility to Tango. III. HISTORICAL PERSPECTIVE Over the past few decades, augmented reality-the artificial view of the physical world has matured from a scientist's plaything to a part of daily life. In 1647 Rene Descartes, a French mathematician and scientist, mused upon the difference between what is real and what is constructed in our mind through a series of thought experiments about reality, as shown in Fig.3 Fig 3: The Proposed Idea In 1647 Ivan Sutherland, a computer scientist and Harvard Associate Professor and his student Bob Sproull invented the first augmented reality head-mounted display system. As shown in Fig 4, the system was suspended from a ceiling for the user to experience the graphics, which was fed by computer programs. It was primitive both in terms of user interface and realism. The graphics that comprised the virtual environment created were very simple wireframe rooms. Depending on the position of the user's gaze, the perspective was showed to the user. It was attached to a mechanical arm suspended from the ceiling, which tracked the head movements. The translucence nature of the system where the unit was partially see-through along with its other
  4. 4. features makes it the perfect predecessor to augmented reality technology. Fig.4 First Augmented Reality System in 1968. In 1990, Tom Caudell, a Boeing researcher introduced the term "augmented reality". In 1994, Julie Martin took augmented reality to television by creating the Theater production "dancing in Cyberspace”. It featured dancers and acrobats manipulating virtual objects in real time, which were projected into the same performance plane and physical space. In 1997, Ronald T. Azuma's research brought light into the different uses of augmented reality in medical, manufacturing, research, mechanical operation and entertainment by publishing "A Survey of Augmented Reality". In 2004, a group of German researchers projected the idea of see-through augmented reality to mobile phones via a weak perspective projection camera model and an OpenGL rendering pipeline [3]. Later in the 2000s, Hirokazu Kato from Nara Institute of Technology developed the ARToolKit - an open-source computer-tracking library for creation of strong augmented reality applications. It combines virtual graphics with real life, and uses video tracking to overlap computer graphics on a video camera in turn resolving the two problems of viewpoint tracking and virtual object interaction. To this day the world is seeing augmented reality growing and applied in many domains and it is continuing to create revolutions in computer interface design. IV. HOW TO AUGMENT THE REALITY There are three basic criteria for augmenting the real world scenario 1) User Augmentation The User uses a wearable technology to get the feel of the reality surrounding them as in fig 1. The wearable computing is actually a physical device, usually worn on the head or hands to gain knowledge about the physical objects [4]. The earliest device used was a head-mounted display, which allowed the used to view and feel artificially, designed objects and become immersed in a world of virtual reality. Then, data gloves were introduced to control slides and videos during presentations. The presentation movements can be controlled by specialized gestures. Some augmentation allowed users to view through the objects just like a mother feeling the baby inside her during ultrasound. Knowledge- Based Augmented Reality for Maintenance Assistance (KARMA) was designed to help the technicians see through the relevant repaired parts and correct the damage in real-time. These devices involve coupling of the images and the view of the physical world. Fig 5: A wearable headset consisting the sensor for augmented reality 2) Physical Object Augmentation Reality is infused into the physical objects by transforming the object’s reality by embedding the input, output and the computational devices associated with the object. In the early 70s, Lego/LOGO language laid the basic foundation for augmentation of the physical
  5. 5. objects. The language was designed to help the kids play with the electronic blocks. The blocks contained a type of sensor, which helped to control the construction. Other sensors and objects can be added when necessary. This approach paved the way for ubiquitous computing which detects specialized objects by sensors placed all over the place. Another example would be the use of active badges, which help support collaborative activities, like sharing documents, reminding events and meetings. 3) Environment Augmentation Instead of affecting the user or the object, the knowledge of the environment surrounding the user or the object is provided by devices. The devices capture the information about the user’s interactions and display the information unto objects. Imagine a user sitting at a place and controlling the movements of an object by directing commands to move the object. Sensors are used to detect the gesture of the users. At the earliest period, video camera and a close-up camera is used to capture the images, which have now been replaced with sophisticated cameras and sensors. In all these cases of augmenting the environment, special devices are not needed to visualize the reality. A. Components of AR 1) Camera Data The live-data capture of the augmented reality is done by camera. Camera feed is the essential part in generating the query for the technology. Using the camera lens, the image for which the information is to be populated can be captured as in fig 2. For example, by pointing the camera to an object, knowledge can be gained about the details of the object. Camera APIs are used to analyze the frame data. Methods such as setPreviewDisplay() and setPreviewCallback() are used to display the camera data. Fig 6: An augmented Reality Camera setup to capture the live feed 2) Location Data Only the camera data would not suffice. There also rises a demand to determine the location of the device and the user accessing the device. The location API can help to listen to the location and populate the events based on the location mostly from the live feed. OpenCV libraries are used for the same to augment data. The location also depends on the tagged information or the marker info. 3) Sensor Data Sensors are useful to determine the connection between the user’s gestures and the objects [5]. The orientation of the device plays a major role in synchronization of the camera with the live feed. Sensor APIs and methods such as Sensor.TYPE_MAGNETIC_FIELD,Sensor.TYPE _ACCELEROMETER help to calculate the accurate position of the devices. The use of sensors allow the user to move the device in any direction and also experience live changes in the screen with relation to the reality. For example, Google Maps or the Street view helps to explore the live data according to the device orientation. 4) Graphics Overlay The concept of augmenting the reality is to add layers above the camera feed data, [5] which makes the user to experience live scenarios. Each frame from the camera feed is overlaid and a frame is drawn corresponding to each screen. Depending on the selected 2D or 3D, WebGL packages, the frames are rendered.
  6. 6. 5) Storage The data for augmentation comes from the database, which might be stored locally or on remote servers. For Example, SQLite database is used for quick and easy search. For web-based data access, web services are used which takes the use of methods such as HTTP or XML parsing. B. Open Technologies 1) WebGL WebGL is a JavaScript API for 2D/3D graphics rendering. It is run on any compatible browsers. It utilizes the effects of physical motion and image processing with additional effects to render the complex image on the canvas. WebGL is based on OpenGL standards and HTML5 Canvas elements and can be accessed by using the Document Object Model in JavaScript hierarchy. var canvas = document.getElementById (‘mycanvas’); var gl = canvas.getContext(‘webgl2’); It does not require additional installation capabilities and has sufficient interfaces and extensions to render images accurately. 2) ThreeJS ThreeJS is a JavaScript API to create and display 3D graphics using WebGL on a web browser. It does not need any plugins. The 3D graphics can be rendered by creating camera, objects, lights and other materials. The scene to be rendered can be decided by the usage of HTML5 canvas, WebGL or SVG. var renderer = new THREE.WebGLRenderer(); 3) WebRTC WebRTC enables audio/video streaming and data sharing between the server and the client. WebRTC provides conferencing facilities which aims to exchange information between peers without the inclusion of any third party network. It uses JavaScript and net stream API’s. There is also a data stream API that helps in real-time text and file transfer. 4) Awe.js Awe.js is a JavaScript library for rendering augmented reality. It also aids in creating and manipulation of the objects using JavaScript. Awe.js uses Three.js in addition with JavaScript to create the effect in the browser. It provides four scenarios to experience Augmented Reality – geo_ar for mapping compass points, grift_ar for making environments compatible with Oculus Rift, leap_ar for leap motion control and marker_ar to create experiments on AR. 5) ARToolKit ARToolKit is a library for constructing the Augmented Reality applications. It solves the main problem of determining the viewpoint of the user. This toolkit uses vision algorithms to calculate the camera positions and physical markers in the real- time. It defines the calibration of the device to position the viewport for the device. 6) Blend4Web It is a web-oriented 3D engine for authoring and interactive rendering of the scenes on the context of 3D graphics and audio in the web. It is tightly coupled with the Blender, which is a 3D animation tool using WebGL rendering. It also has debugging and optimization features. 7) BeyondAR BeyondAR is a framework designed for augmented reality on smartphones and tablets. It deals with geolocation based Augmented Reality on android environment and gaming. The base concept is to use OpenGL libraries for the scene rendering. C. Next Steps 1) Localization and Registration There arises a need to devise a way to localize a device when performing outdoor navigation. Earlier systems used Global Positioning System for Outdoor usage. This GPS was used as a compass for sensing the direction and also for position measurements. However, the accuracy of the GPS is still questionable and it creates large problems for registration of virtual objects since GPS signals deteriorate in urban surroundings and building blockages. Similarly, there is a drift on the sensor data when disturbed by shadowing. For a large algorithm computation, there exists only a small mobile computing processor, which also poses a problem. Hence for good localization criteria, image-matching and geo-location sensors must be working in coordination to provide a robust platform.
  7. 7. 2) User Evaluation The user prototype must be evaluated to provide guidance to application development in augmented reality [6]. Evaluation methods for user prototypes for mobile devices are a field, which is emerging. As of now, the prototypes are concentrated only upon a small arena. The user evaluation must be done for large-scale use and cover a wider arena of social networking applications. 3) Application Development The application areas of developing augmented reality are more and so are the challenges for establishing the hardware and the software for building the environment. For example, to design a geo localization augmented reality, the device has to plan for city exploration, urban locations, tagging the unknown places, getting the culture information, 3D reconstruction of sites and urban maintenance. V. ISSUES WITH AUGMENTED REALITY With this evolving technology, there are some issues or problems, which restricts this technology to be used in every domain. 1) Profiling As the name suggests this includes the profiling of person it may include his/ her personal data as well as some other information. This typically includes Face recognition techniques, geo-location with co- ordination of data from augmented reality. The combination of these technology will lead in integration of Offline as well as online live of the person. With this progress comes the drawback or concern, which states that, a person now walking will no longer be just a physical Human being, but he/she will be walking with his digital profile as well. And this can lead in myriads of problem for the person some of them would be, getting solicited by the society or a group of people because of certain believes or Political or Religious stands. Or maybe singled away for additional Security check on public transport stations, such as Airports, Train stations etc. 2) Unauthorized Advertising Advertising your product is the need of today’s market, and augmented reality can play vital part in this. Companies these days are thinking about the possibilities of using simple objects and physical spaces around. Each of them can possibly use as the digital ads board which can hold ads onto them real-time. The dark side of this possibility is, when you think about the private belongings and intellectual properties being used for the advertising purpose without prior permission can lead into a big problem of “Unauthorized advertising”. If the advertising using augmented reality does not come with the inbuilt controls which prevent them from using any private or intellectual property, this can be the biggest concern in augmented media domain. 3) Physical danger One of the biggest concerns of the augmented reality. Think of the scenario where mobile phones are currently distracting while driving, now think about the augmented windshield providing you continuous feeds of driving directions along with more data of what is happening around you, providing you live feed of current Football game, highlights of the presidential election debate and what not. Moving ahead imagine crossing a street busy with heavy traffic in unfamiliar neighborhood, while feeding you with tweets and snap-chats from your friends and simultaneously giving you information about nearest restaurant along with some advertise. Thus, if the augmented reality is not used smartly can lead into chaos. 4) Spam When there is an opportunity for advertising there will be an opportunity for spamming. Deceptive advertising can lead customers to buy thing they don’t really need. Augmented reality can depict a product bigger and better than actual. Although there are lot of legitimate companies that are harvesting your shared data, more than them, there are scammers who have all your data that can be used to lure you in too good to be true augmented offer in real world. VI. APPLICATION & FUTURE SCOPE With the bulk of possibilities that Augmented opens for the human kind, there is at least one application of Augmented in almost every field. To name a few, the military uses augmented reality to help men and women making repairs in the field. The gaming industry is moving games outside like the old days equipped with wearable headgear of course.
  8. 8. 1) Navigation Navigation using augmented reality is a real world example of how this technology can make human life easier and better. It makes use of Global Positioning system hand in hand with augmented reality, to make it easier to travel from your position to your destination. “Wikitude Drive” is one of the examples of this technology. It uses GPS along with phones camera to depict the exact picture of what is lying in front of your vehicle. This is the 21st century map. Fig 7: Camera along with the Car’s GPS to show the road ahead. Fig 7 shows the camera is used along with the Car’s GPS to see the road ahead. It is also providing many other information such as distance, speed probable reach time etc. [7]. 2) Sightseeing The use of technology in the domain of tourism industry. To think of a best scenario, think of tourist for the museum. The ability to augment a live view of all the scenes in a museum with all the statistics of facts and figures is a natural use of the technology. Their interactive kiosk solution allows guests to interact with the display in 3D. Also out in the real world, sightseeing has been enhanced using augmented reality. Using augmented reality, tourists can walk through historic sites and see facts and figures representing all the live data. These applications use GPS and image recognition technology to look up data from an online database. There could be another advantage of this technology that a picture can show up how the exact same place would look like 200 years ago. 3) Military One of the most productive use of augmented reality is in Military domain. The Heads-Up Display (HUD) is the typical example of augmented reality in military applications of the technology. The term "heads-up" comes from the fact that the pilot doesn't have to look down at the aircraft's instrumentation to get the data they need. It exhibits a transparent display that is positioned directly in the fighter pilots view. Data typically displayed to the pilot includes altitude above the ground, airspeed and the description of the horizon line in addition to other critical data. The Head-Mounted Display (HMD) is used by ground troops as well. Critical data such as enemy location can be presented to the soldier within their line of sight [8]. 4) Medical There are many interesting advantages of augmented reality in medical application. Medical students use this technology to practice surgery in a more controlled environment. Visualizations helps in explaining and understanding the complex medical conditions to various patients. AR helps in reducing the risk of an operation by providing the surgeon with an improvised sensory perception. This technology can also be combined with MRI or X-ray systems to bring everything into a single view for the Doctor. Neurosurgery takes the first position when it comes to surgical applications of augmented reality. The ability to show the image of the brain in 3D on top of the patient's actual anatomy is very important for the surgeon. Since the brain is somewhat fixed compared to other parts of the body, thus the registration of exact coordinates can be achieved for the brain. But concern still exists surrounding the movement of tissue during surgery, might affect the predefined coordinates to move a little bit. VII. CONCLUSION In this paper, we have described the concept of augmented reality, which can be widely deployed on handheld devices and at consumer level. Furthermore, we have showed the working of augmented reality and the requirements, which are essential to develop location-based, augmented reality experiences that can be flourished on a broader or global level. In the future, various domains such as Education, E-commerce and
  9. 9. digital marketing can make use of augmented reality for further development. VIII REFERENCES [1]. Ronal T. Azuma, Tele operators and Virtual Environments 6, 4 (August 1997), 355-385. [2]. Adrian Kingsley-Hughes, August 19, 2014. Inside Google's Project Tango tablet. http://www.zdnet.com/insidegoogles-project-tango- tablet-7000032746/. [3]. M. Mohring; Bauhaus Univ., Weimar, Germany; C.Lessig; O. Bimber: Video see-through AR on consumer cell-phones. 2, 5 (Nov. 2004), 252-253 [4] Azuma, R.T., A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4 (Aug 1997), 355–385, 1997 [5] MacIntyre B., Gandy M., Dow S., and Bolter J.D., DART: A Toolkit for Rapid Design Exploration of Augmented Reality Experiences, User Interface Software and Technology (UIST’04), Sante Fe, New Mexico, 2004 [6] Wagner D., Mulloni A., Langlotz T., and Schmalstieg D., Real-time Panoramic Mapping and Tracking on Mobile Phones. In Proc. of Virtual Reality’10, pp. 211–218, 2010 [7].Burnett, G. E. A Road-Based Evaluation of a Head-Up Display for Presenting Navigation Information. In Proc. HCI Int’l Conference (2003). [8] E. Foxin, M. Harrington: WearTrack: A Self- Referenced Head and Tracker for Wearable Computers and Portable VR, Proceeding of the Forth International Symposium on Wearable Computers, pp. 155-162, 2000. M. Young, the Technical Writer's Handbook. Mill Valley, CA: University Science, 1989.

×