1. CREATING COLLABORATIVE PANORAMAS
Using Smart Phones
By
Anirudh Ladha
Advisor: Dr. John Gauch
Committee: Dr. Craig Thompson, Dr. Nilanjan Banerjee

3. Outline
 Problem
 Objectives
 Background
 Our Approach
 Experimental Results and Analysis
 Conclusions
 Future Work

4. Problem
 Google Street View is an example of commercial
panoramas.
 A fleet of camera mounted vans take images that are
stitched together.
 Problem is images become out of date over time.
 The cost required to take pictures continuously is
prohibitive.
 We need a better way to collect up to date images.

5. Objectives
 Use a social network of mobile phones to gather geo-
tagged images.
 Combine these images to get up to date panoramas to
augment Google Street view.
 Consider energy usage and network/GPS availability
when creating our system.

6. Background
 Sensor Enabled Mobile Phones.
 Energy Optimization.
 Image Rendering and Modeling.

7. Sensor Enabled Mobile Phones
 Camera, GPS, Accelerometer have been used in a wide range
of mobile phone applications.
 Microblog[2008] – treats mobile phones as people centric
sensors.
 AAMPL[2008] – uses accelerometer signatures to place mobile in
the right context.
 CenceMe[2008] –automatically infers someone’s activity.
 Cartel[2006] -Mobile sensing sytem to collect, process, deliver
and visualize sensor data.
 MetroSense[2006]- People centric application for sensing and
communicating information collaboratively using smart phones.
 Second Life of a Sensor[2008]- merges CenceMe and Second
Life to bring real world and virtual world together.

8. Energy Optimization
 The energy consumption varies based on which sensors
are enabled and how long they are switched on.
 Paek[2010]- Rate adaptive positioning system for smart
phone application.
 Zhaung[2010]- Adaptive location sensing framework
includes ( sensing substitution, sensing suppression,
sensing piggybacking and sensing adaptation).
 ENLOC[2009]- energy efficient localization framework.

9. Image Rendering and Modeling
 The availability of camera in mobile phones has enabled a wide
range of image based applications.
 Easy Tour[2004]- image based virtual tour system.
 Teodosio [1993]- builds a panoramic overview from digital video.
 Zheng [2005] obtains complete visual archives of scene with
fewer images.
 Hayashi[1998]- robot navigation in outdoor environments using
GPS information and panoramic views.
 Wagner[2010] creates a panoramic map from live camera stream.
 Snavely[2008] navigate and view large set of unstructured
community or personal photos.
 Crandall[2009] organize a large collection of geo-tagged photos
automatically based on content analysis and structural analysis.

10. Our Approach
 Client Overview.
 Server Overview.
 Client Details.
 Server Details.

11. Client Overview
 Mobile application
 Captures geo-tagged images.
 Transfers these images to server.
 Transfer thread.
 GPS thread.
 Accelerometer reading to
discard motion blurred images.
 Energy efficient algorithm.
 GPS text file.
 Accelerometer text file.
 Transfer text file.

12. Server Overview
 Saves geo-tagged images transferred by each mobile.
 Categorize them into separate clusters.
 K means clustering algorithm approach.
 Fix invalid lat/lon position.
 Nearest neighbor interpolation.
 PTGui stitching software.
 Generates larger panoramic view of each clusters.
 Pano2VR.
 Creates a 360 flash animation.

13. Server Work Flow

14. Client Details
 Mobile Application.
 Data Transfer Thread.
 Power Consumption.

15. Mobile Application
 Hardware/Software
 Windows mobile phone.
 C# on .Net Framework 3.5.
 Application start
 Camera button.
 Starts GPS.
 Starts Camera Dialog box.
 Image Capture- Saves in separate geo-tagged image folder.
 Exit button.
 Intelligent application
 Creates three different text files.
 GPS file to save updated GPS values.
 Transfer file stores the timestamp of last file sent by transfer thread.
 Accelerometer file saves x, y, z values to discard blurred images.
 Saves energy consumption -regular on/off of applications.

16. Application Start

17. Data Transfer Thread
 Timestamp text file
 Saves creation time of last image transferred.
 Used to control which files are transferred.
 Unique identification number of mobile phone
 Sent to the server.
 Creates a new folder for each user.

18. Data transfer thread flow chart

19. Power Consumption
 Power consumption is high.
 GPS- notorious for their high power consumption.
 Wi-Fi- also expensive on battery usage.
 Camera.
 Turns on applications only when required
 GPS turns on only when camera application is on.
 Shuts off when existing camera application.
 Transfer thread connects to wireless network.
 Sleeps until image folder is updated.
 Wakes up to transfer files.

20. Server Details
 Server Application.
 GPS location correction.
 Image clustering.
 Image stitching.

21. Server Application
 Opens a port to connect any number of clients
asynchronously.
 Needs to be on all the time.
 Creates folder for each mobile device with their unique id.
 Checks for availability of the folder.
 Saves all the images as transferred.
 GPS location correction.
 Image Clustering.
 Image stitching- PTGui.

22. GPS Location Correction
 When GPS device is warming up
 Non geo-tagged images are captured.
 Our application fix the geo location of the images based on time gap
at server side or use the last stored geo-location on client application.
 Time gap
 Subtracting the creation time of the bad image from the creation time of
the next best image.
 Needs to be the least.
 Replaces the geo-location with best image location.
 The timestamp remains the same.
 GPS text file
 Stores updated GPS values in a text file.
 Checks this file during GPS warm up time.

23. Image Clustering
 Clusters the group of images.
 Separate folder containing images that are closer to each
other.
 This is how the K means Algorithm works
 Loop K= 1..n.
 Select K coordinates as initial centroid.
 Loop
 Form K clusters by putting all the coordinates that are close to initial
centroid.
 Recalculate the centroid for all K clusters.
 Until The centroid position converges.
 Calculate “error” and pick K with “least” difference.

24. Image Stitching
 Clustering algorithm
 Clusters image data into
separate folders
consisting of images
closer to each other.
 PTGui stitching software
 Stitches each cluster
based on the control
points.

25. Experimental Results and Analysis
 Testing GPS indoors and outdoors.
 Testing camera indoors and outdoors.
 Testing clustering algorithm on GPS coordinates.
 3G and Wi-Fi comparison.
 Evaluation of image stitching.
 Accelerometer Information.
 Battery Testing Experiment.

26. Testing GPS
 Captured 200 geo-tagged images in and around the campus.
 outdoors
 indoors
 Scattered graph below shows
 Closeness of different geo location
 Different Mobile Phones
 The difference of the lat/lon position was 0.0001 degrees.
-94.18595
-94.16595
-94.14595
-94.12595
-94.10595
-94.08595
-94.06595
-94.04595
-94.02595
-94.00595
-93.98595
36.05 36.06 36.07 36.08 36.09 36.1 36.11 36.12 36.13 36.14 36.15
Lat/Lon values of all images taken
Series1
Figure: Lat/Lon values of 400 images taken in and around the campus

27. Testing Camera
 Panoramas are created based on images captured
 Indoors
 Outdoors
 The goal of this experiment
 Image quality
 Image properties
 Camera settings

28. Testing Camera
Figure: Outdoor images captured by camera application.
Figure: Indoor images captured by camera application.

29. Testing clustering algorithm
•Experiment clustering algorithm.
•Different values of K.
•Varied number of iterations.
•Starting centroid point was different on each run.
•Results
•Selected starting point randomly.
•Close images were in same clusters for each run.
•Far images were in different cluster depending on
centroid value selected.

30. 3G and Wi-Fi Comparison
 Experiment - Test transfer rate for sending 100 images
 Using 3G.
 Using Wi-Fi.
 During night and afternoon.
 Results
 Coverage of 3G is much better than Wi-Fi.
 Transfer rate for Wi-Fi is higher than 3G.

31. 3G and Wi-Fi Comparison
Files Min Min/File Tot Bytes Ave Bytes Bytes/Sec
3G Afternoon 25 24.50 0.98 15305984 612239.36 10412.23
50 62.60 1.25 32761686 655233.72 8722.49
75 109.48 1.46 53987331 719831.08 8218.50
100 156.60 1.57 75278144 752781.44 8011.72
Average 1.31 8841.24
3G Night 25 33.73 1.35 15305984 612239.36 7562.25
50 72.25 1.45 32761686 655233.72 7557.48
75 119.28 1.59 53987331 719831.08 7543.29
100 166.38 1.66 75278144 752781.44 7540.63
Average 1.51 7550.91
Wi-Fi Afternoon 25 1.25 0.05 15305984 612239.36 204079.79
50 2.35 0.05 32761686 655233.72 232352.38
75 3.92 0.05 53987331 719831.08 229733.32
100 5.38 0.05 75278144 752781.44 233059.27
Average 0.05 224806.19
Wi-Fi Night 25 1.18 0.05 16073914 642956.56 226393.15
50 2.27 0.05 35010572 700211.44 257430.68

32. Evaluation of Image Stitching
Outdoor Panorama
•Twenty two images combined together to create a panoramic view
• Higher number of images results
•Better quality and much detailed panorama.

33. Indoor Panorama
•Interesting panorama
Hall way being stitched on the ends of the panorama.
Hall way was taken from two different viewpoints.

34. Combining All Images
 Panorama was generated by stitching all sets of images
together
 Results a panoramic view which does not exist.

35. Accelerometer Information
 Image motion blur due to moving camera
 Mobile phones are equipped with accelerometer
 Record accelerometer reading.
 Calculate average motion when capturing an image.
 Classify into two different image categories:
 images without motion blur.
 images with motion blur.
Stationary image
Blurry image
Blur Stationary
Programmatically
result 19 25
Reality 11 33
Blur Stationary
Programmatically
result 16 24
Reality 11 29
Experiment 1 Experiment 2

36. Accelerometer Information
-2000
-1500
-1000
-500
0
500
1000
1
82
163
244
325
406
487
568
649
730
811
892
973
1054
1135
1216
1297
1378
1459
1540
1621
1702
1783
1864
1945
2026
2107
2188
2269
2350
2431
2512
2593
2674
2755
2836
2917
2998
3079
3160
3241
3322
3403
3484
3565
3646
3727
3808
3889
Series1
Series2
Series3

37. Battery Testing Experiment
 Test consumption of energy on
 HTC TILT new and old batteries.
 HTC fuze.
 Eight possible combinations
 Wi-Fi,
 GPS,
 Camera
 Hourly basis experiment.
 Results:
 Highest consumption: GPS,
 Second highest: Wi-Fi,
 Next: Camera,
 All three on: Battery would not last long.
 Fine tuned the algorithm to on the application whenever required.

38. 0
20
40
60
80
100
120
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
Camera GPS Wi-Fi on
Series1 Series2
0
20
40
60
80
100
120
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
GPS On
Series1 Series2
75
80
85
90
95
100
105
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
Backlight ON
Series1 Series2
0
20
40
60
80
100
120
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
Camera On
Series1 Series2

39. Conclusions
 Developed client/server application to capture images and
automatically generate panoramas.
 Energy efficient algorithm to minimize battery usage.
 Nearest neighbor algorithm on server is used to fix invalid
lat/lon positions in geo-tagged images.
 Clustering algorithm is used to organize the images into
separate folders on server.
 Discarded blurred images based on the accelerometer
readings during image capture.
 Panoramas generated by PTGui software.

40. Future Work
 Replace manual image capture with automatic image or
video capture based on the user motion or other inputs.
 Fully automate image stitching process.
 Incorporate knowledge of users connectivity patterns to
schedule image transfer via Wi-Fi instead of 3G.
 Incremental clustering of geo-tagged images.
 Incremental approach to update image panoramas.

41. Q&A