Presentation at the IEEE conference on ongoing work developing new techniques for navigating the past of place using Kinect

1. Interactive Palimpsests
Dr Jerry Fishenden
Senior Research Fellow, Centre for Creative Computing
Bath Spa University
j.fishenden@bathspa.ac.uk

2. objective
to explore and refine computational techniques
and interfaces that enable users to better
explore and interact with the past of place –
visually and aurally
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

3. n-tier navigation
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

4. user experience testing

5. initial visual navigation controls
palimpsest navigator lens
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

6. palimpsest slider control
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

7. 2nd generation techniques
the cursor as palimpsest navigator
lens
dynamically re-sizeable lens
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

8. examples
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden
http://voetek.com/palimpsests/dmusquaremile/lens.html
http://voetek.com/palimpsests/dmusquaremile/slider.html

9. IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

10. if (test == 0)
{
int SensorDistance = e.Value;
Palimpsest_Opacity = 1 - ((float)SensorDistance / 100); //near=past, far=present
image1.Dispatcher.BeginInvoke(DispatcherPriority.Normal, (Action)(() => {
image1.Opacity = Palimpsest_Opacity; }));
}
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

11. IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

12. HandCursorElement.Visibility = Visibility.Visible;
DepthImagePoint point =
this._KinectDevice.CoordinateMapper.MapSkeletonPointToDepthPoint(hand.Position,
this._KinectDevice.DepthStream.Format);
point.X = (int) ((point.X *
LayoutRoot.ActualWidth/_KinectDevice.DepthStream.FrameWidth) -
(HandCursorElement.ActualWidth / 2.0));
point.Y = (int)((point.Y * LayoutRoot.ActualHeight /
_KinectDevice.DepthStream.FrameHeight) - (HandCursorElement.ActualHeight / 2.0));
Canvas.SetLeft(HandCursorElement, point.X);
Canvas.SetTop(HandCursorElement, point.Y);
txtStatus.Text = "X is: " + point.X;
image1.Opacity = point.X / LayoutRoot.ActualWidth;
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

13. video
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

14. IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

15. HandCursorElement.Visibility = Visibility.Visible;
DepthImagePoint point =
this._KinectDevice.CoordinateMapper.MapSkeletonPointToDepthPoint(hand.Position,
this._KinectDevice.DepthStream.Format);
point.X = (int) ((point.X * LayoutRoot.ActualWidth/_KinectDevice.DepthStream.FrameWidth) -
(HandCursorElement.ActualWidth / 2.0));
point.Y = (int)((point.Y * LayoutRoot.ActualHeight / _KinectDevice.DepthStream.FrameHeight) -
(HandCursorElement.ActualHeight / 2.0));
Canvas.SetLeft(HandCursorElement, point.X);
Canvas.SetTop(HandCursorElement, point.Y);
txtStatus.Text = "X is: " + point.X;
bigScene.Visibility = Visibility.Visible;
Ellipse.Center = new Point(point.X, point.Y);
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

16. private void CreateGrammars(RecognizerInfo ri)
{
//user oral instruction format is {“use lens/slider”}, {“location London/Leicester”}
var technique = new Choices();
technique.Add("lens");
technique.Add("slide");
technique.Add("London");
technique.Add("Leicester");
var create = new Choices();
create.Add("location");
create.Add("use");
var gb = new GrammarBuilder();
gb.Culture = ri.Culture;
gb.Append(create);
gb.AppendWildcard();
gb.Append(technique);
var g = new Grammar(gb);
_sre.LoadGrammar(g);
}
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

17. video
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

18. futures
• tracking the distance of a user from the sensor:
– growing or shrinking the size of the lens depending on whether
someone's hand moves closer to or further from the screen
• addition of audio that will come to the foreground as the
lens passes over particular visual items
– for example, emphasising the sound of children playing as the
lens passes over children in the images, or emphasising the
sound of horses hooves when passing over a horse and cart in
the earlier underlying image
• “competitive strategies” for handling multiple users in front
of the Kinect / screen
• continuing user experience testing, feedback and
refinement
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

19. recent work
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

20. IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

21. references / reading
• Fishenden, J and Hugill, A. Palimpsests of Time
and Place. 2011. IEEE COMPSAC
• Webb, J and Ashley, J. Beginning Kinect
Programming with the Microsoft Kinect SDK.
2012. Apress
• Phidgets http://www.phidgets.com
• research site at http://fishenden.com
IEEE ISCC 2014: interactive palimpsests / Jerry Fishenden

22. Interactive Palimpsests
Dr Jerry Fishenden
Senior Research Fellow, Centre for Creative Computing
Bath Spa University
j.fishenden@bathspa.ac.uk