JavaOne 2009 BoF Presentation Mobile application features typical in smartphones can also be implemented on mass-market featurephones using Java ME and Web 2.0 back-end services. In this presentation, we explore the multimodality and rich user interface of a search-driven portal application written in Java ME and broadly ported to mass-market featurephones. Multimodality enables the user to search, browse and discover using familiar activities on a mobile device - saying a phrase, entering text and snapping a photo. Snippets of Java ME code used to implement voice recording, image capture, location awareness and advanced mapping are presented and analyzed.

1. Smartphone Behavior
on a Featurephone Budget
using Java ME
Gail Rahn Frederick
Mobile Software Architect
Medio Systems

2. What You Will Learn…
> A Definition of Smartphone Behavior
> Developing Smartphone Features in Java ME
 MIDlet Demos
 Code Samples
> Real-World Advice about Coding and Porting
 Smartphone Features for the Mass-Market
 Integrating Java ME and Web 2.0
2

3. Agenda
> Introduction
> Smartphone Features in Mass-Market Devices
using Java ME
 Searchbox Suggestions from Web 2.0 API
 Search by Voice
 Maps with Pan and Zoom
 Search by Mobile Location
> Audience Participation and Q & A
3

4. About the Presenter and the Audience…
INTRODUCTION
4

5. Introduction
About the Presenter
> Gail Rahn Frederick
> Mobile Software Architect at Medio Systems
 Mobile search, discovery and content aggregation
for mobile operators and publishers
 Native Applications and Mobile Web
> My Background in Mobile
 Native mobile applications on 100+ device models
 Java ME applications on 50+ device models
 Mobile web sites targeting 500+ device models
 Distributed in North America and Europe
5

6. Introduction
Audience Participation
> Who are you?
 Mobile developers, executives, operators?
> This BoF Session is Informal
> Audience Participation is Encouraged!
 Share your Java ME Expertise and Industry Views
 Discussion and Q&A at the End
6

7. Smartphone behavior on a featurephone budget…
HOW TO BUILD
SMARTPHONE FEATURES IN
JAVA ME
7

8. Smartphone Features in Java ME
Industry Definition of Terms
> Featurephone
 Low-cost, mass-market mobile phone
 Third-party software isolated to Applications menu
 Less integrated with main phone UI and native OS
 Java ME: MIDP, CLDC
> Smartphone
 Mobile phone with PC-like functionality
 Third-party software is a first-class citizen
 Native OS is open development environment
 Easy(ier) integration of Apps with OS and Web
 Java ME: MIDP, CDC (if device includes a KVM)
8

9. Smartphone Features in Java ME
About the Demo Devices
> Mass-Market Java ME Devices
 SonyEricsson C905
 MIDP 2.1, CLDC 1.1
 MSA (JSR-248)
 Nokia N96
 MIDP 2.1, CLDC 1.1
 MSA (JSR-248)
9

10. Smartphone Features in Java ME
About the Application
> Mobile Search and Discovery
 Multimodal, Search-Driven UX
 Search by text and voice
 Surfaces Web, Information and
Consumables
> Ported to Mass-Market Devices
 MIDP 2.0+ and CLDC 1.1
> Asynchronous Network Layer
 Manages all transactions
 Caches recent network requests
10

11. Smartphone Features in Java ME
The Smartphone Features
> Predictive Searchbox
 Suggestions from Web 2.0 Service
> Multimodality
 Search by Voice
> Maps with Pan, Zoom and Traffic
 Search with Mobile Location
11

12. Smartphone Features in Java ME
Search Suggestions
> How to Integrate Search
Suggestions
 User enters search query above
some character length threshold
 MIDlet requests suggestions
from Web 2.0 Service
 Web Service responds with
suggestions in standard JSON
format
 MIDlet reads JSON response
and displays suggestions UI
12

13. Search Suggestions
Server Side is Web 2.0 Service in JSON
> Sample Request URI for Search Suggestions
 http://example.com/suggest?q=love
> MIME type for JSON responses
 application/x-suggestions+json
> JSON response array contains 4 elements
 Query string – the requested search term
 Array of completions – list of suggested searches
 Array of descriptions – short additional information
 Array of URLs – Instant answer for suggested
search
13

14. Search Suggestions
Sample Server Response in JSON
> Example of Search Suggestions response in
JSON format for query “love”:
[quot;lovequot;,
[quot;YouTube - Love Songquot;,
quot;Lovebug - Wikipediaquot;],
[quot;www.youtube.comquot;,
quot;en.wikipedia.orgquot;],
[quot;http://www.youtube.com/watch?v=MR5xv3pt7KIquot;,
quot;http://en.wikipedia.org/wiki/Lovebugquot;]
]
14

15. Search Suggestions
Development Approach
> Keep the Smarts in the Web Server
 MIDlet displays all data in JSON response
 Use HTTP 1.1 keep-alive
 Minimize connection setup and teardown in Java ME
> Read Entire Response as JSON Array
 Read Response Components as Arrays or Strings
 JSONException thrown for Format Problems
15

16. Search Suggestions
JSON in Java ME
> Helpful JSON References
 Open source library from json.org
 Java ME package is org.json.me
 Sun Developer Network tutorial on reading JSON in
Java ME
16

17. Search Suggestions
Sample Java ME Code
// The 'response' variable is the JSON response body as a String.
try {
JSONArray jsonArray = new JSONArray(response);
String original = jsonArray.getString(0);
JSONArray suggestions = jsonArray.getJSONArray(1);
JSONArray descs = jsonArray.getJSONArray(2);
JSONArray urls = jsonArray.getJSONArray(3);
// Get the values for the first suggestion.
String suggestion = suggestions.getString(0);
String desc = descs.getString(0);
String url = urls.getString(0);
}
catch (JSONException ex) {
// JSON Response is not formatted as expected
}
17

18. Smartphone Features in Java ME
Search by Voice
> How to Search by Voice
 UI Cues to Initiate Voice Search
 Hold SEND and Speak
 Key Event Starts Audio Recording
 Canvas.keyPressed()
 User Speaks into Microphone
 Key Event Stops Audio Recording
 Canvas.keyReleased()
 Audio Recording Uploaded to
Voice-to-Text or Search Service
18

19. Search by Voice
Development Approach
> Requires JSR-135 Mobile Media API with
Audio Capture
> Use a low-bitrate codec
 Minimizes byte size of sound recording
> Audio capture in background thread
 Progress updates sent periodically to UI thread
> Audio capture is byte array
 Audio data uploaded to voice-to-text service
> Voice-to-text service responds with translation
19

20. Search by Voice
Sample Code for Audio Capture, 1 of 2
// Request audio capture with low-bitrate AMR encoding
String playerURL = quot;capture://audio?encoding=amrquot;;
// Instantiate player for audio capture
Player player = Manager.createPlayer(playerURL);
player.realize();
// Get interface for recording audio.
RecordControl recCtrl =
(RecordControl)player.getControl(quot;RecordControlquot;);
// Instantiate byte stream to receive audio data
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
recCtrl.setRecordStream(outStream);
// Start recording audio
recCtrl.startRecord();
this.player.start();
// Notify UI that player started recording.
20

21. Search by Voice
Sample Code for Audio Capture, 2 of 2
// Periodically update UI during audio recording
long startTime = System.currentTimeMillis();
While( …max recording duration unmet and user still recording… ) {
try {
Thread.sleep(PROGRESS_CALLBACK_PERIOD_MS);
}
catch(InterruptedException ex) {}
// ... Update UI for progress of recording
}
// After audio recording completes, release player resources.
recCtrl.commit();
this.player.close();
// Audio data stored in byte array
byte[] audioData = outStream.toByteArray();
21

22. Smartphone Features in Java ME
Maps with Pan and Zoom
> How to Implement Maps
 User presses keys to pan, zoom
and switch map styles
 Maps are a Grid of Tiles
 Each Tile has an X,Y Position
Relative to Map Center
> Caching Network Manager is
Critical for Performance
 Sends Multiple Tile Requests in
One Transaction
 Caches Recent Tile Responses
22

23. Maps with Pan and Zoom
Development Approach
> Maps are a Grid of Tiles
 Map tile is an image
> Each Tile has an X,Y Position Relative to Map
Center
 Tile key is (map centroid, map style, tile X, tile Y)
 Sent to mapping server to identify requested tile
> Asynchronous and Caching Network Manager
 Sends Multiple Tile Requests in One Transaction
 Caches Recent Tile Responses
23

24. Maps with Pan and Zoom
Design Approach
Tiles Currently Displayed in MIDlet
Tiles in Network Cache
Tiles to be Requested On Demand
24

25. Smartphone Features in Java ME
Search with Mobile Location
> How to Search with Mobile Location
 User interacts with MIDlet
 MIDlet background thread registers
for location updates
 When location data is available,
MIDlet incorporates into server
request
 Location fix in 5 seconds or 15 minutes
 Transparent to User
 User can disable LBS in Preferences
25

26. Search with Mobile Location
Development Approach
> MIDlet requests location using JSR-179
> Location requests and callbacks in background
> Last known location cached and re-used
 Within some time threshold
> Gotchas?
 Phones used in inconvenient locations for LBS
 Indoors, high floors, etc.
 MIDlet must degrade gracefully and eventually stop trying.
26

27. Search with Mobile Location
Sample Java ME Code
LocationProvider m_locProvider;
// Use default criteria for greatest chance of success.
m_locProvider = LocationProvider.getInstance(null);
// Ask for last known location. If provided, may be stale.
Location loc = m_locProvider.getLastKnownLocation();
// Use default timeout for location lookup.
// This call is blocking – call in dedicated thread
loc = m_locProvider.getLocation(-1);
// Method is data source: cell ID, GPS, network-assisted, etc.
int method = loc.getLocationMethod();
// Get lat/lon co-ordinates and incorporate into search query.
double latitude = loc.getQualifiedCoordinates().getLatitude();
double longitude = loc.getQualifiedCoordinates().getLongitude();
27

28. Smartphone Features in Java ME
Request API Permissions in JAD
> Advanced Java ME APIs are Protected
> Request API Permissions in the JAD
> MIDlet Signing Reduces User Permission Dialogs
 Operator, Java Verified and Third-Party Signatures
MIDlet-Permissions:
javax.microedition.io.Connector.http,
javax.microedition.media.control.RecordControl,
javax.microedition.pim.ContactList.read,
javax.microedition.io.Connector.file.read,
javax.microedition.io.Connector.file.write
Text formatting added for readability. MIDlet-Permissions is one line in JAD file.
28

29. Smartphone Features in Java ME
Related APIs for Smart Features in Java ME
> JSR-256: Mobile Sensor API
 Uses General Connection Framework to access
sensor hardware on a mobile device
 Accelerometer, battery charge state, network field
intensity, etc.
> JSR-172: Web Services API
 Web service client for Java ME with XML Parser
> Mobile AJAX in Java ME
 Sun Technical Paper
 Info and Code Samples @ meapplicationdevelopers.dev.java.net
 Streaming ATOM parser, expression language, async I/O library
29

30. Smartphone Features in Java ME
Key Points
> MIDlets with Web 2.0 and other smart features are
possible in Java ME on mass-market devices.
 MSA and MSA Subset and open source libraries.
> Asynchronous network manager with caching is
critical to supports Web 2.0 functionality.
30

31. Smartphone Features in Java ME
Audience Participation
> How do you define featurephone and
smartphone?
 What is the role of Java ME in each ecosystem?
> What are your challenges to implement
smartphone features in Java ME?
 Permissions, OEM Support for Optional APIs,
Fragmentation, etc…
31

32. Gail Rahn Frederick
Medio Systems
gail@medio.com
32