In this session I'll share my experience integrating Bedework, a CalDAV calendar server, into the Berkeley version of the Sakai OAE. Users can see their calendars inside OAE or in the CalDAV client of their choice (e.g. Apple iCal). The presentation will go into heavy technical detail on the iCalendar data specification, CalDAV, WebDAV, and related protocols. I'll also focus on how to use JUnit to write integration tests that help you learn and exercise complex functionality in external systems, one tiny feature at a time. Audience members should have some background in Java programming and XML.

1. Talk CalDAV To Me: Integrating Bedework into OAE Chris Tweney <chris@media.berkeley.edu> Senior Software Developer Educational Technology Services UC Berkeley June 14, 2011

2. myBerkeley Instance of Sakai OAE with customizations Esp. those that allow advisors to communicate to students

3. myBerkeley widgets

4. myBerkeley widgets

5. myBerkeley Notifications

6. Notification Dependencies

7. Calendaring Fundamentals iCalendar specification iCal4j CalDAV protocol WebDAV protocol

8. iCalendar Simple text-based format for calendar data exchange iCalendar's fundamental objects are: Events (VEVENT) To-dos (VTODO) Journal entries (VJOURNAL) Free-busy info (VFREEBUSY)

9. iCalendar Usually you encounter iCalendar data in the form of files with a “.ics” extension. Here’s a typical (abbreviated) example: BEGIN:VCALENDAR PRODID:-//Ben Fortuna//iCal4j 1.0//EN VERSION:2.0 CALSCALE:GREGORIAN BEGIN:VTODO DTSTAMP:20110603T222847Z DTSTART:20110505T151506 DUE:20110505T151506 SUMMARY:Pay your bill UID:f84347ab-575b-4274-9436-a5ac906381f9 DESCRIPTION:Pay your bill by the deadline of May 5. END:VTODO END:VCALENDAR iCalendar is defined in RFC 5545 http://tools.ietf.org/html/rfc5545

10. iCalendar Parts

11. ical4j Open-source Java library to wrangle messy iCalendar data iCalendar records with time zone data get very complicated; ical4j makes it pretty simple to work with them ical4j home page: http://wiki.modularity.net.au/ical4j/index.php?title=Main_Page

12. ical4j Java Example protected Calendar buildVTodo(String summary) { CalendarBuilder builder = new CalendarBuilder(); Calendar calendar = new Calendar(); calendar.getProperties().add(newProdId("-//Ben Fortuna//iCal4j 1.0//EN")); calendar.getProperties().add(Version.VERSION_2_0); calendar.getProperties().add(CalScale.GREGORIAN); TimeZoneRegistry registry = builder.getRegistry(); VTimeZonetz = registry.getTimeZone("America/Los_Angeles").getVTimeZone(); calendar.getComponents().add(tz); DateTime due = new DateTime(DateUtils.addDays(new Date(), new Random().nextInt(28))); VToDovtodo = new VToDo(due, due, summary); vtodo.getProperties().add(newUid(UUID.randomUUID().toString())); vtodo.getProperties().add(CalDavConnector.MYBERKELEY_REQUIRED); vtodo.getProperties().add(newDescription("this is the description, it is long enough to wrap at the ical specified standard 75th column")); vtodo.getProperties().add(Status.VTODO_NEEDS_ACTION); calendar.getComponents().add(vtodo); return calendar; }

13. ical4j pitfall: Line folding Per the iCalendar RFC, long lines of text get newlines inserted at the 75th column Sadly, ical4j does not handle this by default Need to specify “ical4j.unfolding.relaxed=true” in an ical4j.properties file

14. CalDAV Dialect of WebDAV that provides calendar functionality Much syntax and structure is inherited from its underlying specifications: WebDAV and iCalendar CalDAV is defined in RFC 4791: http://tools.ietf.org/html/rfc4791

15. Sakai 2 CalDAV GA Tech had a CalDAV project in Sakai 2 that was never released due to gaps in Zimbra's API Zach Thomas left excellent docs that hugely helped our efforts Sakai 2 CalDAV Doc Page: https://confluence.sakaiproject.org/display/CALDAV/Developer's+Guide

16. CalDAV, WebDAV, HTTP

17. Learn by snooping

18. Speaking CalDAV with curl Doing a PROPFIND is a shortcut way to get a user's whole calendar: curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/ To curl, "-X PROPFIND" means do an HTTP PROPFIND method. HTTP has several funky methods you've never heard of unless you've worked with WebDAV.

19. Speaking CalDAV with curl curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/ HTTP/1.1 207 Multi-Status <?xml version="1.0" encoding="UTF-8" ?> <DAV:multistatus xmlns:DAV="DAV:" xmlns="urn:ietf:params:xml:ns:caldav" xmlns:ical="http://www.w3.org/2002/12/cal/ical#"> <DAV:response> <DAV:href>/ucaldav/user/300846/calendar/00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:href> <DAV:propstat> <DAV:prop> <DAV:getcontenttype>text/calendar; charset=UTF-8</DAV:getcontenttype> <DAV:getcontentlength>339</DAV:getcontentlength> <DAV:getlastmodified>Tue, 31 May 2011 21:23:32 +0000</DAV:getlastmodified> <DAV:displayname>00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:displayname> <DAV:getetag>"20110531T212332Z-0"</DAV:getetag> <DAV:current-user-principal> <DAV:href>/ucaldav/principals/users/admin/</DAV:href> </DAV:current-user-principal> <DAV:resourcetype/> <DAV:getcontentlanguage>en</DAV:getcontentlanguage> <DAV:creationdate>20110531T212332Z</DAV:creationdate> </DAV:prop> <DAV:status>HTTP/1.1 200 ok</DAV:status> </DAV:propstat> </DAV:response> … more entries snipped … </DAV:multistatus>

20. CalDAV and WebDAV challenges Not your everyday HTTP request Weird methods (PROPFIND, OPTIONS, PUT, etc) Multi-status responses (HTTP 207)

21. Fortunately… Jackrabbit, on which OAE is based, ships with a reasonable WebDAV Java client We can use this to make our basic CalDAV connections

22. CalDavConnector.getCalendarUris() getCalendarUris() is our simplest method Just return URI wrapper objects for a user's whole calendar

23. Complexity's a problem CalDAV and WebDAV are Complicated Not noob-friendly RFCs are almost the only docs available

24. Impatience is a virtue Need lots of trial and error Redeploying a server takes 60-90s Running a test takes 2s 2s is too little time to get distracted

25. Integration-test-driven development JUnit to the rescue! "Unit" tests that actually talk to a running Bedework server are the solution

26. Write the test first… Before doing anything else I write a test: public class CalDavConnectorTest() { @Test() public void getCalendars() throws CalDavException { List<CalendarURI> uris = this.calDavConnector.getCalendarUris(); } }

27. …Then make it pass Now we'll make it pass in the hackiest way imaginable public class CalDavConnector() { public void getCalendars() throws CalDavException { return new ArrayList<CalendarURI>(); } }

28. …Iterate until done Keep adding implementation code until it does what's needed: PropFindMethod propFind = executeMethod(new PropFindMethod(this.userHome.toString())); MultiStatusResponse[] responses = propFind.getResponseBodyAsMultiStatus().getResponses(); for (MultiStatusResponse response : responses) { if (response.getHref().endsWith(".ics")) { Status[] status = response.getStatus(); if (status.length == 1 && status[0].getStatusCode() == HttpServletResponse.SC_OK) { DavPropertySet propSet = response.getProperties(HttpServletResponse.SC_OK); DavProperty etag = propSet.get(DavPropertyName.GETETAG); try { CalendarURI calUri = new CalendarURI( new URI(this.serverRoot, response.getHref(), false), etag.getValue().toString()); uris.add(calUri); } catch (ParseException pe) { throw new CalDavException("Invalid etag date", pe); } } } }

29. Unit testing heresy These tests talk to and expect a running Bedework server This makes them heretical according to true unit test dogma Tests are not supposed to require an external server

30. Gracefully failing test This test will succeed even if the Bedework server does not respond (because we catch the IOException): public class CalDavConnectorTest() { @Test() public void getCalendars() throws CalDavException { try { List<CalendarURI> uris = this.calDavConnector.getCalendarUris(); } catch (IOException ioe) { LOGGER.error("Trouble contacting server", ioe); } } }

31. PROPFIND only gives you URIs… curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/ HTTP/1.1 207 Multi-Status <?xml version="1.0" encoding="UTF-8" ?> <DAV:multistatus xmlns:DAV="DAV:" xmlns="urn:ietf:params:xml:ns:caldav" xmlns:ical="http://www.w3.org/2002/12/cal/ical#"> <DAV:response> <DAV:href>/ucaldav/user/300846/calendar/00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:href> <DAV:propstat> <DAV:prop> <DAV:getcontenttype>text/calendar; charset=UTF-8</DAV:getcontenttype> <DAV:getcontentlength>339</DAV:getcontentlength> <DAV:getlastmodified>Tue, 31 May 2011 21:23:32 +0000</DAV:getlastmodified> <DAV:displayname>00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:displayname> <DAV:getetag>"20110531T212332Z-0"</DAV:getetag> <DAV:current-user-principal> <DAV:href>/ucaldav/principals/users/admin/</DAV:href> </DAV:current-user-principal> <DAV:resourcetype/> <DAV:getcontentlanguage>en</DAV:getcontentlanguage> <DAV:creationdate>20110531T212332Z</DAV:creationdate> </DAV:prop> <DAV:status>HTTP/1.1 200 ok</DAV:status> </DAV:propstat> </DAV:response> … more entries snipped … </DAV:multistatus>

32. …which you then must get with a REPORT method curl –u user:pass -X REPORT http://localhost:8080/ucaldav/user/mtwain/calendar/ -d "<?xml version="1.0" encoding="UTF-8"?> <C:calendar-multiget xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:"> <D:prop> <D:getetag/> <C:calendar-data/> </D:prop> <D:href>http://localhost:8080/ucaldav/user/mtwain/calendar/d12ad881-5769-4d17-85ac-fa0a0196ec04.ics </D:href> </C:calendar-multiget>" (Note: Double quotes not escaped for clarity)

33. REPORT method's response <DAV:response> <DAV:href>/ucaldav/user/mtwain/calendar/02bfa2cd-6c04-40f4-93da-896d54fc2987.ics</DAV:href> <DAV:propstat> <DAV:prop> <DAV:getetag>"20110602T211046Z-0"</DAV:getetag> <calendar-data><![CDATA[BEGIN:VCALENDAR PRODID://Bedework.org//BedeWork V3.7//EN VERSION:2.0 BEGIN:VTODO CATEGORIES:MyBerkeley-Required CATEGORIES:MyBerkeley-Archived CREATED:20110601T171120Z DESCRIPTION:test … [ snip ] ]]></calendar-data> </DAV:prop> <DAV:status>HTTP/1.1 200 ok</DAV:status> </DAV:propstat> </DAV:response>

34. Hairy REPORT Syntax CalDAV's REPORT syntax is hairy XML Have to extend Jackrabbit's ReportInfo classes This blog entry by Ricardo Martin Camarero has very useful starter code: http://blogs.nologin.es/rickyepoderi/index.php?/archives/15-Introducing-CalDAV-Part-II.html

35. REPORT is foundation for search REPORT methods with parameters let you search by date

36. Bedework difficulties Bedework's search implementation spotty No support for search on X-props No support for search on CATEGORIES

37. Filtering on myBerkeley server Due to Bedework bugs we're forced to do some filtering on the myBerkeley server side E.g. Required/Not Required fields

38. Caching on myBerkeley server CalendarURI wrapper class URI: Locates calendar Etag: Uniquely identifies its contents (sort of like a SHA hash) Caching not implemented yet, but easy enough since all calendars are keyed by CalendarURI

39. Implementing CalDavProxyServlet Also done with test-driven development Write tests against JSON files that contain servlet's expected inputs When servlet's finished, UI devs use those JSON files as an API reference

40. Improvements to Nakamura's CalendarService Future work: Store and search calendars using Nakamura's CalendarService Add support for VTODO to CalendarService

41. Beyond CalendarService Future: Create a full-blown CalDAV Calendar Provider component for Nakamura Store calendars externally, transparently refer to them within Nakamura code