Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Getting started contributing to Apache Spark


Eche un vistazo a continuación

1 de 59 Anuncio

Getting started contributing to Apache Spark

Descargar para leer sin conexión

Are you interested in contributing to Apache Spark? This workshop and associated slides walk through the basics of contributing to Apache Spark as a developer. This advice is based on my 3 years of contributing to Apache Spark but should not be considered official in any way.

Are you interested in contributing to Apache Spark? This workshop and associated slides walk through the basics of contributing to Apache Spark as a developer. This advice is based on my 3 years of contributing to Apache Spark but should not be considered official in any way.


Más Contenido Relacionado

Presentaciones para usted (20)

Similares a Getting started contributing to Apache Spark (20)


Más reciente (20)

Getting started contributing to Apache Spark

  1. 1. Effectively Contributing to Apache Spark Beyond the Code Toronto 2016 This talk (as with all) represents my own personal views may not reflect that of the project. I am not a Spark committer - but I’ve been contributing for 3 years
  2. 2. Who am I? Holden ● Prefered pronouns: she/her ● Co-author of the Learning Spark & High Performance Spark books ● Software Engineer at IBM’s Spark Technology Center ● 100+ Spark Commits ● @holdenkarau ● ●
  3. 3. What we are going to explore together! Getting a change into Apache Spark & the components involved: ● Different ways to contribute ● Places to find things to contribute ● Tooling around code & doc contributions
  4. 4. How can we contribute to Spark? ● Direct code in the Apache Spark code base ● Code in packages built on top of Spark ● Yak shaving (aka fixing things that Spark uses) ● Documentation improvements & examples ● Books, Talks, and Blogs ● Answering questions (mailing lists, stack overflow, etc.)
  5. 5. Which is right for you? ● Direct code in the Apache Spark code base ○ High visibility, some things can only really be done here ○ Can take a lot longer to get changes in ● Code in packages built on top of Spark ○ No real review (+/-) ○ Really great for things like formats or standalone features ● Yak shaving (aka fixing things that Spark uses) ○ Super important to do sometimes - can take even longer to get in
  6. 6. Which is right for you? (continued) ● Documentation improvements & examples ○ Lots of places to contribute - mixed visibility - large impact ● Books, Talks, and Blogs ○ The documentation version of Spark Packages (e.g. no need for community review) ○ Can be high visibility ○ Talk to me if you are thinking of writing a technical book :)
  7. 7. But before we get too far: ● Spark wishes to maintain compatibility between releases ● 2.0 just shipped - so most APIs should be stable ○ Notable exceptions include Structured Streaming ● It’s very important to talk about large code changes with the key members before doing them ○ dev list is the simplest way of reaching out ○ Wonder who the key members are? Check the component maintainers on
  8. 8. Adventure path 1: Direct to Spark ● Maybe we encountered a bug we want to fix ● Maybe we’ve got a feature we want to add ● Either way we should see if other people are doing it ● And if what we want to do is complex, it might be better to find something simple to start with ● It’s dangerous to go alone - take this uting+to+Spark
  9. 9. Getting the code This step can take some time - especially over conference WiFi so let's get it started now :) Conference WiFi isn’t working out for you? Ask me and I can make a copy of the repo to a USB stick for you :)
  10. 10. Exercise time Photo by recastle
  11. 11. Spark’s Github (Exercise 1) ● ● Make a fork of it ● Clone it locally
  12. 12. JIRA - Issue tracking funtimes ● It’s like bugzilla or fog bugz ● There is an Apache JIRA for all Apache projects ● You can (and should) sign up for an account ● All changes in Spark (now) require a JIRA ● ● Check it out at: ○
  13. 13. The different pieces of Spark Apache Spark SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Graph X MLLib Community Packages
  14. 14. The different pieces of Spark: 2.0+ Apache Spark SQL & DataFrames Streaming Language APIs Scala, Java, Python, & R Graph Tools Spark ML bagel & Graph X MLLib Community Packages Structured Streaming
  15. 15. What we can do with ASF JIRA? ● Search for issues (remember to filter to Spark project) ● Create new issues ○ search first to see if someone else has reported it ● Comment on issues to let people know we are working on it ● Ask people for clarification or help ○ e.g. “Reading this I think you want the null values to be replaced by a string when processing - is that correct?” ○ @mentions work here too
  16. 16. What can’t we do with ASF JIRA? ● Assign issues (to ourselves or other people) ○ In lieu of assigning we can “watch” & comment ● Post long design documents (create a Google Doc & link to it from the JIRA) ● Tag issues ○ While we can add tags, they often get removed
  17. 17. Finding a good “starter” issue: ● There are explicit starter tags in JIRA we can search for ● But often the starter tag isn’t applied ● Read through and look for simple issues ● Pick something in the same component you eventually want to work in ○ And or consider improving the non-Scala language API for the component(s) you want to work on. ● Look at the reporter and commenters - is there a committer or someone whose name you recognize? ● Leave a comment that says you are going to start working on this
  18. 18. Exercise 2: Find an issue you want to work on Also grep for TODO in components you are interested in (e.g. grep -r TODO ./python/pyspark or grep -R TODO ./core/src) Look between language APIs and see if anything is missing that you think is interesting - .apache.spark.package Feel free to work in groups :)
  19. 19. Exercise 3a: Building Spark ./build/sbt or ./build/mvn Working in Python? Make sure to build the package target so your Python code will run :) You can quickly verify build with the Spark Shell :)
  20. 20. What about documentation changes? ● Still use JIRAs to track ● We can’t edit the wiki :( ● But a lot of documentations lives in docs/*.md
  21. 21. Exercise 3b: Building Spark’s docs ./docs/ has a lot of info - but quickly: SKIP_API=1 jekyll build SKIP_API=1 jekyll serve --watch
  22. 22. Finding your way around the code ● Organized into sub-projects by directory ● IntelliJ is very popular with Spark developers ○ The free version is fine ● Some people like using emacs + ensime or magit too ● Language specific code is in each sub directory
  23. 23. Testing the issue The spark-shell can often be a good way to verify the issue reported in the JIRA is still occurring and come up with a reasonable test. Once you’ve got a handle on the issue in the spark-shell (or if you decide to skip that step) check out ./[component]/src/test for Scala or doctests for Python
  24. 24. After we get our code working (or even better while we work on it) ● Remember to follow the style guides ○ ide ● Please always add tests ○ For development we can run scala test with “sbt [module]/testOnly” ○ In python we can specify module with ./python/run-tests ● ./dev/lint-scala & ./dev/lint-python check for some style ● Changing the API? Make sure we pass MiMa! ○ Sometimes its OK to make breaking changes, and MiMa can be a bit overzealous so adding exceptions is common
  25. 25. A bit more on MiMa ● Spark wishes to maintain binary compatibility ○ in non-experimental components ● MiMa exclusions can be added if we verify (and document how we verified) the compatibility ● Often MiMa is a bit over sensitive so don’t feel stressed - feel free to ask for help if confused
  26. 26. Exercise 4: Open your editors No arguing about which editor please - kthnx Making a doc change? Look inside docs/*.md Making a code change? grep or intellij or github inside project codesearch can all help you find what you're looking for.
  27. 27. Yay! Let’s make a PR :) ● Push to your branch ● Visit github ● Create PR (put JIRA name in title as well as component) ○ Components control where our PR shows up in ● If you’ve been whitelisted tests will run ● Otherwise will wait for someone to verify ● Tag it “WIP” if its a work in progress (but maybe wait) [puamelia]
  28. 28. Code review time ● Note: this is after the pull request creation ● I believe code reviews should be done in the open ○ With an exception of when we are deciding if we want to try and submit a change ○ Even then should have hopefully decided that back at the JIRA stage ● My personal beliefs & your org’s may not align Mitchell Joyce
  29. 29. And now onto the actual code review... ● Most often committers will review your code (eventually) ● Other people can help too ● People can be very busy (check the release schedule) ● If you don’t get traction try pinging people ○ Me ( @holdenkarau - I can’t merge your code but I can take a look) ○ The author of the JIRA (even if not a committer) ○ The shepherd of the JIRA (if applicable) ○ The person who wrote the code you are changing (git blame) ○ Active committers for the component Mitchell Joyce
  30. 30. What does the review look like? ● LGTM - Looks good to me ○ Individual thinks the code looks good - ready to merge (sometimes LGTM pending tests or LGTM but check with @[name]). ● SGTM - Sounds good to me (normally in response to a suggestion) ● Sometimes get sent back to the drawing board ● Not all PRs get in - its ok! ○ Don’t feel bad & don’t get discouraged. ● Mixture of in-line comments & general comments
  31. 31. That’s a pretty standard small PR ● It took some time to get merged in ● It was fairly simple ● Review cycles are long - so move on to other things ● Only two reviewers ● Apache Spark Jenkins comments on build status :) ○ “Jenkins retest this please” is great ● Big PRs - like making PySpark pip installable can have > 10 reviewers and take a long time ● Sometimes it can be hard to find reviewers - tag your PRs & ping people on github
  32. 32. Don’t get discouraged David Martyn Hunt It is normal to not get every pull request accepted Sometimes other people will “scoop” you on your pull request Sometimes people will be super helpful with your pull request
  33. 33. So who was that “Spark QA”? ● Automated pull request builder ● Jenkins based ● Runs all of the tests & style checks ● Lives in Berkeley ● Test logs live on, artifacts not so much ●
  34. 34. Some changes require even more testing ● spark-perf (common for ML changes) ● spark-sql-perf (common for SQL changes) ● spark-integration-tests (integration testing)
  35. 35. While we are waiting: ● Keep merging in master when we get out of sync ● If we don’t jenkins can’t run :( ● We get out of sync surprisingly quickly! ● If our pull request gets older than 30 days it might get auto-closed
  36. 36. So review: Where do we get started? ● Search for “starter” on JIRA ● Look on the mailing list for problems ● Stackoverflow - lots of questions some of which are bugs ● grep TODO broken FIXME ● Compare APIs between languages ● Customer/user reports?
  37. 37. But what about when we want to make big changes? ● Talk with the community ○ Developer mailing list ○ User mailing list ● Create a public design document (google doc normally) ● Consider if it can be published as a spark-package instead
  38. 38. Other resources: ● “Contributing to Apache Spark” - uting+to+Spark ● Programming guide (along with JavaDoc, PyDoc, ScalaDoc, etc.) ○
  39. 39. What about creating a package? ● Relatively simple - need to publish to maven central ● Listed on ● Cross building (Spark versions) not super easy ● If your building with sbt check out to make it easy to publish ● Used to do API compatibility checks ● Sometimes flakey - just republish if it doesn’t go through
  40. 40. Signing your packages ● Required ● Can be a bit odd (sbt-pgp plugin has issues sometimes with keys with passphrases)
  41. 41. What things can be good Spark packages? ● Input formats (especially Spark SQL, Streaming) ● Machine learning pipeline components & algorithms ● Testing support ● Monitoring data sinks ● Deployment tools
  42. 42. How about writing a book? ● Can be lots of fun ● Can also take up 100% of your “free” time ● Can get you invited to more nerd parties ● Most of the publisher are looking to improve/broaden their Spark book line up ● Like an old book that hasn’t been updated? Talk to the publisher about updating it.
  43. 43. Spark Videos ● Apache Spark Youtube Channel ● My Spark videos on YouTube - ○ ● Spark Summit 2014 training ● Paco’s Introduction to Apache Spark
  44. 44. Learning Spark Fast Data Processing with Spark (Out of Date) Fast Data Processing with Spark (2nd edition) Advanced Analytics with Spark Coming soon: Spark in Action
  45. 45. Learning Spark Fast Data Processing with Spark (Out of Date) Fast Data Processing with Spark (2nd edition) Advanced Analytics with Spark Coming soon: Spark in Action Early Release High Performance Spark
  46. 46. And the next book….. First five chapters are available in “Early Release”*: ● Buy from O’Reilly - Get notified when updated & finished: ● ● * Early Release means extra mistakes, but also a chance to help us make a more awesome book.
  47. 47. And some upcoming talks: ● September ○ This meetup (yay)! ○ Toronto - Beyond the Code (Contributing to Spark) ○ New York City Strata Conf (Structured Streaming & Machine Learning) ● October ○ PyData DC - Making Spark go fast in Python (vroom vroom) ○ Salt Lake City Spark Meetup - TBD ○ London - OSCON - Getting Started Contributing to Spark ● December ○ Strata Singapore (Introduction to Datasets)
  48. 48. k thnx bye! If you care about Spark testing and don’t hate surveys: Will tweet results “eventually” @holdenkarau Any PySpark Users: Have some simple UDFs you wish ran faster you are willing to share?: Pssst: Have feedback on the workshop? Give me a shout ( if you feel comfortable doing so :)