Netflix has open sourced many of our Gradle plugins under the name Nebula. These plugins are there to lend our expertise and experience to building responsible projects, internally and externally. This talk will cover some of the ones we've published, why we want to share these with the community, how we tested and published them, and most importantly how you can contribute back to them.
Nebula started off as a set of strong opinions to make Gradle simple to use for our developers. But we quickly learned that we could use the same assumptions on our open source projects and on other Gradle plugins to make them easy to build, test and deploy. By standardizing plugin development, we've lowered the barrier to generating them, allowing us to keep our build modular and composable.
47. Participating
• Use individual plugins
• Get on nebula-plugins
Google Group
• Move your plugin to
nebula-plugins
• Start a new plugin in
nebula-plugins
I work on the Engineering Tools team at Netflix, where we build tools to support the developers. Today I’m here to talk about Netflix Build Language, or as we like to call it, nebula.
And I’m here to talk about how we try to exist in the open source space with plugins.
We had to transition out of a Ant/Ivy. Primarily a JVM shop, there are a few players in this space for build. A lot of conventions were in place and changing them to fit Maven’s model wasn’t an option.
Look after the users, and where we can add value.
Just couldn’t get the experience as smooth as we wanted it in our enterprise environment. Still haven’t.
Patched Version of Gradle - Patch to cover a bug in Ivy. Patch to expose status/statusScheme.
Custom Distribution - So we can embed some init.d scripts, e.g. add our repository servers and add our plugin’s jar.
Custom Wrapper - To force some variables getting set, like memory defaults or GRADLE_USER_HOME. Plans on customizing to support for re-downloading of static URL
End user sees apply plugin nebula. Nebula extension block to hold other extensions via @Delegate annotation.
NetflixOSS pre-dated our internal work, but it was going down the Maven route. And we had a goal that we wanted to move to Gradle. We also had concerns about how to integration POMs and our internal Ivy files.
That was done in a way to make the Gradle work really obvious, i.e. minimal work hidden in plugins, no custom DSL. It’s hideous at this point and based on Gradle 1.6. We had to rush it, to pre-empt projects going out with Maven. github/cloudbees/bintray project plugin forthcoming.
NetflixOSS fell way behind our internal development. Majority of our work was not Netflix specific, but turned out to just be what a responsible project needs.
Also had growing pains with our plugin, so we wanted to decouple them.
Wanted to make new plugins easy, with CI and SCM. We’re build people, right?
Found that many plugins we wanted to use also suffered from basic release engineering practices, which is clearly ironic given the space we’re in.
Group plugin with SCM, CI and Deployment. Chosen of familiarity, I wish I had looked at more options. Common patterns to give the apply plugin nebula like experience, enabled via nebula-plugin-plugin. Lot of hard coding.
Exists as a dedicated Organization, with project names following a pattern.
We learned from the @NetflixOSS work that GitHub doesn’t maintain itself.
We have to eliminate the manual work. And Jenkins is a great place to do that, except that setting up Jenkins is manual work. I don’t want to fight Jenkins, so we have a script to manage this.
The nebula prefix for highly opinionated plugins. While the gradle prefix is for plugins that have general applicability. Many of them are used in the plugin-plugin since they’re so helpful.
Not really a plugin, but integrated into the Gradle ecosystem. Meant to be used in tests.
* ProjectSpec has its limitation, since you can only really apply plugins. Though you can run a hidden evaluate(). Really fast and recommended when ever possible
PluginProjectSpec is a small addition that just applies your plugin by name, tries it twice and tries it in multi-module project.
IntegrationSpec run a real Gradle build.
All the nebula-plugins use these Test classes, so there’s plenty of examples.
Helper methods to create a project, to run a project, and to evaluate the effects of a execution. Will get it’s own directory to run in.
GradleLauncher is necessary for some in-memory checks or debugging.
Not really a plugin, has tasks and Helper method. Meant for runtime.
AlternativeArchiveTask is to provide an implementation of AbstractArchiveTask, which is not also a CopyTask, since those are greaten special.
beforeEvaluate is actually before after evaluates
getTempDir gives a build directory for a task
addDefaultGroup lets you set a group, optionally. Hard otherwise because project.getGroup() will provide the parent project’s name as the group.
Finally a real plugin. Though docs are the worst of the bunch, this is the most important. Actually a bunch of plugins. All meant to get a publication looking just right, down to the signing.
Artifact plugins attempt to make the jars (which we’d all expect).
Publishing plugins attempt to make resulting Ivy/POM file cleaner, primarily by using resolved variables and including excludes.
nebula-sign looks for magical properties and conditionally signs, unlike other approached.
To support these, we needed to use Publish 2.0, which isn’t fully baked. We were unable to use as is, so we made our CustomComponent. Allows any plugins to contribute artifacts, with more control of the resulting dependencies and confs (more important in Ivy).
Two plugins are available for ivy vs maven, nebula-ivy-publishing and nebula-maven-publishing.
‘info-java' version of java being used
‘info-ci' tells about CI system and current build
‘info-scm' derives info about current SCM
‘info-jar' injects into the Jar’s MANIFEST.MF
‘info-props' creates file with values
‘info-jar-props' puts property file in jar for later retrieval
All go through a broker that you can listen to.
contacts-manifest puts people into the JAR manifest
contacts-pom puts people into developers section of POM
Example of how we believe some plugins can talk to each other without a DSL in the way.
Then we can send emails on release by the notify role.
We believe in a reproducible build for every developer all the time. Dynamic versions make that hard. We can’t imagine manually editing versions. We’ve all had the experience of a new dependency get published and everyone is broken. We also have cases where we want to tweak a specific module, for a patch, which can be done via the command line. Inspired by systems like Bundler.io.
Let some automated job update your dependencies when they’ve proven valid.
Properly sets up classpath.
Does the things we’d expect in a responsible project. Even if they don’t like it.
E.g. doesn’t fail on Javadoc error, maintains status even though the java plugin wants to walk all over it.
Adds OJO and Bintray, with publishing. Plugin Portal additions.
Needs to be extracted out for different orgs, different package names. Very meta, since it applies itself.
Most popular plugin, even though we haven’t advertised them at all. Takes the CopySpec idea and applies it to redline and jdeb.
buildRpm and buildDeb are implicitly created, but can still be configured if needed.
Proposed.
Anytime we find a problem internally, we make a plugin to test the problem and fix it. Roll it out and get happy users. Many times, we’re wrapping another plugin and configuring them with our defaults.
Lots of withType or .all {} calls
Ability to create tasks in reaction
Also try to abstract out logic out of Task, so that it can be called sequentially.
Sometimes we even got a third level
Configure tasks
Reacting to the user before the task graph, forces afterEvaluate’s
NamedDomainObjectSet configuration comes after events, so no ability to re-act, except by name
@Outputs have to be Files, not in-memory String. Evaluation of File names for outputs can be tricky.
Can’t debug tests through Tooling API