Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Apache NiFi Meetup - Introduction to NiFi Registry

1.160 visualizaciones

Publicado el

Introduction to NiFi Registry, a sub-project of Apache NiFi, which provides a central location for storing versioned flows.

Publicado en: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Apache NiFi Meetup - Introduction to NiFi Registry

  1. 1. Introduction to NiFi Registry Bryan Bende & Kevin Doran April 17th 2018
  2. 2. 2 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Background à Most frequently asked question… – How do I deploy my flow? à In theory…. – Avoid long turn around time of deployments – Quickly modify dataflow via NiFi UI à In reality… – Classic development lifecycle (dev -> int -> prod) – Promote changes after successful testing – Hands-off UI in production
  3. 3. 3 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Previous Deployment Approaches à Copy flow.xml.gz between environments – Requires copying entire data flow for any change – Can’t tell what changed, hard to diff if you put in version control – Requires all environments use the same encryption key for sensitive properties à Make templates of process groups – Script creation of template and deployment to next environment – Requires stopping flow and removing components, then re-instantiating template – No easy way to see changes, hard to rollback
  4. 4. 4 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Previous Variable Handling à Need to parametrize values in the flow per environment – Connection strings, URLs, File System paths, etc. à Can set variables in bootstrap.conf – -Dmy.var=foo à Can set a properties file in nifi.properties – nifi.variable.registry.properties=production.properties à Both require command line access à Both require restart of entire NiFi instance to pick up changes
  5. 5. 5 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Enter NiFi Registry à NiFi Registry - sub-project of Apache NiFi – https://github.com/apache/nifi-registry – https://issues.apache.org/jira/projects/NIFIREG à Complimentary application, central location for storage/management of “versioned” resources à Initial capability to store and retrieve “versioned flows” à Integration on NiFi side – Start/Stop version control of a process group – Change version (upgrade/downgrade) – Import new process group from a version
  6. 6. 6 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Enter Variable Registry à Parameterize configuration like connection strings, file paths, etc. à Referenced via Expression language – Kafka Brokers = ${kafka.brokers} à Variables associated with a process group à Right-click on canvas to view variables for current process group à Hierarchical order of precedence, resolve closest reference to component à Editing variables automatically restarts any components referencing the variables! Level 1 Level 2 Vars Vars
  7. 7. 7 © Hortonworks Inc. 2011 – 2018. All Rights Reserved NiFi Registry Terminology à Buckets – Logical place to store versioned items/resources – Permissions assigned at bucket level • READ - Retrieve any item from the bucket • WRITE – Create new items in the bucket à Versioned Flows – Currently the only type of item – Each versioned flow has name, description, and 1 or more “snapshots” à Versioned Flow Snapshot – Each snapshot has metadata and contents – Metadata contains a one-up version #, commit message, author, & create date – Content contains a Versioned Process Group i.e. snapshot of PG from NiFi
  8. 8. 8 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Architecture à Metadata Database – Metadata for each bucket – Knowledge of which items are in each bucket – Metadata for each versions of an item – Currently uses embedded H2 DB à Flow Persistence Provider – Stores/retrieves serialized bytes of a versioned flow – Pluggable extension point – Provided implementation uses local file system – NIFIREG-162 Git-based implementation • https://github.com/apache/nifi-registry/pull/112 Client UI (Angular) Server (Spring Boot) REST API (Jersey) Service Layer (Spring, Java) Metadata Database Flow Persistence Provider H2 File System = Extension Point
  9. 9. 9 © Hortonworks Inc. 2011 – 2018. All Rights Reserved StagingDev Deployment Scenario 1 – Shared Registry with Multiple NiFi’s NiFi Registry NiFi NiFi Production NiFi 1. Start version control and save new versions 2. Import flow from Dev and Perform testing 3. Import flow that passed tests in staging LDAP
  10. 10. 10 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Deployment Scenario 2 – Separate Environments StagingDev NiFi Registry NiFi NiFi Production NiFi 1. Start version control and save new versions 3. Import flow version & test 4. Manually move specific version to prod registry NiFi Registry NiFi Registry LDAP LDAPLDAP 2. Manually move specific version to staging registry 5. Import flow version and start
  11. 11. 11 © Hortonworks Inc. 2011 – 2018. All Rights Reserved Tools & Automation à NiPyApi – Python wrapper for NiFi & NiFi Registry REST APIs – https://github.com/Chaffelson/nipyapi à NiFi CLI – Part of nifi-toolkit released with Apache NiFi 1.6.0 – https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli – README file at above location has documentation to get started

×