Using Docker, Neo4j, and Spring Cloud for Developing Microservices
•
7 recomendaciones•1,894 vistas
This document outlines a presentation about using Docker, Neo4j, and Spring Cloud for developing microservices. Specifically, it discusses using these technologies to build an application that ranks Twitter profiles using the PageRank algorithm. It provides code examples and descriptions of how to connect Neo4j and Apache Spark to submit PageRank jobs, build Spring Data repositories and domain models, connect to the Twitter API, and create a ranking dashboard.
Recomendados
Recomendados
Más contenido relacionado
Destacado
Similar a Using Docker, Neo4j, and Spring Cloud for Developing Microservices
Similar a Using Docker, Neo4j, and Spring Cloud for Developing Microservices (20)
Más de Kenny Bastani
Más de Kenny Bastani (9)
Último
Último (20)
Using Docker, Neo4j, and Spring Cloud for Developing Microservices
- 1. Unless otherwise indicated, these slides are © 2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Using Docker, Neo4j, and Spring Cloud for Developing Microservices Kenny Bastani, Spring Developer Advocate, Pivotal @kennybastani
- 3. Ranking Twitter Profiles Using PageRank 3 https://github.com/kbastani/spring-boot-graph-processing-example
- 9. https://github.com/kbastani/spring-boot-graph-processing-example Tools we’ll be using ! Spring Boot ! Neo4j ! Apache Spark ! Docker ! RabbitMQ 9
- 10. https://github.com/kbastani/spring-boot-graph-processing-example Containerize all the things! 10
- 15. Connecting Neo4j and Apache Spark …to submit PageRank jobs 15 https://github.com/kbastani/spring-boot-graph-processing-example
- 16. https://github.com/kbastani/spring-boot-graph-processing-example Request new Apache Spark job 16 Algorithm Type Relationship Type
- 17. https://github.com/kbastani/spring-boot-graph-processing-example Export Neo4j graph to HDFS 17 New Job Request HDFS Path: /../../graph.csv Job Type: pagerank
- 18. https://github.com/kbastani/spring-boot-graph-processing-example Encoding a graph as an edge list 18 edge list G B H B I B K B E B F B J B D B G E H E I E K E …
- 19. https://github.com/kbastani/spring-boot-graph-processing-example Import edge list to Apache Spark 19 Process Job Request HDFS Path: /../../graph.csv Job Type: pagerank graph.csv 0 1 1 2 2 3 0 3 2 1 …
- 20. https://github.com/kbastani/spring-boot-graph-processing-example Apply results back to Neo4j 20 Completed Job HDFS Path: /../../results.csv Job Type: pagerank results.csv 0 .56 1 .42 2 .14 3 .25 …
- 21. https://github.com/kbastani/spring-boot-graph-processing-example Graph processing platform 21 Algorithm Type Relationship Type
- 26. 26 Creating Spring Data Neo4j Repositories https://github.com/kbastani/spring-boot-graph-processing-example
- 27. https://github.com/kbastani/spring-boot-graph-processing-example What our application needs ! Repositories • User repository (to manage and import users) • Follows repository (to manage and import following relationships) • Custom Cypher queries mapped to repository methods ! Domain model • User — (our node entity) • Follows — (our relationship entity) ! REST API • Expose the resources of the domain as a REST API 27
- 28. Creating Spring Data Neo4j Repositories 28 https://github.com/kbastani/spring-boot-graph-processing-example
- 29. Creating Spring Data Neo4j Repositories 29 https://github.com/kbastani/spring-boot-graph-processing-example
- 30. Creating Spring Data Neo4j Repositories 30 https://github.com/kbastani/spring-boot-graph-processing-example
- 31. Creating Spring Data Neo4j Repositories 31 https://github.com/kbastani/spring-boot-graph-processing-example
- 32. Creating Spring Data Neo4j Repositories 32 https://github.com/kbastani/spring-boot-graph-processing-example
- 33. Creating Spring Data Neo4j Repositories 33 https://github.com/kbastani/spring-boot-graph-processing-example
- 34. Exposing repository APIs using Spring Data REST 34 https://github.com/kbastani/spring-boot-graph-processing-example
- 35. Exposing repository APIs using Spring Data REST 35 https://github.com/kbastani/spring-boot-graph-processing-example
- 36. Exposing repository APIs using Spring Data REST 36 https://github.com/kbastani/spring-boot-graph-processing-example
- 37. Connecting to the Twitter API 37 https://github.com/kbastani/spring-boot-graph-processing-example
- 38. 38 https://github.com/kbastani/spring-boot-graph-processing-example Connecting to the Twitter API
- 39. Connecting to the Twitter API 39 https://github.com/kbastani/spring-boot-graph-processing-example
- 40. Connecting to the Twitter API 40 https://github.com/kbastani/spring-boot-graph-processing-example We can override these properties as environment variables at runtime
- 41. 41 https://github.com/kbastani/spring-boot-graph-processing-example Connecting to the Twitter API
- 42. 42 https://github.com/kbastani/spring-boot-graph-processing-example Connecting to the Twitter API
- 43. Scheduling new PageRank jobs 43 https://github.com/kbastani/spring-boot-graph-processing-example
- 44. https://github.com/kbastani/spring-boot-graph-processing-example Scheduling PageRank jobs from Neo4j 44
- 45. Scheduling new PageRank jobs 45 https://github.com/kbastani/spring-boot-graph-processing-example
- 50. Adding static web content 50 https://github.com/kbastani/spring-boot-graph-processing-example
- 55. Creating Spring Data Neo4j Repositories 55 https://github.com/kbastani/spring-boot-graph-processing-example
- 56. https://github.com/kbastani/spring-boot-graph-processing-example Discover new users and update rankings 56
- 57. https://github.com/kbastani/spring-boot-graph-processing-example57 Learn More. Stay Connected. ! Twitter: @kennybastani ! Spring: spring.io/team/kbastani ! Blog: kennybastani.com ! GitHub: github.com/kbastani Twitter: twitter.com/springcentral YouTube: spring.io/video LinkedIn: spring.io/linkedin Google Plus: spring.io/gplus
Notas del editor
- Hi my name is Kenny Bastani. I am a Spring Developer Advocate at Pivotal. Today I will be introducing you to a reference architecture for creating a PageRank analytics platform using Spring Boot microservices.
- A little bit about me. Again, I’m a Developer Advocate on the Spring team at Pivotal. I’m currently writing a book about Cloud Native JVM applications with Josh Long. The early release is available today from O’Reilly media.
- Let’s do an overview of the problem we will solve as a part of our reference application. The problem we’re going to solve is how to discover influencers on Twitter using a set of seed profiles as inputs. To solve this problem without a background in machine learning or social network analytics might be a bit of a stretch, but we’re going to take a stab at it using a little bit of computer science history.
- The PageRank algorithm, created by Google co-founder Larry Page, was first used by Google to rank website documents from analyzing the graph of backlinks between sites. I dug up the original research paper on PageRank from Stanford for some inspiration. In the paper, the authors talk about the notion of approximating the "importance" of an academic publication by weighting the value of its citations. In the paper I found a quote which best summarized how the PageRank algorithm works. The quote reads: …
- PageRank works by calculating the importance of a single vertex by analyzing only the link structure of the graph. Each incoming relationship is considered as a vote. PageRank is calculated in steps, where each step is an approximation of votes. At the end of each step a weight is assigned to each vertex. Each subsequent step applies a weight to outgoing relationships using the approximate value that was calculated for the vertex from the previous step. Here we have an example graph, consisting of nodes and relationships. Each node has a degree of connections, inbound and outbound. The more inbound links a node has, the higher the weight that is applied to its outbound links to other nodes. We see that the red node, node B, is the most important node in this graph. We also see that node C, the second most important node, only has a single inbound relationship, which is coming from node B. This is the basis for how PageRank calculates the importance of nodes in a graph using only the relationships between nodes.
- As a part of today’s webinar, I will be introducing you to a reference application that combines multiple microservices with a graph processing platform to rank users on Twitter. We’re going to use a collection of popular tools as a part of this reference architecture. The tools we’ll use, in the order of importance, will be: Spring Boot Neo4j Apache Spark Docker RabbitMQ
- This diagram shows each component and microservice that we will create as a part of this sample application. Notice how we’re connecting the Spring Boot applications to the graph processing platform we looked at earlier. Also, notice the connections between the services, these connections define communication points between each service and what protocol is used. The three applications that are colored in blue are stateless services. Stateless services will not attach a persistent backing service or need to worry about managing state locally. The application that is colored in green is the Twitter Crawler service. Components that are colored in green will typically have an attached backing service. These backing services are responsible for managing state locally, and will either persist state to disk or in-memory.
- Let’s do an overview of the problem we will solve as a part of our reference application. The problem we’re going to solve is how to discover influencers on Twitter using a set of seed profiles as inputs. To solve this problem without a background in machine learning or social network analytics might be a bit of a stretch, but we’re going to take a stab at it using a little bit of computer science history.
- A graph processing platform is an application architecture that provides a general-purpose job scheduling interface for analyzing graphs. The reference application uses a graph processing platform to analyze and rank communities of users on Twitter. For this we’ll use Neo4j Mazerunner, an open source project that I started that connects Neo4j’s database server to Apache Spark. This diagram illustrates a graph processing platform similar to Neo4j Mazerunner. We can see from the diagram that new job requests are sent from Neo4j to RabbitMQ. Before Neo4j sends a message to RabbitMQ requesting a new job, it will export a graph replica to HDFS. The analysis service, which is the hexagon that is colored in purple, has an embedded standalone instance of Apache Spark, and will listen for messages from RabbitMQ containing new job requests. Each message that is received by the analysis service contains information about where the exported graph replica is stored on HDFS and what graph algorithm to execute. After the analysis service has completed execution of a job, it sends a message to RabbitMQ that will be received by a listener on Neo4j. The message will contain a path on HDFS of the resulting graph that was saved by the analysis service. Neo4j will then import the results from HDFS back into the database without interrupting or impacting transactions that are being made by other database clients. The PageRank results from GraphX will be automatically applied back to Neo4j without any additional work to manually handle data loading. The workflow for this is extremely simple for our purposes. From a backend service we will only need to make a simple HTTP request to Neo4j to begin a PageRank job.
- When we talk about microservices we are talking about developing software in the context of continuous delivery. Microservices are not just smaller services that scale horizontally. When we talk about microservices, we are talking about being able to create applications that are the product of many teams delivering continuously in independent release cycles. In this reference application, I’ve built 4 microservices, each as a Spring Boot application. If we were to build this architecture as microservices in an authentic scenario, each microservice would be owned and managed by a different team. This is an important differentiation in this new practice, as there is much confusion around what a microservice is and what it is not. A microservice is not just a distributed system of small services. The practice of building microservices should never be without the discipline of continuous delivery. For the purposes of this reference application, I hope that it will give you an idea of how to compose complex applications as distributed systems that resemble a microservice architecture in a team environment.
- This diagram shows each component and microservice that we will create as a part of this sample application. Notice how we’re connecting the Spring Boot applications to the graph processing platform we looked at earlier. Also, notice the connections between the services, these connections define communication points between each service and what protocol is used. The three applications that are colored in blue are stateless services. Stateless services will not attach a persistent backing service or need to worry about managing state locally. The application that is colored in green is the Twitter Crawler service. Components that are colored in green will typically have an attached backing service. These backing services are responsible for managing state locally, and will either persist state to disk or in-memory.
- We’ll start building the Twitter Crawler service by creating a set of domain classes and repositories to manage data with the Spring Data Neo4j project. The Twitter Crawler service will act as a REST API to interact with the users we’ve imported from the Twitter API. We can see that this service has an HTTP connection to the Neo4j server. We’ll also use the Twitter Crawler service to schedule PageRank jobs with Neo4j, which will be dispatched for calculation on Apache Spark.
- Spring Data Neo4j is a project in the Spring Data ecosystem that implements the Spring Data repository abstraction using an OGM (Object Graph Mapping) library for Neo4j. Spring Data Neo4j allows you to manage data on a Neo4j server using annotated POJOs as entity references in a Spring Data application.
- Before we can start managing data in Neo4j, we’ll need to design and construct a graph data model for our application’s domain data. The domain model for this application is rather simple, and we’ll construct it using domain objects described in Twitter’s API documentation. We’ll only have one domain concept, which is a User profile, and we’ll source this resource from profiles that are imported from the Twitter API. We’ll then have a relationship entity with the type named FOLLOWS. The FOLLOWS relationship will connect User profiles together in Neo4j after importing follower data from the Twitter API. The graph data model that we will end up with looks like this diagram. We see here that we have three nodes, having the label User, and the relationships between them having the relationship type FOLLOWS. We can see that each of the User nodes have a set of properties. First we have an internal node ID that is assigned by Neo4j. Next we have a profileId, which is a unique identifier that is assigned to the profile by Twitter. We also see that we have a property for PageRank. This PageRank property will be updated as jobs are processed by Apache Spark. The results of the job are applied back to User nodes in the Neo4j database. To represent this domain model as Java classes, we’ll need to create two domain classes, one for the User node, and one for the Follows relationship. We’ll then need to create Spring Data repositories for each of these domain classes. Let’s start by taking a look at how to create a class that represents the User node, which is a profile imported from Twitter.
- We see here that we have a basic POJO that represents a domain class for the User node in Neo4j. We can also describe fields for incoming and outgoing follower connections. These fields will be created with the type Set<User>, and give us a way to load profiles that are connected to a User node with a FOLLOWS relationship. At the top of this class we see that we have a @NodeEntity annotation. This annotation is used to mark this class as a Neo4j entity reference, one that is specific to nodes in the Neo4j database. Each domain class automatically will apply a label to Neo4j nodes in the database, using the name of the class, in this case ‘User’. The next annotation we’ll need to add is the @GraphId annotation. This annotation is used to mark a private field as a ID property, one that is automatically generated by Neo4j as a unique internally issued identifier. The next annotation we’ll use is the @Index annotation. We’ll use this annotation on the profileId field, to indicate that it is a unique ID that we will use to make sure that duplicate User nodes are not imported from the Twitter API. The next annotation we will use is the @Relationship annotation. This annotation is used to indicate that a field references a set of nodes that are connected via a relationship type and direction. In this case, we can load a user’s incoming and outgoing relationships to other users in the graph. Finally, we’ll add in other fields that we will import from the Twitter API, as well as a PageRank field that will be added as a result of our Apache Spark jobs that calculate a user’s PageRank.
- Here we have a basic POJO that represents an entity reference for our relationships in Neo4j with the relationship type FOLLOWS. The basic function of this relationship entity is to act as a index record that connects two records together, from UserA to UserB. The annotations in this POJO differ slightly from the User entity, in that the first annotation is @RelationshipEntity instead of @NodeEntity. This annotation includes a relationship type, which we’ve supplied as a parameter called FOLLOWS. The next annotation is @GraphId, and just like the User POJO, this entity is an internally assigned unique ID from the Neo4j database. Next we have our two users, UserA and UserB. UserA will be annotated as our start node, meaning that this user is following the second user. The second user is UserB, and is annotated with @EndNode.
- Next, we’ll need to create a repository to manage our data that will be mapped to the User domain class. The Spring Data project makes repository-based management of database entities a snap. We’re going to use the GraphRepository<T> interface to create a repository bean that will be created at runtime in our Spring Boot application. Here we create a basic interface that extends GraphRepository<User>. This repository interface will be initialized as a bean at runtime, and provides us with a client to manage transactions on entities for User nodes in Neo4j. We’ll also add in custom repository methods that take advantage of Neo4j’s declarative query language, Cypher. Cypher is a query language that is designed to be familiar to users of SQL, but specific to the traits of a graph database. One example I’ve added for the purpose of this demonstration is the method getUserIdByProfileId. This query will match a user node using a profileId parameter. When we download records from the Twitter API, it will often be the case that we can only see a profileId before downloading additional information, such as a user’s screen name or profile image. In the case that we want to get back the internally assigned unique ID that Neo4j has assigned to a node, we can use this Cypher query to perform a simple lookup that translates a profileId to an internal node ID. The reference application defines many custom Cypher query methods to extend the repository interface’s basic operations. Each one of these methods play a role in the crawling algorithm that allows the application to discover new users. Now that we can manage User nodes and FOLLOWS relationships, we need to think about how performant it will be to save potentially thousands of relationships per second when importing user profiles from the Twitter API. We’ll need to be able to batch transactions so that Neo4j can handle the throughput of ingesting writes at a rapid pace. To do this, we need to create another GraphRepository bean for managing the creation of many FOLLOWS relationships between a set of User profiles.
- The repository interface is similar to the UserRepository interface from the last slide. Here we have defined a Cypher query method that will allow us to save batches of thousands of relationships in a single transaction. The result of this is that User nodes will either be created or connected to other User nodes in our Neo4j database. The Cypher query is fairly simple. For each of the instances of the follow class that are provided as a parameter to this method, the Cypher query will either get or create the users that are referenced for the userA and userB fields. Finally, a relationship is created between these two users, if one does not already exist. Now we are able to save potentially thousands of users and relationships per transaction, which will be essential to the data import process from Twitter.
- Now we have our domain classes and repositories that we can use to manage entities in our Neo4j database. We can now import data from the Twitter API so that we can perform scheduled PageRank operations on Apache Spark.
- Now that we have created our Spring Data Neo4j repositories for managing our Twitter follower graph, we’ll need to expose a REST API interface that allows remote services to manage our domain data over HTTP. Thankfully this is a simple task when using the Spring Data REST project.
- All that we need to do to enable this as a feature on our Spring Data Neo4j repositories is add the spring-boot-starter-data-rest artifact as a dependency to the project’s pom.xml file.
- The database resources will now be returned as a JSON representation that lists hypermedia resources as embedded links. We can use these embedded links to manage resources of our graph repositories over HTTP.
- Now that we have everything we need to manage our Twitter profile data in Neo4j, we can import profiles from the Twitter API. To do this, we can use the Spring Social Twitter project. Spring Social Twitter is a Spring ecosystem project that allows you to manage Twitter API resources using a TwitterTemplate client. The TwitterTemplate will take care of all of our concerns, such as API authorization, and provide client bindings that will allow us to interact with Twitter’s API resources from our JVM application.
- Going back to the architecture diagram from earlier, we can see that we have a dependency on the Twitter API from the Twitter Crawler service. We’ll call the Twitter API to import user profiles and their connections to other Twitter users. The results will be saved in Neo4j using the repositories that we created earlier.
- Before we can start using this client, we’ll need to add the spring-social-twitter artifact as one of our project dependencies in the pom.xml, which is shown here.
- In our Spring Boot application we will map configuration properties as key values on the classpath. To do this, we will map keys in our application’s properties file to values that will be loaded from the environment. These environment-specific values represent the keys and access tokens that we will need to authenticate with the Twitter API. We’ll source these values to configure the TwitterTemplate bean for our Spring Boot application.
- The next step will be to configure the Twitter client that is provided by spring-social-twitter. In order to access operations for importing profiles from Twitter’s API, we will need to provide API tokens and keys that are generated for an application by Twitter. You’ll need to register with Twitter and create a developer app in order to get these keys. Getting API access is a simple process. A step-by-step guide is available from Spring’s website that will show you how to generate Twitter API keys for an application. In the code example I’ve shown here, we’ll use the @Value annotation to automatically load in configurations from the application’s .properties file. These are the keys that we saw from the last slide. We’ll then define a bean for the Twitter Template, which will require as parameters, the Twitter API access credentials. Now we will be able to automatically inject instances of the TwitterTemplate that were initialized from our configuration properties during runtime.
- Now our Spring Boot application for the Twitter Crawler service will be able to use a TwitterTemplate object as a client to interact with the Twitter API. The code snippet I have here is a simplified example of how we will access a TwitterTemplate bean using a Spring framework technique called constructor-based dependency injection. This code snippet is only meant to illustrate how we’ll be using the TwitterTemplate client throughout the application. We can see that in this class that we’re getting a reference to the TwitterTemplate through the constructor, which will be called by Spring when initializing the bean at runtime.
- The last concern we need to address on our Twitter Crawler service is to integrate with the graph processing platform. The graph processing platform is an attached backing service on Neo4j, which makes it easy for us to issue requests for new graph processing jobs. Neo4j exposes an endpoint to an unmanaged extension for Mazerunner on the classpath of the Neo4j database server. This unmanaged extension exposes a REST API for interacting with the graph processing platform’s analysis service that embeds an instance of Apache Spark.
- Going back to our graph processing platform example from earlier, we can understand how Neo4j has been extended to be able to take incoming requests for new PageRank jobs. The only thing we need to worry about for scheduling new jobs from our consumers is to issue an HTTP GET request to the Neo4j server using a route that contains the analysis type and the relationship type we would like to analyze.
- It’s easy enough to make a GET request to the job scheduling interface on Neo4j, but we will still need to create a trigger that will be called on a scheduled time interval from the Twitter Crawler service. To do this, we can use the @Scheduled annotation on a method of an object in our Spring Boot application. We’ll then provide a fixed value for the rate parameter of the annotation that is measured in milliseconds. I’ve decided that the PageRank job should be started about every 5 minutes, so we’ll initialize the fixedRate value to 5 minutes in milliseconds. This snippet of code is an example of how we will register a method using Spring’s @Scheduled annotation. The method will issue a GET request to a REST API endpoint on Neo4j. This method is described by the relative path variable. We’ll load in our environment-specific hostname of the Neo4j server to generate the full URL to the job scheduling interface for Mazerunner. Finally we’ll make a get request to the analysis endpoint.
- Now that we’ve gone over the major functionality of the Twitter Crawler service, let’s go over how to create a user interface that will display our ranked users. We’ll call this service the ranking dashboard.
- We’ve finished creating the backend components of the microservice architecture and can now write a simple client-side web application to interface with the Twitter Crawler service. We’ll need to be able to access the Twitter Crawler service’s REST API as a part of the dashboard’s host. For this we’ll need to create a reverse proxy to the Twitter Crawler’s REST API. Since we are using Spring Cloud, we are able to take advantage of the Eureka discovery service and the @EnableZuulProxy annotation to automatically inject routes from the Twitter Crawler service into our new Ranking Dashboard service. What this means is that the new Ranking Dashboard service will be able to expose the full REST API of the Twitter Crawler service on its own host, without writing a single line of code. Well, maybe just one line of code.
- The Spring Boot application we’ll create is as simple as it gets. The only application code we’ll create is the Spring Boot application class. Here we’ve annotated the application class with the annotation: @SpringCloudApplication. This enables the basic Spring Cloud features for connecting to a discovery service. We’ve also added the annotation: @EnableZuulProxy, which will enable an embedded Zuul proxy in this service. To get this to work, we do need to worry about some configuration properties in our application.yml file.
- The properties here are configured for the production Spring profile, and will provide the necessary settings to connect and register with the discovery service. Since the Twitter Crawler service we created earlier uses the same discovery service connection settings, the Ranking Dashboard will automatically create a proxy to the Twitter Crawler’s REST API. Now when the Ranking Dashboard service is started, it will contact the Eureka discovery service at the serviceUrl’s default zone property. By doing this we are able to embed the request mappings that are exposed by the Twitter Crawler service. The ID that the Twitter Crawler service will use when registering with Eureka will be twitter-rank. This ID will be used as the request path to access the routes of the Twitter Crawler service from the Ranking Dashboard service. All requests to the route with the ID, /twitter-rank from the Ranking Dashboard service will be forwarded as a request to the Twitter Crawler service. This is a pretty nifty feature. We can create local request mappings to any remote service that is registered with the Eureka discovery service. This comes in handy if you’re working with microservice architectures. The next step will be to add static web content to the Ranking Dashboard service that connects to the REST API of the Twitter Crawler service through our newly embedded Zuul proxy.
- I’ve created a simple client-side web application that uses jQuery to make AJAX requests to the Twitter Crawler REST API. Spring Boot makes it easy to map static content by placing it in the resource directory of our application.
- The example here shows a directory of the Ranking Dashboard application. The static content for the single page application has been placed in the resource directory under the folder ‘static’. Now when I run the Spring Boot application, the index.html file will be mapped to the service’s root.
- Now we can start up all the applications in our architecture one by one. I’ve already gone ahead and done this for the demo so that we can explore the working application that has already imported records from the Twitter API. First, let’s take a look at what I did to train the dataset that we’ll see in the demo.
- The dashboard is a single page web application which consumes two REST API methods on the Twitter Crawler service. Let’s first review how the dashboard will be used. The first time the dashboard is loaded, there won’t be any data to display from the Twitter Crawler service.
- Before the Twitter Crawler service will begin to automatically discover new profiles, the user must provide a minimum of three screen names of Twitter users as seeds. The goal is to add three seed profiles of users who are members of a community on Twitter. It’s important to make sure that these users follow each other, which will make it likely that there are other mutual profile connections between these users.
- The seed profiles I’ve chosen for this demonstration are: @kennybastani @starbuxman @bridgetkromhout A few of my teammates at Pivotal. When adding new seed profiles through the UI, an AJAX call will be made as a GET request to a proxied REST API path. This path will route requests to the Twitter Crawler service and respond as if the method was inside our web application. After adding each of these profiles manually through the UI, we’ll end up with the view shown here. We can see that we’ve imported the profile attributes of each of the seed profiles. These are indicated in the ranking dashboard, but the PageRank has yet to be calculated. If you remember from earlier, this job is scheduled every 5 minutes, so we’ll have to wait.
- What’s happened at this point was illustrated earlier. Here again is our example graph data model that we used to create our Neo4j domain classes and repositories. In the model we see the three seed users I’ve selected. The relationships for these users have now been imported from Twitter. They also have relationships to many other users, which have also been imported. This will be the training data set that the discovery algorithm will use to crawl new users, using PageRank as the metric of relevancy to prioritize which users to crawl next from the current data set.
- Now that I’ve added the three seed profiles, each of these user’s connections will be imported to Neo4j on the Twitter Crawler service. After about 5 minutes, the PageRank job will have been scheduled and completed its analysis of the initial users. After a PageRank value has been assigned to the initial users, you will see new users that have automatically discovered. The screenshot here shows users that were discovered automatically by the Twitter Crawler service. Newly discovered users are indicated by a green plus icon in the Rank column. These results are highly appropriate to the seed selection. We can even infer that the original three users have distributed systems development and DevOps as a common interest. Now let’s go ahead and explore the working application.
- If you have questions for me after this webinar, please connect. You can reach out to me on Twitter, my screen name is here. Also, be sure to stay connected with the Spring team on these sites.