Introducing Apache Geode and Spring Data GemFire

2© 2014 Pivotal Software, Inc. All rights reserved. 2© 2014 Pivotal Software, Inc. All rights reserved.
Introducing Apache Geode
with Spring Data GemFire
John Blum / Luke Shannon
24 June 2015

3© 2014 Pivotal Software, Inc. All rights reserved.
Agenda
 What is Apache Geode?
 Apache Geode Technical Specification
 High-level Implementation Details
 Spring Data GemFire
 Q&A

Pivotal GemFire
What is Apache Geode?
 In a nutshell…
– Open Source core of Pivotal GemFire
– Apache Incubator
▪ https://wiki.apache.org/incubator/GeodeProposal

One definition is…
A distributed, in-memory compute and data
management platform that elastically scales
to achieve high-throughput, low-latency
access to big, fast data in order to power
business critical, analytical applications in
real-time.
Nodes
Ops /
SecLinear scalability
Elastic capacity +/-
Latency optimized
data distribution

 Technically…
– Distributed, in-memory database (IMDG) with ACID properties
– Transactional Key/Value Store (Cache)
– Glorified-version of ConcurrentHashMap
– Message Bus with guaranteed message delivery
▪ Reliable, asynchronous event notifications

What Apache Geode Does?
 Enables new or existing Java applications to operate at cloud-scale
with predictable, low-latencies and be highly-available without
sacrificing consistency.
 Applications can transact and analyze Big Data in real-time to garner
insights and achieve meaningful and impactful business results.
Ap
p
Ap
p
Ap
p
Ap
p

Apache Geode Characteristics
 In-Memory
 Distributed
 Consistent
 Highly-Available / Fault Tolerant
 High-Throughput / Low, Predictable Latency
 High Scalability
 Data-aware Parallel Compute

How are Apache Geode’s features implemented?

Memory
 Geode stores data in memory…
– JVM Heap
– Off-Heap (Native Memory)
 Geode (optionally) stores data to disk
– Persistence + Overflow

Management
Memory
 Eviction (LRU)
 Expiration (Region & Entry TTL/TTI)
 Automatic resource management (critical & eviction Heap %)
 Region Compression (Snappy)
 JVM GC Tuning

IMDG
 An In-Memory Data Grid (IMDG), such as Apache Geode…
– Pools system resources across multiple nodes in a cluster to
manage both application state and behavior.
– Memory, CPU, Network and (optional) Disk

High-Throughput / Low-Latency
 Geode uses concurrent, in-memory data structures with a highly
optimized distribution infrastructure to minimize context switching and
contention.
– synchronous/asynchronous replication for high read throughput
– partitioning for both high read and write throughput

Shared-Nothing Disk Persistence Architecture
 Manages data in local disk files on each node in the cluster
independently from other nodes
– Members are self-sufficient; Members do not share memory or disk;
– Data/Transactions appended to Oplog files
– Compaction
 Design goals (all about In-Memory)…
– Minimize IO operations (Write-Behind vs. Write-Through)
– Minimize locking
– Minimize serialization

REPLICATE & PARTITION Regions
High Availability
 Data is partitioned across nodes in the cluster and distributed uniformly
– Rebalanced as nodes join/leave
– Redundant copies

Persistence – Shared Nothing
Server 3Server 2
Bucket 3
*primary*
Bucket 1
secondary
Bucket 3
secondary
Bucket 2
*primary*
Server 1
Bucket 2
secondary
Bucket 1
*primary*

Server 1 Server 3Server 2
Bucket 2
secondary
Bucket 1
*primary*
Bucket 3
*primary*
Bucket 1
secondary
Bucket 3
secondary
Bucket 2
*primary*
What if a server
crashes?

Server 1 Server 3Server 2
Bucket 2
secondary
Bucket 1
*primary*
Bucket 3
*primary*
Bucket 1
*primary*
Bucket 3
secondary
Bucket 2
*primary*
Bucket 2
secondary
Primary
Failover
Restore
Redundancy
Bucket 1
secondary

Network Partitions

Partition Regions
Configurable Consistency
 Redundant Copies
– 0 to 3
 Colocation
 Total Number of Buckets
– 113 (Default)
 Local Max Memory
– 90% of Heap
 Total Max Memory
– All Members, Both Primary/Secondary
Copies

REPLICATE vs. PARTITION
Configurable Consistency
 Partition Regions
– Updates routed through primary bucket
– Holds lock on key when distributing updates
 Replicate Regions
– Possible conflicts due to out-of-order updates
– Conflict checking (--concurrency-checks-enabled)
ReplicatedPartitioned Disabled

Apache Geode Topologies

Peer-to-Peer (P2P)
 Configuration with the Cache embedded in the application process
– Close proximity of application logic and data

Client/Server
 Single-Hop Data Access
 Continuously load balanced between
servers to get the best response-times
`
 Automatic fail-over when servers crash or
become unresponsive

WAN
 Site-to-Site Replication for DR

Use Cases
Apache Geode
 Persistent OLTP/OLAP
– System of Record (SOR)
 Caching with JSR-107 (JCache) or Spring Cache Abstraction
 HTTP Session State Management
– Caching and Replication
 Distributed L2 Caching for Hibernate
 Memcached Server (Gemcached)

Additional Features
Apache Geode
 Functions
– Application logic is distributed, routed and run in parallel with close proximity to the data required by the
application.
 Serialization (PDX)
 Delta-propagation
– difference of new/old value
 JTA-compliant Transactions
 Security
– SSL, Auth

Roadmap
Apache Geode
 HDFS persistence
 Off-Heap storage
 Lucene search/indexing
 Spark integration
 Cloud Foundry service

Interface
Apache Geode
 Language Support
– Native: Java, C/C++, C#
– Non-native: JavaScript, Python,
Ruby, Scala, etc
 API
– Java HashMap (Region)
– OQL / Function
– Memcache client
– REST API
– Spring Data GemFire
 Data Structure
– User-defined Classes / POJOs
– Portable Data Exchange (PDX)
– Documents (JSON)
 Powerful Data Access
– Object Query Language (OQL)
– Data-aware, Scatter-Gather
Functions
– Publish/Scribe framework for RI/CQs
with reliable, async event queues

Spring Data GemFire
http://projects.spring.io/spring-data-gemfire/

So… Why Spring?

“Simple things should be simple;
complex things should be possible”
– Alan Kay

Spring Data GemFire (SDG)
Consistently applies Spring's powerful, non-invasive programming model
to simplify configuration and development with Apache Geode.
Spring Ecosystem Integration…
– Spring Cache Abstraction / Transaction Management
– Spring Data Commons + REST
– Spring Integration (Inbound/Outbound Channel Adapters)
– Spring XD (Input Source and Sinks)

Apache Geode combined with Spring Data GemFire
can be used as JSR-107 (JCache) caching provider
+ +

GRAILS
Full-stack, Web
XD
Stream, Taps, Jobs
BOOT
Bootable, Minimal, Ops-Ready
Big,
Fast,
Flexible
Data Web,
Integration,
Batch
WEB
Controllers, REST,
WebSocket
INTEGRATION
Channels, Adapters,
Filters, Transformers
BATCH
Jobs, Steps,
Readers, Writers
BIG DATA
Ingestion, Export,
Orchestration, Hadoop
DATA
NON-RELATIONALRELATIONAL
CORE
GROOVYFRAMEWORK SECURITY REACTOR

Use Cases / Configurations
Spring Data GemFire
 Apache Geode Configuration / Bootstrapping
– Replacement for cache.xml; Can be used with Cluster Configuration
 Application Peer (Cache)
 Application Client (ClientCache)

Examples…
SDG-based Configuration

with Spring
Bootstrapping Apache Geode
@SpringBootApplication
@ImportResource("/spring-data-gemfire-cache.xml")
@EnableGemfireRepositories
@EnableTransactionManagement
public class SampleDataGemFireApplication {
public static void main(String[] args) {
SpringApplication.run(SampleDataGemFireApplication.class, args);
}
}
Or…
gfsh>start server –name=Example … --spring-xml-location=“<classpath-to-spring-application-
context.xml>”

Basic (Region) Data Access
Data Access with Spring
@Repository
public class CustomerDao {
@Resource(name = “Example”)
private Region<?, ?> example;
…
}
Example…

GemfireTemplate
Data Access with Spring Data GemFire
 Advantages…
– Simple, Convenient Data Access (CRUD, OQL, Function)
– Protects developer from GemFire/Geode API changes
– Exception Translation into Spring DAO Exception Hierarchy
– Transaction Management
<bean name=“customersTemplate” class=“org.springframework.data.gemfire.GemfireTemplate”
p:region-ref=“Customers”>

GemfireRepository
Data Access with Repositories
public interface CustomerRepository extends GemfireRepository<Customer, Long> {
}
// Spring context.xml
<gfe-data:repositories base-package="org.example.app.customer.dao.repo">
<repo:include-filter type="assignable"
expression=”org.example.app.customer.dao.repo.CustomerRepository"/>
</gfe-data:repositories>
Example…

Implementation
Annotation-based Function Implementation/Execution
package org.example.app.functions;
import …
@Component
public class ExampleFunctions {
@GemfireFunction
public Customer update(…) {
…
}
…
}
Example…
(Optional)…
<bean class=“org.example….ExampleFunctions”/>

Execution
package org.example.app.function.executions;
import …
@OnRegion(“Example”)
public interface ExampleFunctionExecution {
Customer update(…);
}
And…
<gfe:annotation-driven/>
<gfe-data:function-executions base-package="org.example.app.function.executions">

Execution
 Additionally…
– @OnServer(s), @OnMember(s)
 Examples…

JSR-107 (JCache), Spring Cache Abstraction
Geode Caching Provider
 With @Cacheable
 And @CachePut, @CacheEvict(allEntries=[true|false]),
@CacheConfig
– http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#cache-jsr-107
 Or with JSR-107 using…
– @CacheResult, @CachePut, @CacheRemove, @CacheRemoveAll,
@CacheDefaults

Demo
By Luke Shannon

On the horizon…

Roadmap
Spring Data GemFire
 Support for GemFire 8.2 & 9.0
– HDFS, Off-Heap, Apache Lucene, Spark
 Support for pre-defined data sets using Snapshot Service
– SGG-408
 Expiration annotation support with CustomExpiry
– SGF-404
 Spring Session Adapter for GemFire/Geode
– SGF-373

Roadmap Continued…
Spring Data GemFire
 Repository Query method annotation extensions...
– SGF-392
public interface ExampleRepository implements GemfireRepository<Customer, Long> {
@Trace
@Limit(20)
@Import(“org.example.app.domain.SomeOtherType”)
List<Customer> findByLastname(String lastName);
}

Get Involved…

Apache Geode
Contribute
 In addition to developing Spring applications with Apache Geode, now is
a perfect time to get involved in the Apache Geode community and
contribute growing Apache project from the ground-up.
– Ways to participate…
http://geode.incubator.apache.org/contribute/

Finally…

Washington, DC – September 14 – 17, 2015
SpringOne2GX
 Early Bird price registration ends August 14th, Register Now!
– https://2015.event.springone2gx.com/register
 Building High-Scalable Spring Applications with Distributed In-Memory
Data Grids.
– Luke Shannon & John Blum
– Tuesday, September 15th, 2015
– https://2015.event.springone2gx.com/schedule/sessions/building_highly_scalable_spring_applications_with_in_m
emory_distributed_data_grids.html

References
 http://geode.incubator.apache.org/
 http://geode.incubator.apache.org/contribute/
 https://github.com/project-geode/docs/wiki
 https://github.com/apache/incubator-geode
 https://pivotal.io/big-data/pivotal-gemfire
 http://gemfire.docs.pivotal.io/latest/userguide/index.html#getting_started/topics/gemfire_overview.html
 http://gemfire.docs.pivotal.io/latest/javadocs/japi/index.html
 http://stackoverflow.com/questions/tagged/geode+or+gemfire
 http://projects.spring.io/spring-data-gemfire/
 http://docs.spring.io/spring-data-gemfire/docs/1.6.0.RELEASE/reference/html/
 http://docs.spring.io/spring-data-gemfire/docs/current/reference/html/
 http://docs.spring.io/spring-data-gemfire/docs/current/api/
 https://github.com/spring-projects/spring-data-gemfire
 https://github.com/spring-projects/spring-gemfire-examples
 http://stackoverflow.com/questions/tagged/spring-data-gemfire

Q&A
Any questions…

Thank you!

Introducing Apache Geode and Spring Data GemFire

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Introducing Apache Geode and Spring Data GemFire

Similar a Introducing Apache Geode and Spring Data GemFire (20)

Último

Último (20)

Introducing Apache Geode and Spring Data GemFire

Notas del editor