Integration daysBizTalk Used for Real Time Payments

Introducing Luup

BizTalk Used for Real Time Payments

Luup provides payment services for mobile phones in emerging markets in Asia and the Middle East. This presentation will
discuss challenges in building a mobile payment platform, which supports high transaction volumes and a minimum of
latency, using BizTalk for internal workflow services in a SOA architecture.

© Luup International Ltd. #1

Introducing Luup
Topics

• Introduction to Luup
– Real-time Payments
– The Luup System Platform
– The Main Architectural Driving Factors
– How Luup Uses BizTalk
– The Development Teams
• Our Experiences and Challenges With BizTalk
– Performance
– Consistency
– Maintainability
– Automated Testing
– Automated Deployment

Introducing Luup
Luup

• Started in Norway > 10 years ago
• Development in Oslo, HQ in London, Sales Office in Dubai
• Experience with several mobile payment products
• Main product is a global remittance eco-system
• Offers bank-grade mobile payment solutions
• A managed service
• An integrated banking platform
• Our projects are centered around:
- Integration towards new partners and customers
- Make business processes work on our standard platform
- UI: SMS, Mobile App and Web


Introducing Luup
Real-time Payments Examples

• Remittances - customers send money to their
families conveniently, quickly and securely
using their mobile devices
• Domestic Payments - customers pay people,
bills, charities and merchants
• Disbursements - customers receive their
wages or social benefits into an account linked
to their mobile device


Introducing Luup Driving
Factors
Architectural Decisions

• Decided to use standard software as much as
possible (came from a history where
everything was in-house and custom made)
• Robust orchestration of services is important
• Building a scalable platform is vital, since our
business model relies on a high volume of
transactions to be profitable


Introducing Luup
The Luup Platform

Luup Payment Processing Platform
Employee UI End user UI SMS
web site web site / web services server Telco
Q Q Q
of ser ration

Q Q Q
BizTalk
vices
st

MsgBox Partner
Partner Bank web
Partner
Orche

Core/BPM Gateway services
MsgBox Gateway
Gateway

http http In
Ty teg
Bi pic rat
zT al io
Luup al n
Core banking k usa –
system Services ge
of

Introducing Luup
The Luup Team

• Our internal developers:
– Had no experience with BizTalk 2 years ago
– Were mostly experienced .Net developers –
learned quickly
– Now this background helps in deciding whether
code should be made in BizTalk shapes, C#
components or services
• Offshore team with 3-5 developers:
– Developed the Core/BPM application

Introducing Luup
End of Introduction

• Now over to challenges we have faced and
experiences made during the last 2 years


IntroducingExperiences and Challenges
Luup’s Luup
in Using BizTalk

• Introduction
• Performance
• Consistency
• Maintainability
• Automated Testing
• Automated Deployment


Introducing Luup
The Luup Platform - Recap

Q Q Q Q Q
Q
BizTalk
MsgBox Partner
Partner Bank web
Partner
Core/BPM Gateway
Gateway services
MsgBox Gateway

http http

Core banking Luup
system Services


Introducing Luup

PERFORMANCE


Introducing Luup
Challenges

• Complex business process orchestrations
– Many (10-20) web service calls in each workflow
• Important for our design choices
• Low latency required for some scenarios
– Serve requests from web and mobile app
frontends
• Performance tuning of BT is complicated!
– Each system/setup is unique


Introducing Luup
Current Configuration

• Shared server for Partner Gateway and BPM
• Active-Passive cluster
– Due to licence costs, until the number of
transactions increases
• Separate hosts for receiving, sending and
processing
• We use the defaults for most settings


Introducing Luup
Design Considerations

• The database will be our bottleneck
– The database is the hardest component to scale
– We need to limit database roundtrips


Introducing Luup
Latency Example

• Message box round trips (polling every 0,5s):
– If 10 service calls:
• Adapter polls for messages to send, orchestration polls
for response messages
• 10 requests and 10 responses = 20 msg box roundtrips
– 20 msg box roundtrips * 0,25 sec = 5 sec
(for polling msg box only)
– If 1 sec used on actual processing: 5 + 1 = 6 sec
• How can we avoid this latency?

Introducing Luup
Low Latency Considerations

• Alternatives for calling web services from
workflows
– WCF Adapters
• Polling interval can be tuned down to 50 ms
– Use .Net code – «Service Caller»
• Used in our solution


Introducing Luup
Using Adapters

• Pros
– The default and recommended way when using
BizTalk
• Cons
– Orchestration is persisted for each web service
call
– Message Box roundtrips for request and response
message


Introducing Luup
The «Service Caller»

• Uses C# code to do SOAP calls
• Two versions:
– Old version using WebRequest
– New version using WCF
• We only use it in the CORE/BPM application
– Only calls to locally deployed services
• We use BAM for logging requests and
responses

Introducing Luup
Service Caller - Advantages

• Low latency and DB load
– No message box roundtrips
– Orchestration is not persisted after the request is
sent


Introducing Luup
Service Caller - Disadvantages

• Need to be careful about when the
orchestration is persisted
– When replaying request we need to use the
original request ids
– Generate request ids up-front and start using the
service caller after the ids have been persisted


Introducing Luup
Service Caller - Disadvantages

• We take control of resource usage throttling
away from BizTalk
– Threads are used while waiting for web service
replies
– Use short timeouts (< 60s) and short retry
intervals


Introducing Luup
Summarizing Performance

• Performance tuning BizTalk is hard and time
consuming
• Using .Net code to call web services has
improved our latency
• We need to focus on performance in the near
future as the number transactions increases


Introducing Luup

CONSISTENCY


Introducing Luup
Consistency

Q Q Q Q Q
Q
BizTalk
MsgBox Partner
Partner Bank web
Partner
Core/BPM Gateway
Gateway services
MsgBox Gateway

http http
called with serv
Inside circle: Id

Core banking Luup
system Services


Introducing Luup
Consistency

Q Q Q Q Q
Q
BizTalk
MsgBox Partner
Partner Bank web
Partner
Core/BPM Gateway
Gateway services
MsgBox Gateway

http http
Throug

Core banking Luup
system Services
h
circle:

A

Introducing Luup

MAINTAINABILITY


Introducing Luup
Size of Orchestrations

• Complex business logic gives complex
orchestrations
– Results in orchestrations which are hard to:
• Understand
• Change
• Test


Introducing Luup
Shapes vs C# Code

• Use shapes as much as possible?
– Results in huge orchestrations which are
• Hard to test
• Hard to read
• Has duplicated logic
– Pros
• All the logic is visible in the orchestration designer and
debugger
– Call Orchestration can be used to split out
orchestration parts


Introducing Luup
Shapes Example


Introducing Luup
Shapes vs C# Code

• Or use C# code as much as possible?
– Keeps the orchestrations dense and focused
– Use the orchestration only for controlling the
transaction (at a high level)
• Which is the main reason why we needed BizTalk
– C# code is easy to unit test


Introducing Luup
Using Internal Contracts

• Vital for having a clean modular architecture
• Hide gory details from external systems
– Internal contracts which are easier to understand
• Mapping complex external contracts
• Greatly improves maintainability
– Anti-corruption layers
• Keeps the core stable even when external services
change
• Small and reusable orchestrations

Introducing Luup

Q Q Q Q Q
Q
BizTalk
MsgBox Partner
Partner Bank web
Partner
Core/BPM Gateway
Gateway services
MsgBox Gateway

http http

Core banking Luup
system Services


Introducing Luup

Contracts

Contracts
Reusable
orchestrations
Contracts


Introducing Luup
Summarizing Maintainability

• Keep orchestrations small
• Don’t be afraid of using C# code
– But be careful about orchestration persistence
when changing state in C# code
• Use internal contracts


Introducing Luup

I don’t usually test my
code.

But when I do,
I do it in Production.

AUTOMATED TESTS


Introducing Luup
Unit and Integration Tests

• Using unit tests for
– Maps
– C# code

• Integration tests
– Using BizUnit for testing orchestrations


Introducing Luup
Testing Maps – The Built-in Test Func


Introducing Luup
Testing Maps – The Built-in Test Func

• Disadvantages of built-in test functionality:
– Compiles maps inheriting from test base class
• References test assembly
Microsoft.BizTalk.TestTools.dll which must be
deployed to production
– Exceptions are swallowed
• Hard to figure out why a test failed


Introducing Luup
Testing Maps – Our Solution

• Custom-made library
• Calls XLANG component to do transformation
• Fluent language for writing assertions


Introducing Luup
Testing Maps – Our Solution

Schema validation

Field validations


Introducing Luup
Testing Orchestrations

• Complex workflows
– Requires automated tests
• Custom made fluent interface on top of
BizUnit
– Reuse of code
– Terse tests which are refactor-friendly
• Currently we have around 350 BizUnit
orchestration tests

Introducing Luup
Testing Orchestrations - Example
[Test]
public void RemittanceSend_WhenInvalidBankTransactionDate_ShouldReturnGeneralError()
{
const string invalidTransactionDate = "12/24/2012";
var testCase = new BizUnitTestCaseBuilder()
.WithTestCaseSteps(SharedTestSteps.CreateAndCleanupDirectories()) Initialize the test input file
.WithCreateXmlFileStep(
new CreateXmlFileStepBuilder(@"TestDataMtoRemittanceSendRequest_ToAccount.xml",
SharedConstants.Pgw.ReceiveDirectory +
TestContext.CurrentContext.Test.Name + ".xml")
.WithUpdateXml("SenderBankTransactionDate", invalidTransactionDate,
"//*[local-name()='SenderBankTransactionDate']")
)
Wait for orchestration
.WithWaitForFileStep(SharedConstants.Pgw.SendDirectory) execution to finish
.WithXmlFileValidation(new XmlFileValidationBuilder(SharedConstants.Pgw.SendDirectory)

.WithSchemaValidation(@"......SharedBizTalkSchemasPartnerGatewayMtoRemittanceSendResponse.xsd",
"http://www.luup.com/schemas/")
.WithXPathValidation("ErrorId", MtoRemittanceSendResultCode.GeneralFailure.ToString(),
"//*[local-name()='ErrorId']")
)
.Build();
Verify the result message

var bizUnit = new BizUnit.BizUnit(testCase);
bizUnit.RunTest();
}

Introducing Luup
Build Server Integration

• After each code checkin:
– Build packages
– Deploy all BizTalk applications to a dedicated
BizUnit server
– Run BizUnit integration tests


Introducing Luup
Summarizing Automated Testing

• The built-in unit testing functionality for maps
is no good
– Using BizUnit is a better choice
• Both maps and orchestrations can and should
have automated tests
• Build server runs automated tests


Introducing Luup
Test Environments
TFS
Build
Source
Servers
Control

Code
Changes Deployment

Production

AUTOMATED DEPLOYMENT


Introducing Luup
Our Deployment Challenges

• Our solution consists of about 20 BizTalk
applications
• 9 test environments
• Time consuming deployments to Production


Introducing Luup
How We Do Deployments

• Each application use BizTalk Deployment
Framework (BTDF)
• Using the Sprinkler framework to control
deployment of multiple applications
• The build server creates installation packages
each time we check in code
– Also deploys and runs BizUnit integration tests
• Automated deployments to test environments

Introducing Luup
The Sprinkler Framework

• Open source on CodePlex
• A set of PowerShell script built on top of BTDF
• A bit immature
– We had to make a couple of changes to the scripts
in order to make it work with Active-Passive
clusters
• Great when having multiple applications and
servers
• Great for automated deployment

Introducing Luup
Deploying to Production

• Active-passive cluster
• Using Sprinkler for parallel deployment to the
nodes
• IT Ops handle deployment
– Developers should not touch the Production
environment


Introducing Luup
Summarizing Automated Deployment

• BTDF and Sprinkler are good frameworks
• Automate your deployments
• Use the same binaries when deploying to test
and production
• Deploy early and often


Introducing Luup
Main Summary

• Flexible and scalable solution
• Usage of BizTalk both as a traditional integration tool
and orchestration of services
• Reduce latency by reducing message box roundtrips
• Improve maintainability by creating a modularized
architecture with clean interfaces
• Testing of maps and orchestrations can and should be
automated
• Automated deployment is useful when having many
BizTalk applications


Introducing Luup
Questions?


Integration daysBizTalk Used for Real Time Payments

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (10)

Similar a Integration daysBizTalk Used for Real Time Payments

Similar a Integration daysBizTalk Used for Real Time Payments (20)

Último

Último (20)

Integration daysBizTalk Used for Real Time Payments

Notas del editor