This white paper covers how users are testing their SAP PI/PO solutions.
It shows you how why your interfaces are important to your business and SAP Landscape.
Then you will see what good tests will do to lower the expected failures and stability of your system.
You will see what the different ways of automating the testing are and how you can make your SAP PI/PO testing even faster. Then finally you will see how you can make a full automation of the test using Figaf IRT
2. Overview
For a business to run smoothly, a good system integration is
vital. Part of this integration is developing improved capabilities,
functionality, and interfaces. The challenge for Integration
developers is that these kinds of upgrades can introduce new
bugs and compromise existing functionality, and must be
carefully monitored and resolved to ensure the delivery of a
clean and smoothly running system to the business.
SAP developers know that testing is a huge part of any successful
project. Integration projects, in particular, require extensive
testing to identify and fix any bugs before they are moved to
production and lead to problems for the business. Despite the
key role that testing plays in any integration project, most testing
methods are time consuming.
For those working with SAP PI/PO, the leading platform used
by integration developers, there are a few good tools to help
automate this testing process to both save time and help identify
issues before they become problems in the integration.
This white paper will show why testing is a vital element in the
development process. It will also show how you can perform
better tests and even automate them.
3. About SAP PI/PO
Why Test?
SAP PI/PO (Process Integration/Process Orchestration) is the
leading SAP integration platform today. It allows SAP systems
to communicate with all other systems in the landscape, and
supports a multitude of communication protocols such as
IDOC, RFC, File and more. PI/PO works by converting messages
between formats and matching the required target structure.
Newer integration is sometimes done using SAP Cloud Platform
Integration, which is a flexible platform that supports many
different types of cloud integration, as well as other types of
integrations. But at this time, most customers are still using SAP
PI/PO. The challenges with testing are similar, if not in fact more
complicated, with Cloud Integration.
Today’s marketplace is fiercely competitive, and businesses are
constantly innovating to offer new features that may give them a
competitive advantage and/or cut costs. The developers who are
supporting these innovations are integrating new solutions and
working hard to make new system elements run smoothly. But
without comprehensive testing, problems will inevitably arise.
Some examples of typical problems in an SAP PI/PO system
include:
3
A mapping that is moved, but not tested with other orders and which
caused 90% of orders to fail.
An updated PI system that changed how some mapping functions
worked and resulted in different messages, is not working.
A patch to PI resulting in a file name that can’t process the file name of
messages.
4. 4
How does testing work?
When a new feature has been delivered it usually has a number
of bugs or errors. Some of these errors will be easy to spot, while
others are hidden because they only occur outside of the normal
range. Each time you test something the number of bugs will
decrease. With enough tests you may get close to zero errors,
but there will always be the possibility of errors if the input can
vary and be outside the scope of normal usage.
Tests performend
Error Left
Number of errors will decrease with
the number of tests performed -- if the tests are good.
5. 5
Impact of errors
System errors always have an impact on your business. Aside
from wasting valuable time, a poorly functioning system that
has not been properly tested can cause significant problems
for the business.
For example, if the orders your system receives are not
processed correctly, your business may miss a deadline
for sending supplies. Even worse, an error that changes
the order from 10 pieces to 10.000 it will have an impact
on the processing. Either error would mean that someone
has to spend extra time correcting the errors with manual
processing. Or, perhaps there could be a loss of business
when orders are not properly relayed. The impact that these
sorts of malfunctions can have on relationships with partners
and customers can be serious. In some industries you have
to follow some service level agreement (SLA) with the partner
about how many errors are allowable in your EDI or response
time. If you cannot live up to the SLA you may not be able to
bid for next year’s contract.
In addition to the business problems, errors are costly
because they require extra support time and IT change
management to fix the problem, and may even affect reporting
to the authorities. This can involve correcting the mapping,
generating patches for the system, testing again to make
sure that everything works, and then resolving the messages
involved in the error. Sometimes developers must perform
manual steps in the SAP system to correct potential problems.
No matter where the error occurs, it will have a negative
impact on the business and it will cost. This is why testing your
interfaces before moving them to production is so important.
Comprehensive testing will eliminate much of the cost of
expensive errors.
6. 6
What does a good test
look like?
To run better tests, you need to know what a good test case
looks like. To maximize the results of each interface you are
testing, we recommend the following method:
The biggest challenge is that users often don’t test with high
enough complexity because making test data in backend
systems is challenging. There are a lot of steps to create multiple
line items on, for instance, orders or invoices. To make testing
more complete, items need to simulate real life operations as
much as possible. In ERP it is often difficult and time-consuming
to create “real” looking orders. This is one of the big reasons for
tests that lack sufficient complexity. The best approach is to use
the replay/record method, because it will simulate the productive
message, and is often how developers perform a regression test.
Tests need to be as close to real data as possible.
Create tests with a minimum of 3 line-items to generate the errors that
most often occur.
Verify that the mapping output is correct.
Verify that adapter modules are executed, and that data is correct
either by looking at the XML documents, or even better by looking in
the backend systems.
Perform many different tests to get best possible coverage for
unexpected results. It can be difficult to fit all test cases into just one
document.
If the data is complex, test for all existing variations. For example,
if there are different order types, test for them all to generate the
different expected outputs.
7. 7
Locate messages in message monitor
Download source and target messages from monitor
Send message with test tool
Download output message
Verify that messages look identical
Repeat the process for many messages
1
2
3
4
5
6
Developer Tests
The most common SAP PI/PO testing method developers use is
a complex process and involves a regression test, which is done
after a change has been implemented and is performed by going
through some variation of the following six steps, depending on
what integration area they are working in. Some variations are
more time consuming than others. The six steps are:
Find Test
Data
Run Test
Get
Results
Verify Repeat
8. 8
Assumptions
If you are going to perform many tests, it may not be possible
to verify all results in the receiving system. Instead, you can set
up programmatic validations, or assumptions, for the expected
outcome. There are two ways to do this:
Assumptions can work for any test for which you have an input and
some assertions on what you should get as result. An assumption
could be that you expect the field order number to be 123. If the
order number is 125, then the test should count as an error. Or
maybe you expect a test to return 3-line items, and the sum of the
total of those items should be 354 EUR. You can set then set up the
assumptions in programming language where you write what to
expect for each output. Configuring all the assumptions and making
sure they are updated when the mapping changes can be a tedious
process.
The other approach is to use a template which can then be compared
with an existing outcome. In most cases you would take the XML
document from production and compare each line to the test
document, and verify that they are identical. You can use tools like
XMLSpy or any other diff tool that can work on text elements. You
will, however, need to find a way to accept/ignore the fact that the
date changes each time document is processed, to avoid having it
generate an error.
1
2
9. 9
The Costs of Integration
Testing
In the SAP PI/PO context, testing is of enormous importance.
Studies have shown that testing accounts for 30-40% of the
development cost in IT projects, in general, and the consensus
among Integration professionals is that the same pattern holds
true for integration projects.
In other words, testing accounts for a significant portion of the
development budget, and although much of that testing time is
spent fixing bugs to ensure that everything runs smoothly, other
processes also account for significant time spent. It makes a lot
of sense to streamline the testing process as much as possible.
The area with the highest costs for integration development
is upgrade testing. Most customers tend to upgrade the SAP
PI/PO platform once a year. To ensure that the upgrade runs
smoothly and without disruption, rigorous testing must be
performed. A lot of the tests in this context are conducted by
the business people and will require you to do testing with them.
After the upgrade, some functions may behave a little differently
than before the upgrade. Some common upgrade differences
have been changes in the way mappings are executed or how
adapters are working, in which cases, the documents will fail or
just be wrong.
What goes into a system test?
When we are looking at the cost of a project, it is important
to know which activities are necessary to perform testing. The
following is a list of common testing activities:
All of these activities
are manual tasks
that should be
repeated each time
anything is changed.
Developer test – usually a manual test, running message mapping
Collection of test data from a productive system
Unit test – performed on the test system
User acceptance test – allows the user to try out the system
Creation of automated test – generally labor and time-intensive
Test documentation – document, document, document!
Test coordination – ensure that all testing has been done before
delivery of the system to the customer
10. 10
When to test
The decisions about when testing should be performed and
what type of testing will be used on a system depends on what
has changed. Below is a list of the different types of changes that
can occur in a system and how they should be tested.
New development
When a new interface is requested, the business user should work with the
developer to test the system to ensure the correct functionality is delivered. This
is most likely a manual test. During the development you may find documents
that are mapped correctly, while other parts of the mapping may need to be
changed. You can reuse the valid test cases and automate them to ensure that
they are still valid.
Changes to interfaces
When you make a change to an interface in production, you generally expect
existing messages to be processed as usual, except they may have a small
change. Therefore, it is a good idea to run a good set of different messages
through the interface to verify that existing messages still work. This can be done
by reusing test data from production. You will also need to verify that the new
functionality works as expected. This should be verified by both the developer
and the business user.
Change of common library or functions
Many times, developers will implement frameworks, user defined functions, or
modules that are used across multiple modules to customize the functionality
for the individual company. Depending on the change, you will need to perform
a regression test on the interfaces that are using that functionality. The testing
performed will depend on what has been changed. It is recommended that you
test more than just the interface you wanted to change, if you are not sure which
interfaces may have been affected by a change.
System Upgrade or Patching
Once or twice a year, you will need to upgrade the SAP PI/PO system to fix bugs
or add new functionality. The patches have an effect on the way you process
messages, as well as on message mappings, modules, or adapters. Often, the
errors can be unexpected, depending on whether you have used a special
mapping function. To test this you will need to perform a good regression test to
make sure that you get around as many criteria as possible.
Migration
If you have been using XI/PI for a long time, you will need to migrate to a single
stack system. This is a better performing system, which is easier to maintain and
develop. For this you will need to perform a test on all interfaces because there
may be changes just as there would be with support packages.
11. Types of Tests
Within this basic testing framework, there are a variety of
approaches that a developer may choose, including manual
tests, custom-built tools, and automated tests such as general
workflow tools and specific record replay tests.
Specific Record Replay Tools
Specific Record Replay tools allow the developer to set up test cases and
run them multiple times. Because they have some restrictions on what you
can test, they are simple to set up. This will cover most integration logic and
you will not be spending months on setting up a test case. There are some
limitations that cannot be tested as you would communication outside the PI/
PO system. These limitations to specific record replay tools are that they will
not test everything, are limited to testing only SAP PI/PO, and may have some
restrictions on what can be tested.
General Workflow Tools
General workflow tools such as HP ALM or IBM Rational are automated
testing tools which are flexible and allow testing of changed objects, much like
manual tests. They also offer full end-to-end flow, but again, they are very time
consuming to set up – perhaps as time consuming as creating the integration
in the first place, and the developer must learn a new development
environment where they can program the tests and assumptions. This will be
time they spend on creating tests instead of developing new functionality.
Custom-Built Testing Tools
One approach to manual testing uses a custom-built tool that matches the
specific requirements of your business and does not have any associated
license cost. But these tools are time-consuming to build and set up,
they cover only what is specifically configured, and they require ongoing
maintenance as the system grows and changes.
Manual
The advantage of the manual testing approach is that it is flexible, it allows the
testing of changed objects, and offers good coverage of full end-to-end flow.
The drawbacks to manual testing are that it is time consuming for each test,
and generally lacks the required complexity for full testing.
11
12. The Best Testing Tools
for SAP PI/PO
There are two available testing tools on the market today that
address many of the limitations discussed above. These are the
Figaf IRT and the Int4 IRTT.
Figaf IRT
Figaf’s Integration Regression Tool (IRT) is an automated testing tool that
covers all types of integrations. It tests message logic and mapping; it
compares XML, Json, or EDIFACT format messages; and best of all, it takes a
mere 5 minutes to set up tests for a given interface. It is possible to configure
multiple scenarios at a time with IRT.
Int4 IRTT
The Int4 IRTT tests with ABAP backend for messages to or from the system’s
backend. It performs comparisons at the database level to verify any
differences, and it comes with templates for common message types. With
this tool, transaction recording should be used when sending messages.
Read more in the following pages or at https://figaf.com/irt
You can read more on https://figaf.com/irtt
12
13. Simply put, IRT is a simple tool for testing complex
scenarios. Learn more at https://figaf.com/irt
Why go with Figaf IRT?
As we develop IRT, our focus has been to give you the greatest
number of tests while being as automated as possible. We
wanted to create an application that allows you to test 95% of
your interfaces with just a click of the mouse.
One of the best ways to test is to be able to send many different
types data on all existing scenarios. In the development of IRT we
are working to make it easier to perform tests, so you can set up
testing much faster. There is a wizard that guides the developers
through the process by simply selecting the number of ICOs to
test and how many messages to test with. IRT will then retrieve
the messages from the productive system.
To run a test, IRT sends an original message from production
and receives an output. Then it compares the output message
to the original message from production. If the two messages
are identical, the test has been successful. If there are any
differences in the values of the messages, developers can
determine if the error is a correct difference, such as a
timestamp, or whether there is a genuine error.
The tool can be used every time a test should be performed.
Current customers are using IRT in their transport strategy to
ensure that new development works and does not break existing
functionality, or when they upgrade/patch the system.
13
