This a Architecture Solution to the customized batch file processing and online processing using Apache Camel and EAI concepts.

1. PROCESS EXECUTION
AUTOMATICS, INDIVIDUALS AND
MASSIVES
Abimael Desales López 22/10/2015

2. Processes
Segment
bootstrap
Individual
Massive
Automatic
Individual
Massive
Automatic
Processsubtypes

3. Automatic Process
Automatic
Verifypreconditions
Quartz
Controller Executor
Scheduler
Getparameters
VerifyMatrixcoexistenc
ExecuteBusinessRules
Calculationofthreads
ThrownThreads
Makecalculations
Writeoutputfiles
Dumptodatabase
DB
DB

4. Automatic processing it will carry out through a component whose
mission is verify execution times by the means of scheduler that it will
have maped the time and frecuency of each process in data base, and
further it will have the ability to build the expression that feed to
quartz to make the scheduling of task being performed.
It plans split the automatic processing in two coarse components that
it will execute each one their corresponding function, the (fisrt)
Controller it will verify every input of the process and it will decide if it
going to execute, fulfilling its pre-conditions, defining the commit
frecuency and number of threads in parallel to execute. The (second)
Executor is responsible to execute the process business logic.
Automatic Process

5. Individual Processing
Process
Bootstrap
Individual
SubtypesofProcess
F6s f5ds9f f9sf55 f6sd5f f6sd5lm ñlk
F5d f5d6xf d5f6d 5fd6s f6ds5f6 wa
Result
DB
Getparameters
VerifiyConvivenceMatrix
ExecuteBusinessRules
MakeCalculations
Typing
Processing
DB

6. In the individual processing typing the data in screen and accepting it;
to begin the processing the system it will create a ticket, will assign it a
ticket number to the request (if and only if the validation rules have
been successes and the required precondition is ok), it will get the
process parameters, if applies their execution this will be carried out,
in other way the ticket is automaticly canceled. The business rules will
be get and applyed in predefined order for the process. If all
processing was executed successfully and the business rules are ok
then it begin the dump of information to database, inserting and/or
updating registries in the respective tables of the process, and finally
updating the ticket status to PROCESSED.
Individual Processing

7. Massive Processing
Segment
Bootstrap
Massive
SubtypesofProcesses
CAMEL
TXT
XLS
CSV
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5
F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d
Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5
85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2
Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2
6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f
65ds56sd5f6f5d6g5df5gdf6df6h5gfh5
Df65f5d65gd6fg56df5gdf6gdf5gd6fgf
5dgffd6g5df6g5df5gdf5g651df5g6f5d
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5
F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d
Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5
85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2
Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2
6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f
65ds56sd5f6f5d6g5df5gdf6df6h5gfh5
Df65f5d65gd6fg56df5gdf6gdf5gd6fgf
5dgffd6g5df6g5df5gdf5g65df5g62f5d
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5
F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d
Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5
85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2
Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2
6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f
65ds56sd5f6f5d6g5df5gdf6df6h5gfh5
Df65f5d65gd6fg56df5gdf6gdf5gd6fgf
5dgffd6g5df6g5df5gdf51g65df5g6f5d
OK File
FAILED File
OK File
FAILED File
OK File
FAILED File
CAMEL
Result of Processing
DB
Processing
Export
Export
Export
Controller Executor

8. Massive Processing
Confirm
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5
F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65d
Dsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f5
85fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2
Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf2
6df5s6df6ds5f5sd6f5s65f6sd5f65ds6f
65ds56sd5f6f5d6g5df5gdf6df6h5gfh5
Df65f5d65gd6fg56df5gdf6gdf5gd6fgf
5dgffd6g5df6g5df5gdf5g651df5g6f5d
OK_File
CAMEL
Processing
Controller Executor
Calculationofthreads
ThrownThreads
Definecommits
GettheFile
Creategenericlayout DB
Thread 2
Thread 3
Thread 4
Thread n-1
Thread 5
Thread n
Thread 1
Commit x Thread and saved of processing data to
keep the trace to reprocess the file in case of
failure
Updating to ARCH_PROCESSED
Here it will execute final
insert

9. Massive Processing
In the files massive processing highlight three generales steps:
• File load and format recognition.
• File processing, upon their format, and dump in database the
summary of the process.
• Dump in database of each file records (or lines) processed.
To carry out the first two steps it is intended to exploit the features of
the framework Apache Camel, and apply the procedure made in the
proof of concept.

10. The tasks that will be carry out through Camel in the first stage of the
massive processing are below:
• Routing with dependence of file format
• Get the file registries for the processing
• Validation of structure of the file
• Locate the suitable place to apply the business rules.
• Depending on the result, on the rules/validations enforcement
decide wheter the registry will be write in a OK file or in a FAILED file.
• Routing the outcoming files to a specified route.
Massive Processing

11. The task to be carry out by Camel in the second phase of the massive
processing (since that is already integrated the OK file) will be the
following:
• Obtaining the OK file
• Receiving as a parameter the number of threads to create.
• Do the parcial Commits towards database by each thread.
• Saved trail processed and not processed blocks
• Positioning in the appropiate place in the file to run only a block of
the same, or no secuential blocks as effect the start of processing of
failed registries.
• Writing to database the summary of failed records of the file.
• Summary of cuantity by columns when is required.
Massive Processing

12. Structure of the Processes Manager
The Process Manager it will be a component composed
of another 3 main components, the Scheduler, the
Controller and the Executor.

13. 13
Design Guidelines
10/22/2015
Solution Design Processing
Quartz Job
(Modulo)
Web App Controller ExecutorJob Business Logic
Business
Tables
Processes Manager Components Application ComponentsApplication
Java Classes
(thread)
Java Classes
(Process)
B/L
Java
Classes
Call
Call
Repository
Business
Application DB
Shell Script
Specific Tasks
Call
Shell Unix
►Run
Web App
Web App
Restringed
Admin
SuperAdmin
OS
JVM
OS
JVM
Execution
engine
Execution
engine
OS
Controller Executor
Trámite
1
Trámite
2
Trámite
3
Trámite
4
Assign Job ID
and register to
queue
Get and execute
Job
Business DB
Block 1 Block 2 Block 3

14. Flow from loading the file
Web Sytem
UploadFileBeanuploadFile(…)
Upload File to
Server
Create File
Structure
Create Ticket
Get the tickets
to Process
Camel class
move file to
processing
path and route
it to right
method
ProcessedFile
insert()
Get the
Shedule
Put the OK
and FAILED
files
ProcessedFile
Service
update()
TicketService
update()
Process
registries by
Camel (apply
Business
Rules)
UploadFileBean

15. Structure of the Processes Manager
Scheduler Responsabilities:
• Schedule the execution times of the process, if are there some
window for modify de execution times, have a funcionality that is
capable to traslate date-time to Quartz expression and back again.
• Create a Job Group by process.
• Start each process to the time of day that is scheduled in
automatic case, or at once in massive e individual cases, this is to
pass the control flow to the Controller.

16. Approach of the Scheduler
We look that a Job is scheduled, which is only responsible to invoke to one Manager, although
this could change if necessary invoking something more.
Manager is responsible for orchestrating their services and these ones to their corresponding
Daos.

17. Structure of the Processes Manager
Responsabilities of the Scheduler:
• It will be created so many Job classes as different types of services
you have to run, namely, services will can be grouped in a single
Job so that these conform transactional units or database access,
each Job must implement the org.quartz.Job interface.
• The package structure will be as follows:
– For the orchestator classes
com.axxi.gestor.procesos.planificador
– For the Job classes:
com.axxi.gestor.procesos.planificador.job
– For the job launchers (classes with main method) if required:
com.axxi.gestor.procesos.planificador.launcher

18. Structure of the Processes Manager
package com.axxi.quartz;
import org.quartz.Job;
import org.quartz.JobDataMap;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
// Otros imports
public class ControllerJob implements Job {
// Services injected by Spring should be injected by setter injection
private IProcessXManager processXManager;
public void setProcessXManager(IProcessXManager processXManager){
this.processXManager=processXManager;
}
public void execute(JobExecutionContext arg0) throws JobExecutionException {
//Get JobDataMap from the context
JobDataMap dataMap = arg0.getMergedJobDataMap();
processX.orchest(datamap.getWrappedMap());
}
}

19. Structure of the Processes Manager
Responsabilities of the Scheduler:
• Inside each Job in the execute(…) method it should get the
context’s JobDataMap parameters and set them to the service.
JobDataMap dataMap = arg0.getMergedJobDataMap();
Solicitud solicitud = (Solicitud) dataMap.get("solicitud");
ArchivoProcesado archProcesado = (ArchivoProcesado) dataMap.get("archProcesado");
• In the Job class it will be injected the service by Spring through
setter injection.
• The Manager, Service and Dao clases still work the same way as in
the web project, injecting the instances in the Job, Manager and
Service classes respectively, by Spring.

20. Manager Example
public class TramiteManager implements ITramiteManager {
// Get the instance by Spring
private ISolicitudService solicitudService;
private IArchivoProcesadoService fileProcessedService;
private IVerificadorPrecondicionesService verifyPreconditions ;
//Setters …
public void orchest(Map<String, Object> parameters) {
Solicitud solicitud = (Solicitud) parameters.get("solicitud");
FileProcessed fileProcessed = (FileProcessed) parameters.get(“fileProcessed");
Integer xParameter = (Integer) parametros.get("xParameter");
solicitudService.createSolicitud(solicitud);
fileProcessedService.createFileProcessed(fileProcessed);
verifyPreconditions.verifyPreconditionsAfiliacion(xParameter);
}
}

21. Structure of the Processes Manager
Responsabilities of the Scheduler:
• It will be created one Scheduler class for massive processing, one
for individual processing and one for automatic processing, each
one must have as minimum one method by each process type.
– MassiveScheduler.java
– IndividualScheduler.java
– AutomaticScheduler.java
• Each class must implement one specific interface with the
required methods for the functionality.

22. Structure of the Processes Manager
Responsabilities of the Scheduler:
• It must create one properties file with the mapping of groups with
the processes name.
#Mapping of execution groups with processes
Grupo1=Affiliation
Grupo2=Collection
Grupo3=ABCAdministrative
Grupo4=Unification
Grupo5=Withdraw
Grupo6=ReinvestmentResources
Grupo7=RGPE
Grupo8=Yield
Grupo9=Comission
Grupo10=Loan

23. Structure of the Processes Manager
Responsabilities of the Scheduler:
• Must have one orchestrator class by each process type (massive,
automatic, individual) by way of pseudo-workflow to execute the
flows.
• Each one of the orchestrator class must be within of the next
package:
com.xxx.manager.processes.scheduler.launcher
• In order to maintain there should be one service to build a
request and one processed file of each process type since each
process has specific values for their request.

24. Structure of the Processes Manager
Responsabilities of the Controller:
• Verify the preconditions to start the process.
• Determine if the process goes forward in function of the matrix of
coexistence.
• Get the parameters to start the process, this is, the layout, and maybe
some specific execution parameters, as the ID of the process, etc.
• Give way to the Executor so it get the file and begin their processing,
passing it the entire set of parameters obtained.
• When applicable, depending of the process, make the calculations of
threads, and launching threads.
• Control the threads processing and commits, saving the track of the
processed registries, when applicable the processing for threads.
• Having ability to restart.

25. Structure of the Processes Manager
Responsabilities of the Controller:
For the phase of final processing:
• Getting file to be processed with the corresponding parameters (i.e.
enterprise, product, process y subtype), ticket number.
• Invocation to generic layout maker class.
• Determination of the number of commits.
• Calculation of execution threads –if is staticly assgned, this step is not
necessary.
• Launching and control of threads, unexecuted blocks recovery.
• Allow recovery of certain blocks of the file.

26. Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• Receive the parameters get in the Controller.
• Start the flow of Camel processing.
• In case of to be interrupted the flow of Camel, send one flag to the
Controller that the process was interrupted in certain point or in
certain block that includes since the X registry to the Y registry
(may be writing to database), so that this can reprocess the file.
• Notify the conclusión of the processing to the Controller
• Get a generic service that records and updates the request and the
table of processed files.

27. Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• Receiv as parameters the generic layout, the ticket file number,
the frecuency of commits.
• Invoke to the class of each process so that it executes the insert to
the database.
• Return the control of the flow to the Controller so that this
register the block of processed information.
• Invoke to the generic service so that it updates the process state.

28. Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• It must be created one transactional method for each process to
insert to database the lists of particular layouts corresponding to
each process, each one of this methods must be developed by the
owner responsible of that process logic, can’t be a generic
functionality. The method will receiv one list of Map from the
Camel class to get registries. Within this method it will be invoked
one method that assemble the generic layouts.
• The method must return a Map holding three data:
– 1.- Position of initial registry
– 2.- Position of final registry
– Flag indicating if it can to do the insert (True or False)

29. Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• It must be created one method for the assembly of the generic
layout that will receiv one Map conteining the field ID - value of
the field, and the last data of the Map will be the number
(position) of the registry in the original file to track what is being
inserted, and we can be aware not inserted records to try the
rework/restart. This method must return one bean with the
received data of the specific layout, but conteining all possible
generic fields.

30. Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• The required methods will have the next suggested signs:
LayoutGenericoRecaudacion armarLayoutRecaudacion(Map<String, String> layGenerico);
@Transactional
Map <String, Object> insertRecaudacion(List<Map<String, String>> listaLayouts){
LayoutGenericoRecaudacion layGenerico;
for(Map<String, String> map : listaLayouts){
layGenerico = armarLayoutGenerico(map);
//Processing …
}
}

31. THANKS
Abimael Desales López
Software Architect
Ibermática México
a.desales@ibermatica.com
It.adesales@gmail.com
Feel free to contact me!