This a presentation presented in the CMG International conference in 2010 about my experiences as Performance Manager during the Bank Panic in Argentina in 2001. It won the Edward Mullen Award in that year.

1. _la force de l’engagement MC
An informal Report
IT Performance Management in a
Financial Crisis

2. 2
Index
• Historical causes of the Argentinean financial crisis
• IT situation at the Bank
• The crisis starts
• Impact of the crisis on the transactional channels
• The “solution” to the crisis
• Financial impact of the “solution”
• Impact of the “solution” on software development
• Long duration impact on the country and the bank
• Lessons learned

3. 3
Argentina in the world
Argentina USA Canada
Area 1M 3.8M 3.9M
Pop. 36M 310M 31M

4. 4
Economic evolution of Argentina

5. 5
Causes of the poor economic growth
• Protectionist policies
• Populist governments
• Commercial deficit
• Fiscal deficit
• Inflation
• Public debt
Commercial deficit
Fiscal deficit
Inflation
&
Public debt
Protectionist policies
&
Populist governments
Political
Instability

6. 6
Inflation & hyperinflation

7. 7
Growth & Unemployment in the 90s
-5
0
5
10
15
20
1980 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000
% GDP per capita % Unemployment

8. 8
Situation at the Bank (2001)
• From 40 to 130 branches
• From 1500 to 4,000 employees
• From 300,000 to 900,000 accounts
• Mainframe
• 30 AIX servers
• 300 Windows servers
• Transactional channels
• Human tellers
• ATMs
• Call Center
• Internet
• A Capacity planning and performance process in place and
working

9. 9
Chronology of the crisis
• 30/11/01 : A run on the banks begins, with
the central bank reserves falling by $2
billion.
• 01/12/01 : The government imposes a
$1,000 per month limitation on personal
bank withdrawals in cash. Checks and
electronic transfers go up.
• New Year’s Eve party, the president told us
that “this will be a hard year”.
• The Internet traffic and the call center calls
suddenly skyrocketed. ATMs became
empty in a matter of hours.

10. 10
Internet transactions
•The number of concurrent user increased 5 to 6 times from one day to the other.
•We proceeded to change the architecture of the main page of the bank to reduce
the workload in the system, but it was only a partial solution.

11. 11
Internet infrastructure
Internet

12. 12
Call center
•The talk time increased
•People don’t like to talk with machines when in a crisis
•Try not to be too cheerful in the recorded message during the waiting time
•Try not to be too obvious in the recorded message during the waiting time
•Having young call center operators is cheap, but it could backfire
•Having call center operators abroad is probably inconvenient

13. 13
The trick of the withdrawal limit
Bank 1
Bank 2
Bank 3
Bank 1
Bank 2
Bank 3
Interchange bank
system
The system became saturated
The checks clearing took more time than the
allowed legal period
The number of bank accounts exploded,
branches started to open Saturdays to cope
with the demand

14. 14
Mainframe – The perfect Storm
There weren’t enough
resources for the on-line
The branches worked
in extended mode
The nocturnal batch
process took more
Than 24hs
The Perfect
Storm
There wasn’t an easy way to increase the capacity
There wasn’t hardware available to increase it
All the banks had the same problem

15. 15
Chronology of the crisis
• 30/11/01 : A run on the banks begins, with
the central bank reserves falling by $2
billion.
• 01/12/01 : The government imposes a
$1,000 per month limitation on personal
bank withdrawals in cash. Checks and
electronic transfers go up.
• 20/12/01 : A riot leaves 28 people dead, the
president has to resign. The two
subsequent presidents last only days.
Several supermarkets are looted. Argentina
declares itself in default.
• 02/01/02 : President Duhalde takes office.
• 06/01/02 : The “solution” is approved by the
Congress.
• New Year’s Eve party, the president told us
that “this will be a hard year”.
• The Internet traffic and the call center calls
suddenly skyrocketed. ATMs became
empty in a matter of hours. The IT
infrastructure collapses under the workload.
• The software development team starts to
work to adapt the systems to the “solution”

16. 16
The solution
• Default on the national debt. The country
declares itself in bankruptcy.
• End of the dollar as a legal currency in
Argentina.
• The old parity 1 dollar = 1 peso became 1
dollar = 3 pesos.
• The mortgages are converted to a 1 = 1
parity.
• All the bonds that the banks and the
persons held became “junk bonds” with 0
value.
• All the bank accounts in dollars had to be
converted to pesos, the same with
mortgages and investments.
• All dollar investments lose 66% of their
value.
• Most of the bank assets are reduced to
33%, all the banks are technically broken.
But, didn’t all this mess help protect the financial system from becoming
insolvent?

17. 17
The people were not happy with the solution…

18. 18
Working in the Solution
• Modify or replace almost all the core programs of the bank.
• Checks, mortgages, savings accounts.
• General Ledger, financial reports
• Transactional channels (web, IVRs)
• New programs like the system to manage the legal claims of
those people which were on “especial situations” were made
to extract more money that the central bank allowed
• Everything had to be done in a question of days.

19. 19
How they worked in the Solution
• There were only 3 environments :
• unitary test
• integration test
• production
• The tests were reduced to their bare bones.
• The emphasis was on the communication not in the
documentation.
• The development teams were all on the same floor.
• The technical support team was 2 floors below.
• The operations people that had to interpret the Central Bank measures
were there.
• Everybody understood the importance of what they were doing.

20. 20
Can it happen to you? Can it happen now?
YES
The IQ of the governments doesn’t look like it
has improved in the last years
The financial system is more and more
dependent on IT

21. 21
But in 10 years it should have new solutions…
YES, and new problems…
The mainframe has reserve capacity like CPUs that can be
activated on demand
Cloud computing could also help…
But the providers of these services could have an attitude of “pay
now, use later”
If the cloud is shared by companies of the same market (e.g.
banks), and the cloud is in the same geographical place it could
become saturated
What will be the impact on new technologies like smartphones?

22. 22
Long duration impact in the bank
1992 2001 2003
Branches 40 130 60
Empl. 1500 4000 2000
Accounts 300,000 900,000 700,000
Hw = ↑ Outsourced
Sw = ↑ Opportunity

23. 23
Long duration impact in the country
• -11% GDP in 2002 alone (the 2008 financial crisis in US was a - 3.8%)
• 40% inflation in 2002
• 20% of unemployment
• 63% of families below the line of poverty
• 20% of families without enough money to buy food
• The emigration rate multiplied by 6 in 2 years
• Larger between university graduates and post-graduates

24. 24
Lessons learned
The possibility of a financial crisis has to be part of the IT contingency plan

25. 25
Always have a plan “B”
“Plan B” for the rescue effort of the Chilean trapped miners

26. 26
Document the way that performance can be
improved

27. 27
Design infrastructures as flexible as possible

28. 28
Never underestimate “ad-hoc” solutions

29. 29
Keep an eye on your team’s morale

30. 30
Be prepared to throw everything you know out
of the nearest window

31. 31
Don’t worry about what top management
thinks

32. 32
CMG 2010