Actionable insight on how your Oracle EPM system is performing and how to fix it
As we all know, the back-end of our Oracle EPM systems have become more and more complex. Not only does it take longer to install, but it requires both Hyperion experience and IT involvement to properly monitor and support the environment. Finit's Infrastructure Group has been involved with some of the most complex Oracle EPM environments in the United States. As a result, we have developed our own in-house tools to assist clients with Performance Monitoring, System Tracking, System Logging, Crash Analysis, and Infrastructure Best Practices. Finit has refined these tools and has incorporated them into a packaged solution called Lumberjack.
Please join us on Friday July 20th to learn about Lumberjack. Frank Zidar, head of Finit Infrastructure, will be your host. In this webinar we will provide an overview of Lumberjack and discuss how several clients are leveraging the tool to monitor their Oracle EPM environment's efficiency / performance and triage degradation in performance, systems errors, and alerts. As a result of the use of Lumberjack, many organizations have gained better insight into how to keep their EPM environments healthy and keep end users happy!
Presenter: Frank Zidar
Date: 07/13/2018
2. OUR HISTORY
A brief history of our partnerships
Oracle Gold Partner Finit
• Founded in 2002
• 90+ employees in 20+ states
• Mostly CPAs, MBAs
• 300+ Clients – all CPM / EPM
Focused
• 1,000+ CPM / EPM projects
• 20+ Fortune 100, 60+
Fortune 500, 100+ Fortune
1000
OneStream Platinum Partner
• 150+ HFM Projects
• Managed largest roll-out of
HFM in North America
• 175+ FDM Projects
• World leader in FDM
implementations and
custom solutions
• 100+ Planning / Essbase
projects
• 40+ OneStream Clients
(Market Leader)
• #1 Implementer of
OneStream
• Platinum Partner
• First OneStream
MarketPlace App
• RequestIT
3. The Finit Model
Fully aligned with
our clients, not just
bottom line $
No debt or external
ownership
Work with Finit
employees, no
subcontractors
Compensation
based on CLIENT
SATISFACTION
PRIVATELY
OWNED
CLIENT
SATISFACTION
NO
SUBCONTRACTORS
DEBT FREE
4. Our values, culture, and approach to
becoming a trusted advisor to our
customers has led to
100% CUSTOMER SUCCESS
for every project since our inception
in 2002
300+ Clients
1,000+ Projects
7. Agenda
• Why did we build this tool and what is it good for?
• Why is this tool different than other Finit Project Tools
• How does it work and how long to install and bring online?
• Common questions it helps solve and other use case detail
• How much does it cost?
8. About the Presenter
Frank Zidar
(fzidar@finit.com)
Experience
• Director of Infrastructure
• 7+ years at Finit
• Over 50 Oracle EPM installations
• Over 20 years experience with all aspects of
the software development lifecycle
Certification/Education
Bachelor of Science in Computer Science and
Mathematics, Bowling Green State University
Professional Summary
Frank lives in Cleveland, OH and has been part of Finit since 2011 and is focused on the
installation, migration, and server architecture of Oracle EPM systems. Frank has provided
these services and been involved with over 50 installations during the last 5 years and is the
Director of the Infrastructure group at Finit.
10. Why did Finit build Lumberjack?
• EPM is complex and Finit infrastructure gets involved with a very common set
of questions and support situations
• Challenge:
• Most problems occur during the close and IT, in interest of time, must simply reboot
• Often very challenging (especially with performance issues) to piece back together a root cause
long after the fact
• Performance must be captured continuously
• Log files roll over and are automatically removed
• In the past we have traveled to the client site or worked over VPN to actively monitor the
system during a close cycle
11. Why did Finit build Lumberjack?
• Tried other utilities in the past with limited success
• Time consuming and complex to setup as well as expensive
• Started as an internal project and has grown over time
• Goals of the system:
• Quick and easy setup
• No additional processes on EPM servers
• Relies on technology built into the operating system as much as possible
12. How Does It Work?
• All that is required is Microsoft .NET Framework 4.5.2 (built into Windows OS)
• No installation - unzip several standard Windows executable files
• Add configurations for each of the environments (PROD, DEV, etc.)
• Add the list of servers into each configuration
• Lumberjack will automatically determine which EPM services are installed on each server
and where the logs are located
13. How Does It Work?
• Setup the service to execute in unattended mode every 5 minutes
(configurable)
• Normally installed on a non-EPM server/computer in the datacenter to avoid network
latency and stealing CPU cycles from EPM servers
• The Lumberjack service will process all of the information and send it to a central server
located in Finit’s datacenter
• This allows us to be able to provide support without VPN and/or meetings
• Entire setup process can be completed in 15-30 minutes
• Use the Lumberjack viewing application to see the captured information from
any computer
15. EPM Log Files
• Log files!
• EPM creates tons of log files
• Many servers, many directories
• C:OracleMiddlewareuser_projects
epmsystem1_HFMFDMdiagnosticslogshfm
• C:OracleMiddlewareuser_projectsdomains
EPMSystemserversErpIntegrator0logs
• Where are they all located?
• Which logs are important?
• What should I look for?
• What does this stuff mean?
16. EPM Log Files
• Lumberjack filters out the unimportant logs
• Can configure to gather everything or only errors
• Centralizes everything into one place and displays in Excel-like
format
• Ability to drill into the details of the message
17. EPM Log Files
• There are many ways to filter and sort the data (Excel-like features)
• Text searching (all servers, all logs):
18. Performance
• What is causing our consolidations to slow?
• Is the system optimized for best performance?
• We have many times where we get the “spinning wheel of death”, why and
can it be prevented?
• Performance has significantly degraded recently, why?
• In order to provide answers to these types of questions we gather from the
server(s) the following types of information:
22. Performance
• Ability to view consolidation performance over time
• This list can be used to see problems in your consolidations and learn more
about them
23. Performance
• Consolidations grouped by: Entity, Type, Period shows something interesting:
• Drilling down we find a situation where the same consolidation executed on
the same day has very different execution times:
32. Crash Analysis
• Armed with all of the log and performance data, someone can determine why
problems have happened after the fact.
• Provide root cause for questions like:
• The system is currently down – nobody can connect to HFM (spinning wheel) and some
users aren’t even able to log in.
• The system has been unstable recently. What is the cause?
33. Crash Analysis
• We are monitoring 3 systems for a client and the morning of 6/26 it was
reported to Finit infrastructure that a significant issue occurred in the
development EPM environment the prior evening
• Around 7pm the system became unusable. No one able to log in and anyone in the system
was unable to open forms and HFM applications.
• 2 server configuration
• Server performance
showed nothing
unusual overall:
34. Crash Analysis
• Log analysis for the same timeframe displayed normal and expected system behavior until
6:46pm when the web server started reporting errors:
35. Crash Analysis
• Continued searching shows this classic error starting at 7:52pm:
• At this point, FDMEE would be unusable and would have been so for at least 20 minutes
36. Crash Analysis
• Continuing to search for the root cause:
• Ultimately, we learned the issue was with the database running out of
transaction log space and suffering performance issues leading to the full EPM
crash and restart.
37. Crash Analysis
• We have performed similar analysis at other clients and ended up
recommending tuning changes to dramatically increase HFM consolidation
performance
• Another time we discovered that I.T. had reduced the amount of RAM available
to the Essbase server VM leading to degraded performance
38. Financial Report History
• What Financial Reports do we actually utilize?
• Can be used to show if report execution is slowing over time
40. Alerts
• While monitoring the EPM environment, Lumberjack can be configured to
send out alerts
• The messages can be easily configured to go to multiple addresses or internal
distribution lists.
41. Stress Testing
• Will the system handle the expected concurrent users?
• Will the system perform well if we add additional users?
• Easy to record user actions in the system
• Once recorded, they are applied to a stress test to be played back to simulate
stress on the system
• All performance and log data is captured during playback/testing
• Additional testing can be performed with Python-based scripting utilities
• FDM data loads, multiple concurrent HFM consolidations
42. Customized Scripting
• We built a layer into Lumberjack which exposes the HFM programmer
interface (API) in a different way
• Oracle provides a Java based interface and we have created an extra layer accessible via
Python scripting
• Python is very popular and also the foundation of the scripting used in FDMEE (Jython)
• Many easy to use extensions to perform anything imaginable
• Oracle completely changed the HFM API from 11.1.2.3 to 11.1.2.4
• The added layer allows scripts built for 11.1.2.3 to be reused without modification in 11.1.2.4
and potentially beyond
• Currently analyzing the potential for use with Oracle’s cloud solutions
43. Customized Scripting
• FDM
• Import data file to a specified location in FDM
• Validate and export the mapped data
• Example: We used this at a client to perform FDM stress testing to simulate the processing
of 75 concurrent end users loading multiple GL files
44. Customized Scripting
• HFM
• Execute calculations, consolidations, and on demand rules
• Process management
• Automate promotion process for all phases
• Extract and load data files
• Execute Extended Analytics exports and work with the resulting data
• Easier to work with and more powerful than Task Flows
• Example: Lights out automation of HFM data extracts for several scenarios with a 5-day
rolling backup maintenance
• Example: Data “recast” process where data extracted from HFM according to values
entered on an Excel sheet. Converted a CUSTOM field from one value to another and
reload the data back to HFM and kick off a consolidation
45. Customized Scripting
• LCM
• Script commands to run LCM exports
• Normally done with DOS batch files which are difficult to work with
• Common problem of long file names with Financial Reporting – Lumberjack can delete
these folders without issue and with a single command
• EPM patch installation
• Single script to analyze the environment and update every component to the latest Oracle
EPM patch
• Takes a very time consuming and tedious process from hours to minutes
46. Cost
• The Finit infrastructure and integration teams have been using these tools for
several years and they have recently been packaged into a suite of software
products
• $12,000/year which includes:
• Initial setup and configuration
• Ongoing support and upgrades
• Performance monitoring and health alerts
• Monthly reports
• System performance charts
• Memory utilization charts
• Disk space utilization
• HFM consolidations executed with average execution times
• Financial report execution history with counts and average execution times
• Essbase calculation count and average execution times
• Significant error and restart counts
• HFM database error, task, and audit log counts
• Oracle EPM patch recommendations (as needed)
• Additional system tuning recommendations (as needed)
48. Upcoming Webinars
• Friday, August 17– Cash Flow 2D vs 3D – Part I
• Presented by Matt Spencer
• Friday, August 24– Cash Flow 2D vs 3D – Part II
• Presented by Matt Spencer
49. Thank you for Attending!
Today’s Presenter:
Frank Zidar
fzidar@finit.com
Greg Barrett
gbarrett@finit.com
Copy of the slides or Recording:
Email us for a copy of the slides or a
link to the recording
insights@finit.com
Past webinars:
www.finit.com/webinars
Follow us on Twitter for updates:
@Finit_Solutions
Notas del editor
12 Clients using LJ for everything from stress testing, to on-going system monitoring
Over the years other ‘consultant’ tools such as Smart Merge, HFM Security Macro, HFM Metadata Macro, Auto Data Tie Out, Essbase Nightly Back-Up
With Oracle’s move to the Cloud, many tough decisions, for first time wanted to promote a tool for companies that haven’t moved to the Cloud yet
There’s a need for many such companies
When HE went off of support, Finit also ‘zagged’
Our goal has always been to provide our clients with ALL the information
Direction is cloud for most all application vendors, software
High level overview of an internal Finit test environment
I’ll drive into the details of all the items available and how they are used to diagnose performance and error conditions in complex EPM environments
Common thing the infrastructure team looks out for are “Stuck thread errors” as they lead to UI hangs for end users
The details gathered are consistent with the Oracle published guides on EPM performance monitoring and tuning.
The chart depicts several weeks of the HFM app server CPU utilization
Shows how the CPU gets more heavily utilized initially with loads and consolidations taking place and then subsides as time goes on
Same timeframe, showing deeper level of detail
HFM process using between .5 and 1.5GB of RAM
These charts just displayed are “standard” performance charts that don’t have a significant amount of value --- until you start to overlay them with very specific EPM related details
Real example
Managing memory means putting the consolidation on the back burner
We suspect the performance gains were not solely related to server resources and performance because the consolidation times were so significantly different.
Often times we will go to the logs screen and filter all of the entries during the suspected timeframes to see if any other potential detail was captured.
This is easy to do with Lumberjack.
See how active a cube is
Similar to HFM consolidations, these screens allow an infra or Essbase consultant to be able to see problematic Smart View queries and calculations to determine what is causing the problem
The following is an example of how we used Lumberjack to find the root cause of an issue at one of our clients.
This error is interesting but not terribly informative
Able to rule out performance issues
Patch # linked to English description to make it easier to read
Scripted stress testing was just the start
Next, I’ll list some of the features available and review examples of how we have utilized this capability at clients
What EPM versions are supported?
Does it work with Linux?
Does it work with EPM Cloud applications?
Will it support 11.2?
