Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Data Interface Test Automation for Internal & External Data Feeds

28.260 visualizaciones

Publicado el

Your organization has a major system that is central to running its business - maybe it's an ERP system (SAP, Oracle), a CRM system (Salesforce, Microsoft Dynamics), or trading systems - that system is constantly sending or receiving data feeds (generally in XML or flat file formats) to or from a customer, vendor, or another internal system.

In these circumstances, you may have little to no control over the format or quality of the data. Now this data needs to be integrated, mapped, and transformed into your systems.

Can your existing testing process handle this task?

This slide deck will describe:

+ How to take your testing process to its full potential using our Maturity Model for Data Quality & Testing
+ How to centralize and standardize your testing process
+ How to automate data interface testing for integrating your data feeds
+ How to easily compare XML files and flat files to each other and to a database
+ How to gain 100% coverage with a 95% decrease in testing time

to watch the recorded video of this slide deck, please visit:

http://www.querysurge.com/solutions/data-interface-testing

Publicado en: Tecnología
  • Sé el primero en comentar

Data Interface Test Automation for Internal & External Data Feeds

  1. 1. Bill Hayduk CEO/President RTTS How to Automate Data Interface Testing for Internal & External Data Feeds Christopher Thompson Senior Domain Expert QuerySurge Automate your Data Warehouse & Big Data Testing and Reap the Benefits built by
  2. 2. Today’s Agenda Data Interface testing • take your testing process to its full potential using our Maturity Model • centralize and standardize your testing • automate data interface testing • compare XML files and flat files to each other and to a database • gain 100% coverage with a 95% decrease in testing time AGENDA Data Interface testing • take your testing process to its full potential using our Maturity Model • centralize and standardize your testing • automate data interface testing • compare XML files and flat files to each other and to a database • gain 100% coverage with a 95% decrease in testing time • Demo built by QuerySurge™
  3. 3. built by QuerySurge™ About FACTS Founded: 1996 Locations: New York (HQ), Atlanta, Philadelphia, Phoenix Strategic Partners: IBM, Microsoft, HP, Oracle, Teradata, HortonWorks, Cloudera, Amazon Software: QuerySurge RTTS is the leading provider of software & data quality for critical business systems
  4. 4. “Unfortunately companies often don't spend enough time aligning the data testing…and validation cycles to the project timeline”. "You really need to make sure that you're validating and testing throughout the process”. - InformationWeek Question: How are you going to test the data? Failure to validate and test the process built by QuerySurge™
  5. 5. The average organization loses $8.2 million annually through poor Data Quality. - Gartner 46% of companies cite Data Quality as a barrier for adopting Business Intelligence products. - InformationWeek Data Quality Best Practices boost revenue by 66%. - Research firm Sirius Decisions built by QuerySurge™
  6. 6. Data Maturity Models & Data Interface testing built by QuerySurge™
  7. 7. Data Maturity Model - Process source: IBM Data Governance Council Maturity Model • Patterned after the Capability Maturity Model Integration(CMMI) from the Software Engineering Institute (SEI) at Carnegie Mellon University • Devised by IBM, along with 55 other companies • Few stable processes exist • “Just do it” mentality • Data-related policies become more clear & reflect the organization’s data principles. • Data integration opportunities are better leveraged. • Risk assessment for data integrity & quality becomes part of the organization’s project methodology. • Further defined value of data for more data elements • Data Governance methodology is introduced during the planning stages of new projects • Enterprise data models are documented & published • Data Governance is second nature • ROI for data-related projects is tracked • Business value of data mgmt is recognized • Cost of data mgmt is easier to manage • Costs are reduced as processes become automated • More data-related controls are documented • Metadata becomes an important part of documenting critical data elements. built by QuerySurge™
  8. 8. Sampling Level 1 Sampling a % of data by visually comparing data sets. Not repeatable. Excel, Ad Hoc Reporting Level 2 Using Excel or other homegrown method. Ad hoc reporting. Minus Queries Level 3 Utilizing SQL editor & minus queries to test data. More detailed reporting. Data Test Automation Level 4 Fully repeatable test automation, centralized reporting. Data Maturity Model – Data Testing What is the maturity level of your data testing? source: RTTS • Patterned after CMMI • Devised by RTTS based on observations Data Quality Optimizing Level 5 Full automation, tracking of ROI, predictive data issues, auditable history & results. Business value is fully understood/supported by management. built by QuerySurge™
  9. 9. A Data Interface is a set of attributes representing a given entity, used to create processes that read from, or write to, interfaces rather than directly from or to sources or targets of data. Data Interface - definition - Oracle built by QuerySurge™
  10. 10. Data Interface Testing: internal/external feeds mainframe Distributed apps web apps client/vendor data built by QuerySurge™
  11. 11. • Import into Excel • Use SQL editor to query database • Import results into Excel • Use the CountIF function • Compare column by column • Excel is incredibly slow • The process is inefficient Data Interface Testing: Popular Test Strategy built by QuerySurge™
  12. 12. Question: Is there a better way? Data Interface Testing built by QuerySurge™
  13. 13. Our Solution: Automated Testing of Data Interfaces built by QuerySurge™
  14. 14. QuerySurge is the premier test tool built to automate Data Testing What is QuerySurge? built by QuerySurge™
  15. 15. • Reduce your costs & risks • Improve your data quality • Accelerate your testing cycles • Share information with your team with QuerySurge™ you can: built by QuerySurge™ • Provides huge ROI (i.e. 1,300%)* *based on client’s calculation of Return on Investment
  16. 16. the QuerySurge advantage built by QuerySurge™ Automate the entire testing cycle  Automate kickoff, tests, comparison, auto-emailed results Create Tests easily with no SQL programming  ensures minimal time & effort to create tests / obtain results Test across different platforms  data warehouse, Hadoop, NoSQL, database, flat file, XML Collaborate with team  Data Health dashboard, shared tests & auto-emailed reports Verify more data & do it quickly  verifies up to 100% of all data up to 1,000 x faster Integrate for Continuous Delivery  Integrates with most Build, ETL & QA management software
  17. 17. QuerySurge™ Architecture Web-based… Installs on... Linux Connects to… …or any other JDBC compliant data source built by QuerySurge™ QuerySurge Controller QuerySurge Server QuerySurge Agents Flat Files
  18. 18. Data Interface Testing: Data Flow SQL SQL SQL SQL SQL SQL built by QuerySurge™
  19. 19. Collaboration Testers - functional testing - regression testing - result analysis Developers / DBAs - unit testing - result analysis Data Analysts - review, analyze data - verify mapping failures Operations teams - monitoring - result analysis Managers - oversight - result analysis Share information on the built by QuerySurge™
  20. 20. built by QuerySurge™ all QuerySurge™ Modules Design Library SchedulingDeep-Dive Reporting Run Dashboard Query Wizards Data Health Dashboard
  21. 21. Fast and Easy. No programming needed. built by QuerySurge™ QuerySurge™ Modules Compare by Table, Column & Row • Perform 80% of all data tests •Automatically generates SQL code • Opens up testing to novice & non- technical team members • Speeds up testing for skilled SQL coders • provides a huge Return-On-Investment
  22. 22. built by QuerySurge™ QuerySurge™ Modules 3 Types of Data Comparison Wizards: The also provide you with automated features for: o filtering (‘Where’ clause) and o sorting (‘Order By’ clause) Column-Level Comparison: This is great for Big Data stores and Data Warehouses where tables will have some columns containing transformations and some columns with no transformations. Many tables and columns can be compared simultaneously and quickly. Table-Level Comparison: This comparator is great for Data Migrations and Database Upgrades with no transformations at all. Many tables can be compared simultaneously and quickly. Row Count Comparison: Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades. Many tables and rows can be compared simultaneously and quickly.
  23. 23. Design Library • Create custom Query Pairs (source & target SQLs) • Great for team members skilled with SQL QuerySurge™ Modules Scheduling  Build groups of Query Pairs  Schedule Test Runs for: • immediately • at a specific date/time • automatically after build or ETL process built by QuerySurge™
  24. 24. Deep-Dive Reporting  Examine and automatically email test results Run Dashboard  View real-time execution  Analyze real-time results QuerySurge™ Modules built by QuerySurge™
  25. 25. QuerySurge Test Management Connectors built by QuerySurge™  Drive QuerySurge execution from your Test Management Solution  Outcome results (Pass/Fail/etc.) are returned from QuerySurge to your Test Management Solution  Results are linked in your Test Management Solution so that you can click directly into detailed QuerySurge results • HP ALM (Quality Center) • Microsoft Team Foundation Server • IBM Rational Quality Manager Integration with leading Test Management Solutions
  26. 26. 26 QuerySurge Value-Add QuerySurge provides value by either: in testing data coverage from < 1% to upwards of 100% in testing time by as much as 1,000 x combination of in test coverage while in testing time built by QuerySurge™
  27. 27. Return on Investment (ROI) • redeployment of head count because of an increase in coverage and decrease in need for testers • an increase in better data due to shorter / more thorough testing cycle, possibly saving $ millions by preventing bad data. built by QuerySurge™
  28. 28. Sampling Level 1 Sampling a % of data by visually comparing data sets. Not repeatable. Excel, Ad Hoc Reporting Level 2 Using Excel or other homegrown method. Ad hoc reporting. Minus Queries Level 3 Utilizing SQL editor & minus queries to test data. More detailed reporting. Data Test Automation Level 4 Repeatable test automation, agreed-upon process, centralized reporting. On which Level should your process be? Data Quality Optimizing Level 5 Full automation, tracking of ROI, predictive data issues, auditable results. Business value is fully understood/supported by management. Data Maturity Model – Test Execution built by QuerySurge™
  29. 29. Ensuring Data Warehouse Quality Demonstration Christopher Thompson Senior Domain Expert QuerySurge built by QuerySurge™ To see the video of this demonstration please visit: http://www.querysurge.com/solutions/data-interface-testing
  30. 30. • “The IBM Data Governance Council Maturity Model: Building a roadmap for effective data governance”, October 2007. • “Data Interfaces”, Enterprise Data Quality Help, Oracle • “What is CMMI?”, CMMI Institute • “Capability Maturity Model Integration”. Wikipedia References built by QuerySurge™

×