Organisations looking to build and offer Cloud services on Apache CloudStack need to be able to either monetize their offerings and charge for usage or monitor and report on their Cloud's consumption.
Majority of such organisations already have existing billing or business support systems and do not require an integrated billing or reporting system, provided the usage data can be exported from CloudStack in a standard and structured format such as XML, JSON, or CSV.
Tariq Iqbal of ShapeBlue provides an overview of the Apache CloudStack Usage Metrics, Usage Server and the multitude of ways for accessing the Usage Data for billing and reporting purposes, including real world use cases of some of the integration projects he has worked on.
1. CloudStack Metering
Working with the Usage Data
Tariq Iqbal
Senior Consultant
tariq.iqbal@shapeblue.com
Twitter: @TariqIqbal_ @ShapeBlue
2. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
Involved with CloudStack before donation to Apache
Built and deployed CloudStack/CloudPlatform based clouds for
Enterprises and Service Providers globally
Specialise in integrating CloudStack with Business Support and
Operational Support systems, which include:
CloudPortals (including Payment Gateways, Fraud Control solutions)
Ticketing Systems
CRM/ERP Systems
Monitoring Solutions
About Me
3. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
“ShapeBlue are expert builders of public & private
clouds. They are the leading global independent
CloudStack / CloudPlatform integrator &
consultancy”
About ShapeBlue
4. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudStack metering - What and how?
CloudStack configuration to enable Usage capture
Usage Data generation process
Methods of accessing the Usage Data
Real World Use Cases
Overview
5. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
Organisations need to be able to meter their cloud's
consumption and monetize the cloud offerings by charging for
usage
Majority of organisations already have existing billing or business
support systems and just need access to CloudStack Usage Data
Organisations tend to have different billing criteria and charging
models requiring a comprehensive integrated solution
Metering Requirements
7. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudStack does not offer any integrated usage monitoring or
billing/chargeback capability
Root Admin can view
System Capacity
Domain Admin and Users can view
VM status
Metering in CloudStack today
8. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudStack includes a Usage Server that creates summary usage records for
the various resources consumed in CloudStack
The Usage Server is an optional, separately-installed component called
‘cloudstack-usage’
The Usage Server runs at least once per day. It can be configured to run
multiple times per day and it’s behaviour is controlled by the following global
configuration settings:
•
Cloud Usage Server
9. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
As CloudStack resources are created, consumed and destroyed,
appropriate Event records are created in the ‘cloud’ database
table: usage_events
There are 110
different Events:
(51 shown)
CloudStack Events
10. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The Usage Server schedules a job at the time specified by
usage.stats.job.exec.time parameter. This job queries the ‘cloud’
database table: usage_events and generates summary usage records in
the ‘cloud_usage’ database table: cloud_usage as follows:
1. CloudStack events have been logged into the cloud.usage_event table
2. cloudstack_usage job gets a list of the latest usage events
3. Inserts the latest usage events into the cloud_usage.usage_events table
4. Helper tables populated from cloud_usage.usage_event table after
parsing
5. cloud_usage.cloud_usage table populated with aggregation range wise
data. Helper table data is used here
Usage Record Generation
11. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The Usage Server collects 14 different types of resource usage:
Usage Types
12. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The Usage Records show the amount of resources consumed by guest instances.
The Usage Record format for Allocated & Running VMs is shown below:
The other 7 Usage Record formats are:
Network Usage
IP Address
Disk Volume
Template, ISO & Snapshot
Load Balancer & Port Fwd
Network Offering
VPN User
Usage Records
15. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The usage records can be accessed through the CloudStack API
This is best done using the authentication port 8080 and requires
both the API Key and Secret Key and for the API request to be
signed
CloudStack provides the following Usage API calls with Root
Admin access and supports XML and JSON response formats
listUsageTypes – list of Usage Types accessible
generateUsageRecords - asynchronous usage record generation
listUsageRecords - provides Usage records for a date range
CloudStack API
16. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The base command is:
http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y
yyy-MM-dd&enddate=yyyy-MM-dd&signature
Append the various options:
ListUsageRecords API
18. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudMonkey is a CLI tool for CloudStack and can be used as an
interactive shell or from within a shell script to call the API
It can output the Usage Data in both JSON and Tabular form to a
file or pipe it to another application for further processing
Filtering can be used to limit the result set
It supports argument passing and shell automation
The host, port, apikey and secretkey can be configured on setup
CloudMonkey CLI
19. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudMonkey list usagerecords
The command to retrieve Usage Records is:
cloudmonkey list usagerecords domainid=7ded1404-d7fc-11e2-a70f-
080027cfaf0b startdate=2013-06-01 enddate=2013-06-23 accountid=2
20. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
The usage records can be also accessed by directly querying the
cloud_usage.cloud_usage MySQL table in CloudStack
Usage Data can be exported in a CSV format from a SELECT
query
The SQL query can become complex if the ID fields in the data
need to be de-referenced
Typically multiple passes of the query results may be required to
extract the required information
SQL
21. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
Once you are able to access the Usage Data, you will need to
design the mapping between the fields in the CloudStack Usage
Data and the fields of the target system.
Considerations:
Do you actually need all the Usage Data for all Resources?
Any UUID values of the CloudStack resources should also be
imported into the target system for reference purposes
Decide which system will be the master for the user account data
and if synchronisation is required
Mediation
23. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
MS Excel can be used to analyse CloudStack Usage Records
The usage records are imported from the cloud_usage.
cloud_usage table into MS Excel via an MySQL ODBC connection
Either PivotTables or reports can be used to present the data.
This is convenient and
great for integration
testing between CloudStack
and external systems.
MS Excel
24. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
Citrix CloudPortal Business Manager tightly integrates with the
cloud_usage database on CloudStack/CloudPlatform.
The Usage Data is cross referenced against the utility pricing and
subscription pricing in CPBM.
Scheduled Billing and Invoicing jobs generate the invoices in XML
or PDF based on the
tenants billing cycle.
Citrix CloudPortal
25. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
splunk> is an operational intelligence tool that was initially used
for collecting and indexing infrastructure logs
We then connected splunk> to the CloudStack Usage database
and automated the collection and indexing of Usage Data
Setup feature-rich dashboards
and the capability to drill down into
the Usage Data for BI reporting
splunk>
26. @ShapeBlue #CloudStack #CCC13 CloudStack Collaboration Conference 2013
CloudStack Developers Guide, API Documentation & Wiki:
http://cloudstack.apache.org/
CloudMonkey
https://cwiki.apache.org/CLOUDSTACK/cloudstack-cloudmonkey-
cli.html
How to Use MS Excel to Analyze CloudStack Usage Records:
http://support.citrix.com/article/CTX132030
CloudStack Mailing Lists (Users, Development, Marketing):
http://cloudstack.apache.org/mailing-lists.html
Further Information
27. CloudStack Metering
Working with the Usage Data
Tariq Iqbal
Senior Consultant
tariq.iqbal@shapeblue.com
Twitter: @TariqIqbal_ @ShapeBlue