Shahid Lecture-9- MKAG1273

MAL1303: STATISTICAL HYDROLOGY
Trend Analysis
Dr. Shamsuddin Shahid
Department of Hydraulics and Hydrology
Faculty of Civil Engineering, Universiti Teknologi Malaysia
Room No.: M46-332; Phone: 07-5531624; Mobile: 0182051586
Email: sshahid@utm.my
11/23/2015 Shamsuddin Shahid, FKA, UTM
You created this PDF from an application that is not licensed to print to novaPDF printer (http://www.novapdf.com)

Time Series
Measurements of a variable taken at regular intervals over time form
a time series.
• The times are usually equally spaced and form a continuous
sequence but can be unequally spaced and often contain gaps.
• Each sample can be a snapshot of the variable or some form of
average value taken over a time sample.
Examples:
• Daily rainfall recorded at a station
• Monthly record of river water chemistry
• Weekly fluctuation of groundwater level
• Daily record of river discharge

Plotting of Time Series Data
A time plot of a variable plots each observation against the time at which it
was measured. Time is marked on the horizontal scale, and the variable of
interest is marked on the vertical scale. Connecting sequential data points
by lines helps emphasize changes over time.
Annual Total Rainfall in Different Years

Component of Time Series
• Secular Trend (T): Secular trend is relatively smooth long-term
movements of a time series. It can be linear or nonlinear.
• Cyclical Variation (C): Rises and falls over periods longer than one year
• Seasonal Variation (S): Patterns of change within a year, typically
repeating themselves
• Irregular Variation (I): Effects of unexpected or irregular occurrences

Trends: A trend in a time series is a persistent, long-term rise or fall.
Seasonal Variation: A pattern in a time series that repeats itself at known
regular intervals of time is called seasonal variation.
Trend Analysis

• We have time series of rainfall data. We want to see whether
the rainfall of the area has changed over a time?
• Groundwater level data are recorded for years. We want to see
if there any change in groundwater level.
• We have temperature records of an area of few decades. We
want to see if the global warming also evident at the area?
• Concentrations and loads of phosphorus have been observed at
a channel over a 20-year period. Have concentrations and/or
loads changed over time?
Trend Analysis: Example Questions?
Tests for trend have been of keen interest in environmental sciences over
the last two decades.

Trend Analysis: Example Questions?
Rainfall record of a station for fifty years. Visually or using general
statistics or mathematics, it is not possible to measure whether there is
significant change in the rainfall. We need to use trend analysis.

Trend Analysis
Trend analysis refers to the concept of collecting information and
attempting to spot a pattern, or trend, in the information.
In other words trend analysis can be defined as below:
Trend analysis is a mathematical technique that uses historical results to
predict future outcome.
Although trend analysis is often used to predict future events, it could be
used to estimate uncertain events in the past, such as how many floods
probably happened between two dates, based on data such as the
average number of floods historically occurred.

Linear and Non-linear Trends
 In Linear trend, the model function is a linear combination of parameters. Such
as y = mx + c, i.e the mode can be represent a straight line.
 In Non-linear trend, the parameters appears as a non-linear combination of
parameter. Such y = x3 + 5e-3

Spatio-Temporal Modeling
Spatio-temporal analysis or modeling tells us how a variable is
changing over space and time.
1. Temporal Change: How a variable changes over time
2. Spatial Change: How a variable changes over space
Spatial-temporal models arise when data are collected across
time as well as space.

A series of observations of a
random variable, e.g.,
rainfall, concentration, well
yield, etc. are collected over
some period of time. We
would like to determine if
their values generally
increase or decrease
(getting "better" or
"worse"). In statistical terms
this is a determination of
whether the probability
distribution from which
they arise has changed over
time.
Temporal Modeling

Spatio-Temporal Modeling
Trends are spatially
interpolated to model the
spatio-temporal pattern of
rainfall.
This extremely important in the
context of environmental
change for environmental
management, policy planning,
decision making, disaster
mitigation, adaptation, etc.

We test hypothesis:
The Null Hypothesis, H0 : There is no trend.
Alternative Hypothesis, HA : There is a trend.
• Null hypothesis can not be rejected does not mean that no trend in
the data has been proved. It says that trend is not evident from the
data.
• If the null hypothesis is rejected, we try to find the magnitude to
trend.
Temporal Modeling: Trend Analysis

One of the major task in trend analysis is to find the suitable
method to test the trend. Selection of method depends on nature
of your data.
• A test may be slightly more powerful in one instance but may be
much less powerful in some other reasonable cases.
• The test selected should therefore be robust -- it should have
relatively high power over all situations and types of data that
might reasonably be expected to occur.
Trend Analysis: Selection of Method

Some of the characteristics commonly found in water
resources data, and discussed in this chapter, are:
• Distribution (normal, skewed, symmetric, heavy tailed)
• Outliers (but true measurement)
• Cycles (seasonal, weekly, tidal, diurnal)
• Missing values (a few isolated values or large gaps)
• Censored data (less-than values, historical floods)

First, the suitable method depends on whether the X data has been
adjusted or not. For example, we want to find trend of chemical
concentration in a stream. We have collected concentration data at
different flow levels. As our intension to see the change in chemical
concentration at constant stream flow, we first need to calculate the
flow-adjusted concentration before trend analysis.
Types of method:
• Simple trend tests (not adjusted for X).
• Tests adjusted for X. When there is some attempt to remove
variation caused by other associated variables, we us these test.

The methods used for trend test:
• Parametric method – Regression analysis
• Non-parametric method – Mann-Kendall Test
Trend test just tells us whether there exists a significant
change or not over the time
Magnitude of change:
• Parametric method – Regression analysis
• Non-parametric method – Sen’s slope method

Trend Estimation by Linear Regression
Linear regression is a statistical technique used for finding the best
fitting straight line for a set of data. The resulting line is called the
regression line.
Regression requires one independent variable and one or more
dependent variable. In case of trend analysis independent variable
is always time.
Trend analysis through regression is the process of finding the
equation that best describes the change of variable with time.
Trend analysis using linear regression is the best line fitting method

The Least Squares Solution
 We define the best fitting line as the one that has the smallest total squared
error
 This line is commonly called the least squares solution. In symbols the linear
equation is
y = mx + c or y = a + bx
 For each value of x in the data, this equation will determine the point on the
line that gives the best prediction of y
 The problem is to find the specific values for m and c that will make this line
the best fitting. Least squares estimate of m
Where:
SP is the sum of products
SSx is the sum of squares for the X scores and
m =
SP
SSx
= −

The Least Squares Solution
 This line is commonly called the least squares solution. In symbols the linear
equation is
y = mx + c or y = a + bx
 For each value of x in the data, this equation will determine the point on the
line that gives the best prediction of y
 The problem is to find the specific values for m and c that will make this line
the best fitting. Least squares estimate of m
Where:
SP is the sum of products
SSx is the sum of squares for the X scores and
m =
SP
SSx

Time (Hour) DO (mg/l)
Trend Analysis Example
Variation of Dissolved Oxygen (mg/l) with Time (hour)

Trend Analysis Example

Null Hypothesis, H0 : There is no trend, m = 0
Alternative Hypothesis, HA: There is a trend, m ≠ 0
If |t(calculated)| > t (critical, α, n-2), Null hypothesis rejected. The change is
significant.
Test of Significance of Change

Confidence Interval Estimates of Y
A confidence interval reports the mean value of Y for a given X.

Assumptions in Linear Regression Trend Test
Assumption in parametric regression for trend analysis:
 Y values are normally distributed. The means of these normal
distributions of Y values all lie on the straight line of regression.
If assumption is not met, we need to go for non-parametric trend test.

In Mann-Kendall test (Mann 1945; Kendall 1975) the data are
evaluated as an ordered time series.
1. Each data is compared to all subsequent data.
2. The initial value of the Mann-Kendall statistic, S, is assumed
to be 0 (e.g., no trend).
3. If a data from a later time period is higher than a data from
an earlier time period, S is incremented by 1.
4. On the other hand, if the data from a later time period is
lower than a data sampled earlier, S is decremented by 1.
5. The net result of all such increments and decrements yields
the final value of S.
Mann-Kendall trend test

Null Hypothesis: There is no trend in time series data

Variation of Temperature with time

Mann-Kendall trend test:
Example

Mann-Kendall trend test: Example
VAR (S) = 487.7

Mann-Kendall trend test: Example
S = 93
VAR (S) = 487.7
Z (calculated) > Z (critical)
Hypothesis rejected. There is a trend at 99% level of significance
Result: Temperature is increasing with time

Linear Regression Analysis

Magnitude of Change
• Kendall-Theil non-parametric rank based method
• Sen’s Slope Method
Related to Kendall-tau rank correlation, it is a robust nonparametric
line applicable when Y is linearly related to X.
These are the advantages of Kendall-Theil or Sen’s Slope methods in
contrast to OLS Regression are:
• They not depend on the normality of residuals for validity of
significance tests
• They are not strongly affected by outliers

If the model form specified in a regression equation were known to be correct
(Y is linear with T) and the residuals were truly normal, then fully-parametric
regression would be optimal (most powerful and lowest error variance for the
slope). Of course we can never know this in any real world situation.
If the actual situation departs, even to a small extent, from these assumptions
then the Mann-Kendall procedures will perform either as well or better.
When one is forced, by the sheer number of analyses that must be performed
to work without detailed case-by-case checking of assumptions, then
nonparametric procedures are ideal.
Non-parametric tests are always nearly as powerful as regression
The failure to edit out or correctly transform a small percentage of outlying
data will not have a substantial effect on the results.
Simple Trend Analysis: Comparison

• Variables other than time trend often have considerable influence on
the response variable Y. These influencing variables are called
exogenous variables.
• These exogenous variables are usually natural, random phenomena
such as rainfall, temperature or streamflow.
• It is necessary to remove the influence of exogenous variables over Y to
find the desired trend.
• By removing the variation in Y caused by these variables, the
background variability or "noise" is reduced so that any trend "signal"
present can be seen.
• The ability (power) of a trend test to discern changes in Y with T is then
increased.
• The removal process is not often easy. It often involves modelling.
Trend Analysis: Adjusted Variable

Adjusted Variable
Let us consider that whether
there is any trend in chemical
concentration in a stream. We
have collected concentration
data at different flow levels. As
our intension to see the change
in chemical concentration at
constant stream flow, we first
need to calculate the flow-
adjusted concentration before
trend analysis.
Concentration data without
adjusted (up) and after adjusted
(down) will give different
trends.

Adjust A Variable
We can simply adjust a variable by
simple regression analysis.
For our example, if you find the
regression equation between
concentration and flow as,
Concentration = 1.3 Flow + 0.87
Using this equation, we have
calculate the residual of
concentration value. The residuals R
from the regression describe the
values for the Y variable "adjusted
for" exogenous variables. Resultant
residula values are then used for
trend test.

Adjust A Variable: Example
We want to see, if there any change in salinity in coastal estuary with time. We
have time series data of water salinity. Problem is that water salinity depends on
rainfall. High rainfall dilutes the salt and reduce the salinity value. Therefore, first
aim is to remove the influence of rainfall on salinity.

Adjust A Variable: Example
Calculate the regression
equation between rainfall and
river discharge. Use that
equation to find the residual in
predicted value. This residuals
are due to error in prediction
and change due to time.

Adjust More than One Exogenous Variable
Sometimes, a variable may be influenced by many variables. Such as,
some chemical concentration in water depends on streamflow, rainfall
and temperature.
We need, multiple regression to find the relation between the variable
whose trend to be found and the exogenous variables.
Y = a0 + a1x1 +a2x2 + a3x3
The equation is then used to predict the value and calculate residual.
Trend analysis is carried out over the residual to find the trend.

Adjustment With Non-parametric Regression
• Kendall-Theil non-parametric rank based method
• Sen’s Slope Method
if the variables are not normally distributed or not linearly related, we
need to transform the data before adjustment.

 What can we do to explore other (nonlinear) relationships?
 One possibility is to transform one of the variables. For
example, instead of using Y as the dependent variable, we
might use its log, reciprocal, square, or square root.
 Another possibility is to transform both of the variable in the
same way.
 There are many other transformations, but log, reciprocal,
square, or square root are the most common.
Transformation before Adjustment

Mixed Approach: Mann-Kendall on Regression Residuals
 Once adjusted, we can use either parametric or non-
parametric method on the adjusted variable to find the
trends.
 Usually, if the adjusted variable does not obey the rules that
are necessary for application of parametric methods, then we
can use non-parametric methods for trend tests.
 Therefore, in mixed method, data is adjusted with
parametric method and trend is analyzed by non-parametric
method.

Dealing With Seasonality
• There are many instances where changes between different
seasons of the year are a major source of variation in the Y
variable.
• For example, rainfall is different in different season, river water
salinity varies with season, etc.
• As with other exogenous effects, seasonal variation must be
compensated for or "removed" in order to better discern the
trend in Y over time.
• If not, little power may be available to detect trends which are
truly present.
• We may also be interested in modeling the seasonality to allow
different predictions of Y for differing seasons.

Methods used to dealing with seasonality:
• Seasonal Kendall test for trend on Y
• Parametric Regression of Y on T and seasonal terms
• Mixed Regression of deseasonalized Y on T - The seasonal
Kendall test can be applied to residuals from a regression of Y
versus X
Dealing With Seasonality

We can forecast the future temperature using the Regression equation:
Temperature = -0.004 x Year + 27.752
What will be the temperature in January, 2008?
January, 1999 is the 1st month of the time series. January, 2008 is the 85th Month.
Therefore, Temperature in January, 2008 = -0.004 x 85 + 27.752
=27.412 Deg Centigrade.
The trend line ignores seasonal variation in the temperature. Using the equation
above to forecast Temperature for say, July 2008, will result in a gross
underestimate.
Trend Analysis: Linear Regression

• The seasonal Kendall test accounts for seasonality by computing the
Mann-Kendall test on each of m seasons separately, and then
combining the results.
• Therefore, a particular season data of a year is compared with that
season data of other months.
• No comparisons are made across season boundaries.
• Kendall's S statistic Si for each season are summed to form the overall
statistic Sk.
Kendall Test for Seasonality



m
i
ik SS
1
The null hypothesis is rejected at significance level α if |ZSk| > Zcrit where
Zcrit is the value of the standard normal distribution with a probability of
exceedance of α/2.

Kendall Test for Seasonality

• Mixed procedure involves deseasonalizing the data by subtracting
seasonal medians from all data within the season, and then
regressing these deseasonalized data against time.
• Multiple Regression With Periodic Functions
Deseasonalizing Data

Mixed procedure involves deseasonalizing the data by subtracting
seasonal medians from all data within the season, and then regressing
these deseasonalized data against time.
One advantage of this procedure is that it produces a description of the
pattern of the seasonality (in the form of the set of seasonal medians).
However, this method has generally lower power to detect trend than
other methods, and is not preffered over the other alternatives.
Subtracting seasonal means would be equivalent to using dummy
variables for m−1 seasons in a fully parametric regression.
Mixture Methods

Test for Seasonality:
Mixture Methods

Deseasonalization:
Multiple Regression With Periodic Functions
The simplest case, one that is sufficient for most purposes, is:
Y = 0 + 1•sin(2πT) + 2•cos(2πT) + 3 [12.3]
where "other terms" are exogenous explanatory variables such as flow,
rainfall, or level of some human activity (e.g. waste discharge, basin
population, production). They may be continuous, or binary "dummy"
variables as in analysis of covariance.
The residuals must be approximately normal. Time is commonly but
not always expressed in units of years.

• One advantage of the mixed method is that it produces a
description of the pattern of the seasonality (in the form of the set
of seasonal medians).
• However, the mixed method has generally lower power to detect
trend than other methods, and is not preferred over the other
alternatives.
• The Mann-Kendall and mixed approaches have the disadvantages
of only being applicable to univariate data and are not amenable to
simultaneous analysis of multiple sources of variation.
• Multiple regression allows many variables to be considered easily
and simultaneously by a single model.
Comparing Deseasonalizing Method

Differences Between Seasonal Patterns
The approaches described above all assume a single pattern of trend
across all seasons. This may be a gross over-simplification and can fail to
reveal large differences in behavior between different seasons. It is
entirely possible that the Y variable exhibits a strong trend in its summer
values and no trend in the other seasons. Even worse, it could be that
spring and summer have strong up-trends and fall and winter have strong
down-trends, cancelling each other out and resulting in an overall
seasonal Kendall test statistic stating no trend.
No overall test statistic will provide any clue of these differences. This is
not to suggest they are not useful. Many times we desire a single number
to characterize what is happening in a data set. Particularly when dealing
with several data sets (multiple stations and/or multiple variables),
breaking the problem down into 4 seasons or 12 months simply swamps
us with more results than can be absorbed.

The test for homogeneity examine "contrasts" between the different
seasonal statistics. This provides a single statistic which indicates
whether the seasons are behaving in a similar fashion (homogeneous)
or behaving differently from each other (heterogeneous).
For each season i (i=1,2,...m) compute,
Sum these to compute the "total“ chi-square statistic, then compute
"trend" and "homogeneous" chi-squares:
Seasonal Patterns: Test of Homogeneity
)(/ iii SVarSZ 



m
i
itotal Z
1
22
)(

Seasonal Patterns: Test of Homogeneity
Trend:
Homogeneous
m
Z
ZwhereZm
m
i
i
trend


 122
)( .
2
)(
2
)(
2
)(hom trendtotalogeneous  
The null hypothesis that the seasons are homogeneous with respect to
trend (τ1 = τ2 = . . . = τm) is tested by comparing χ2
(homogeneous) to tables of
the chi-square distribution with m−1 degrees of freedom. If it exceeds
the critical value for the pre-selected α, reject the null hypothesis and
conclude that different seasons exhibit different trends.

Use of Transformations in Trend Studies
• Water resources data commonly exhibit substantial departures from
a normal distribution. Surface-water concentration, load, and flow
data are often positively skewed.
• Trends which are nonlinear will be poorly described by a linear slope
coefficient, whether from regression or a nonparametric method. It is
quite possible that negative predictions may result for some values of
time or X. By transforming the data so that the trend is linear, a
Mann-Kendall or regression slope can later be re-expressed back into
original units.
• One way is to take a transformation (log, square, inverse, etc) of the
data prior to trend analysis. The trend slope will then be expressed in
log units. A linear trend in log units translates to an exponential trend
in original units.
• If m is the estimated slope of a linear trend in natural log units then
the percentage change from the beginning of any year to the end of
that year will be (em − 1).

Change = 0.043 ( in log scale)
Actual Change = e0.043 -1
= 0.044
Change (%) = 0.044 x 100
= 4.4%
Sometimes, it is argued that data should always be transformed to normality, and
parametric procedures computed on the transformed data. Transformations to normality
are not always possible, as some data are non-normal due not to skewness but to heavy
tails of the distribution. You can use non-parametric test in those situations.

Step Trend
Study of long term changes in hydrologic variables can be carried out
in either of two modes:
1. Monotonic Trends: It means overall trend. It is discussed so far in
this lecture.
2. Step Trends: It compares two non-overlapping sets of data, an
"early" and "late" period of record.
Step Trends:
• Changes between the periods are called "step trends", as values of Y
step up or down from one time period to the next.
• Testing for differences between these two groups involves
procedures similar or identical to the rank-sum test, two-sample t-
tests, and analysis of covariance. Each of them also can be modified
to account for seasonality.

Step Trend
t-Test (parametric)
The basic parametric test for step trends is the two-sample t-test. The
magnitude of change is measured by the difference in sample means
between the two periods.
The disadvantages of using a t-test for step trends on data which are non-
normal -- loss of power, inability to incorporate data below the detection
limit, and an inappropriate measure of the step trend size.
Rank-sum Test (non-parametric)
The primary nonparametric alternative is the rank-sum test of step-trend
magnitude. The rank-sum test can be implemented in a seasonal manner
just like the Mann-Kendall test, called the seasonal rank-sum test. It
computes the rank-sum statistic separately for each season, sums the test
statistics, their expectations and variances, and then evaluates the overall
summed test statistic.

Step Trend: Wilcoxon Signed-Rank Test
• Compute the differences between the paired observations.
• Discard any differences of zero.
• Rank the absolute value of the differences from lowest to
highest. Tied differences are assigned the average ranking
of their positions.
• Give the ranks the sign of the original difference in the
data.
• Sum the signed ranks individually (“+” together and “–”
together)
• Wilconxon Statistics W = minimum (“+” Rank; “-” Rank)
• Compare calculated value to Wilconxon Tabulated value.
• If your value less than the tabulated value Reject Null
Hypothesis

Example: Wilcoxon Signed-Rank Test
+ Rank = 49.5; - Rank = 5.5;
W = Mininmum (+Rank; - Rank) = 5.5
H0: There is no difference
Ha: There is a difference

Wilcoxon Critical Value Table
W = 5.5
N = 10
W(cal) < W (tab)
Decision:
Reject H0. There is no
sufficient evidence to
conclude that there
exists difference
between the two
period.

Step Trend: Applicability
Step trend procedures should be used in two situations:
1. The first is when the record (or records) being analyzed are
naturally broken into two distinct time periods with a relatively
long gap between them. There is no specific rule to determine how
long the gap should be to make this the preferred procedure. If the
length of the gap is more than about one-third the entire period of
data collection, then the step trend procedure is probably best.
2. The second situation to test for step-trend is when a known event
has occurred at a specific time during the record which is likely to
have changed water quality. The record is first divided into "before"
and "after" periods at the time of this known event. Example events
are the completion of a dam or diversion, the introduction of a new
source of contaminants, reduction in some contaminant due to
completion of treatment plant improvements, or the closing of
some facility

Step Trend: Applicability
Trend Tests
Rank Tests

Trends with Censored Data
Censored samples are records in which some of the data are known only to be
"less than" or "greater than" some threshold.
The two most common examples in hydrology are constituent concentrations
less than the detection limit and floods which are known to be less than some
threshold of perception.
Example:
Arsenic (As) in groundwater values recorded in ppm as 9.1, 7.3, <5.0, 6.2, 5.2,
<5.0, etc.
The annual flood of 1987 was not sufficiently large that local record keepers
bothered to record the maximum stage.
The existence of censored values complicates the use of trend tests and the
procedures involving removal of the effect of an exogenous variable.

Mann-Kendall Test with Censored Data: Single Threshold
The Mann-Kendall test can be used without any difficulty when only one
censoring threshold exists.
Comparisons between all pairs of observations are possible. All the "less
thans" are less than the other values and are considered to be tied with each
other.
For example: If data is like below:
9.1, 7.3, <5.0, 6.2, 5.2, <5.0, …..
All <5.0 data are considered as less than other recorded values. All <5.0 data
are also considered as tied with each other.

Mann-Kendall Test with Censored Data: Multiple Threshold
When more than one detection limit exists, the Mann-Kendall test can not be
performed without further censoring the data.
For example: If data is like below:
9.1, 7.3, <5.0, 6.2, <3.0, <1.0, 5.2, <5.0, …..
How can a <1.0 and <5.0 be compared? These ambiguities make the test
impossible to compute.
The only way to perform a Mann-Kendall test is to censor and recode the data
at the highest detection limit. Thus,
If data series: 9.1, 7.3, <5.0, 6.2, <3.0, <1.0, 5.2, <5.0, …..
It becomes: 9.1, 7.3, <5.0, 6.2, <5.0, <5.0, 5.2, <5.0, …..
There is certainly a loss of information in making this change, and a loss of
power to detect any trends which may exist.

If missing data is not very large, trend test can be used without major
difficulty.
But the major question, how much missing data (or how much
completed record) in tolerable in trend test.
One reasonably objective rule for deciding whether to include a record is:
1. Divide the study period into thirds (three periods of equal length)
2. Determine the coverage in each period (e.g. if the record is generally
monthly, count the months for which there are data),
3. If any of the thirds has less than 20% of the total coverage then the
record should not be included in the analysis.
Trend Test with Missing Data

Tolerable Missing Data

Shahid Lecture-9- MKAG1273

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (8)

Similar a Shahid Lecture-9- MKAG1273

Similar a Shahid Lecture-9- MKAG1273 (20)

Último

Último (20)

Shahid Lecture-9- MKAG1273