For many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirements on their IT infrastructure. Drivers in their business such as pricing of equities moving from decimal to penny resolution and the growing need for markets to provide improved liquidity are resulting in huge opportunities for financial gain. Taking advantage of these opportunities is, in part, dependent on the care taken in the network’s time synchronization and the management of latency. Wall Street firms who were involved in the early phases of High Frequency Trading have been early adopters of high performance timing solutions utilizing a variety of signals including GPS, IRIG, 1PPS, NTP and now the Precision Time Protocol (PTP) which allows for precision time transfer on Ethernet networks. The implementation of specific timing solutions depends on the trading infrastructure and the network topology. Through a combination of hardware, software, and careful network management, it is reasonable to expect microsecond level time-transfer from traceable time sources to Linux applications.

1. White Paper:
Synchronization for High Frequency Trading Networks
For many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirements
on their IT infrastructure. Drivers in their business such as pricing of equities moving from decimal to penny resolution and the
growing need for markets to provide improved liquidity are resulting in huge opportunities for financial gain. Taking advantage of
these opportunities is, in part, dependent on the care taken in the network’s time synchronization and the management of latency.
Wall Street firms who were involved in the early phases of High Frequency Trading have been early adopters of high performance
timing solutions utilizing a variety of signals including GPS, IRIG, 1PPS, NTP and now the Precision Time Protocol (PTP) which allows
for precision time transfer on Ethernet networks. The implementation of specific timing solutions depends on the trading
infrastructure and the network topology. Through a combination of hardware, software, and careful network management, it is
reasonable to expect microsecond level time-transfer from traceable time sources to Linux applications.
Introduction
IT managers in financial services institutions, especially those who are participating in high frequency trading, find themselves facing
multiple converging and difficult to meet system-level requirements:
• Surging network traffic in capital markets requiring accurate time-stamping of market data feeds which can approach a
pace of over one million per second.
• Processing data and making trading decisions in an ever more real time environment with high performance trading
algorithms requires precision timing within the Linux kernel and software application.
• Pressure to reduce trading latency and transaction times to sub-millisecond levels across a geographically disperse set of
trading servers and exchanges.
• Increasingly stringent regulatory oversight requiring high precision timed log files.
• Ability to post-process nearly any network scenario for optimization.
Spectracom is able to provide a time
synchronization solution for financial institutions:
precision time sources referenced to worldwide
time standards, a variety of time distribution
protocols and standards, important precision time
transfer to trading server operating systems (then
into the application), verification solutions and
portable clock calibrations where GPS is not
available.
The specific application of these products is
dependent on the needs of the scenarios that are
common for financial institutions now participating
in High Frequency Trading. However, there are
several common approaches such as using GPS as a
traceable time source, the new PTP protocol for
precise time transfer over a LAN, and sophisticated
client software to improve precise time transfer to
the application within the Linux environment.
Legally Traceable Time Through GPS
An important aspect of any time synchronization
deployment is the notion of accurate, traceable
time, versus relative time. Time is absolute and one
of the seven legally-defined units of measure.
www.spectracomcorp.com 1 | Timing & Synchronization White Paper

2. Since 1972 the world’s time standard is UTC (coordinated universal time). It is based on atomic clocks maintained by National
Metrology Institutes and is distributed in a number of ways. The most accurate and simplistic method (when afforded a clear view of
the sky) of synchronizing to traceable time is through the GPS broadcast. Spectracom offers synchronization to GPS, in addition to a
number of other time sources, to transfer time to where it is needed in many different ways. Because of the myriad of choices of
timing references and timing signals, Spectracom offers SecureSync®, a network-based master clock that is feature-rich and
configurable so you only pay for the functions that you need. Configured with a GPS receiver, SecureSync, like any other GPS-based
timing hardware, offers better than 50 nanosecond accuracy to UTC. A good start for precision timing.
A Combination of PTP Hardware and Software Improves Synchronization
Leveraging the network is a low-cost way to synchronize multiple servers (blades or 1U appliances) in the trading network. This has
been done for years through network time protocol (NTP). Today, a new network protocol is available to offer improved time
transfer over the network. Precision time protocol (PTP) uses hardware time-stamped messages to measure the delay between the
master and slave. The result is better accuracy when adjusting the slave clock for minimum offset compared to the master.
A typical SecureSync configuration combines a GPS receiver and a PTP module. This “PTP grandmaster clock” is used to synchronize
PTP slaves (clients) within the local area network with high accuracy when variation in bi-directional packet delay is low. Even under
the worst case scenario, PTP performance will be equivalent to NTP.
An example of a PTP slave is the Spectracom TSync-PCIe-PTP time code processor card. The PCI express card has an integral LAN port
to send and receive time-stamped PTP messages across the network to synchronize an on-board clock with 4 nanosecond resolution.
This PTP slave offers the best possible time precision for a computer that can accommodate plug-in cards.
Another example of a PTP slave is TimeKeeper Linux PTP client software. The software is a transparent and very lightweight “applet”
that does not alter the Linux kernel and does not require modification of existing applications. It responds to the standard Linux time
calls such as gettimeofday. However it greatly improves the accuracy of timing functions in a number of ways:
1) Improves the accuracy of the local clock by disciplining it to the PTP time reference.
2) It quickly and smoothly converges local time to the master time with a typical offset less than 10 microseconds.
3) Time is monotonic and never goes backward which can occur with standard Linux system clocks..
4) Time is computed without leaving the processor, therefore, is immune to interrupts and other operating system
“overheard” that can delay time reads and affect timing accuracy.
5) Allows the application to build a high precision time-scale by making the current time available every 30-50 nanoseconds.
Specialized time synchronization hardware improves TimeKeeper accuracy by a factor of 5. Therefore the best performance is
achieved with a combination of TimeKeeper PTP client software and the TSync-PCIe-PTP card.
The following describes the implementation of synchronization solutions in several scenarios using GPS, PTP, hardware master
clocks, PCI express cards, and timing management software. Also included are solutions for GPS-denied environments and NTP
improvement software.
Typical Network Scenarios
Scenario I: Time synchronization within a local trading center, or between trading centers.
Assumptions:
• Timing packet delay variation (PDV) is within acceptable limits to produce the required time synchronization performance
• GPS is available in all trading center locations
• Most servers requiring synchronization are blade servers, however, some critical 1U servers require the highest level of
synchronization
• All servers are running Red Hat enterprise Linux
www.spectracomcorp.com 2 | Timing & Synchronization White Paper

3. Spectracom Solution:
• Network Master Clocks: SecureSync® PTP Grandmaster (redundant pair)
• Linux Time Client: TimeKeeper™ PTP Client
• Local Hardware Clock/PTP Slave (for 1U application servers): TSync-PCIe-PTP plug-in card
• Synchronization Hardware Testing: STA-61 Sync Tester/Analyzer
As critical infrastructure, it is typical to deploy network master clocks as redundant pairs. Two SecureSync GPS PTP grandmaster
clocks require their own GPS antenna (and RF cabling) with a clear view of the sky. PTP includes a “best master clock” algorithm so
no special switches are required to accommodate a failover from one SecureSync to the other. Special care is required to minimize
packet delay variation (PDV) by eliminating as many indeterminate time delays in the network as possible and practical. Timing
accuracy is degraded due to timing packet path asymmetry. Use low traffic or directly connected PTP masters and slaves via hubs
since switches and routers queue messages depending on network conditions. Alternatively, special PTP-enabled routers can be
deployed. Known as PTP transparent clocks, they are able to compensate for any delay of PTP messages.
PTP slaves are deployed as PCI express cards for 1U application servers that can accept such hardware, and PTP Linux clients. Timing
performance across the network can be analyzed by measuring the phase of the 1 pulse-per-second signals from the PTP
grandmaster and the PTP hardware slave.
This configuration can be duplicated across all data centers involved in the trading scenario. When each LAN is synchronized to a
traceable time source via a local master clock, then any data passed from one LAN to another can be perfectly correlated –
extremely important when managing latencies.
Sidebar: Table 1 provides typical performance of a variety of time transfer protocols assuming minimal asymmetry between the PTP
grandmasters and slaves.
Table 1: Time Synchronization Performance of GPS-based Time Transfer Technologies
PTP w/ SW timestamps via TimeKeeper PTP w/HW timestamps via PCIe card NTP w/existing standard NTP Server
alone and TimeKeeper
4-5 microseconds 1 microsecond 10 milliseconds
www.spectracomcorp.com 3 | Timing & Synchronization White Paper

4. Scenario II: Time synchronizing within a GPS-denied trading center.
Assumptions:
• Either the remote trading center is connected to another network with access to GPS and the variable path delay between
networks is managed to minimize asymmetry or,
• A procedure can be followed to calibrate a local master clock to GPS to keep it within the desired accuracy limits
Spectracom Solution:
• Network Master Clock: SecureSync PTP Master/Slave
• Linux Time Client: TimeKeeper PTP Client
• Portable GPS Calibrator: GPS-12R Time-Transfer Standard
www.spectracomcorp.com 4 | Timing & Synchronization White Paper

5. In the case of a network without access to GPS, there are 2 choices:
1) Manage the variable packet delay from a network with access to GPS (scenario 1) through the use of PTP-enabled
infrastructure, or a directly connected dedicated low-traffic timing network, so a PTP slave can achieve accurate time from
another network’s GPS PTP grandmaster.
2) Routinely calibrate a local PTP master using a “flying clock” portable calibrator.
As previously mentioned asymmetric packet delay degrades the accuracy of PTP so it offers no advantage over the existing NTP
infrastructure. One approach is to manage the packet delay variation. A local SecureSync can be configured with two PTP modules;
one to act as a PTP slave and one to act as a local PTP master for all the other clients on the network.
If the PDV is unmanageable, then the local SecureSync should be configured as an “atomic clock” with a Rubidium oscillator to
provide the best timekeeping performance in the absence of continuous access to a traceable time source. (Configuring a
SecureSync with a Rubidium oscillator is also a redundancy option in GPS-accessible environments to protect against failure of the
GPS antenna system, RF cabling, or the entire GPS system itself.) The drift rate of a Rubidium oscillator is on the order of a part in 10
th
to the 11 (<10 microseconds per day) and is several orders of magnitude better than crystal oscillators even when combined with
techniques to reduce the effect of temperature variation.
While a Rubidium clock solves the “relative seconds” problem, traceability, or accuracy, to absolute time is required so events on the
local network can be correlated to other networks for managing latencies, etc. Spectracom offers the model GPS-12R battery-
powered GPS time-transfer standard to be used as a “flying clock”. Simply synchronize the unit in an area that can receive GPS
signals while connected to AC mains. Then transport it to the SecureSync unit under its internal battery power (2-3 hours, but a
+12VDC option extends transportation time by using a vehicle battery). Then connect it to the SecureSync and reconnect mains
power. The SecureSync’s internal Rubidium oscillator will use the GPS-12R 1PPS signal as its primary reference. Once it is
synchronized, disconnect the GPS-12R, and the SecureSync will resume in an accurate “holdover” condition until the next calibration
cycle.
Sidebar: The following table below gives you the typical performance you will be able to achieve synchronizing remote locations
held over with Rb oscillators.
Table 2: Timing Performance between remote sites
Between two sites both One site running on Between two sites both
running GPS-based servers with Rb Oscillator running Rb servers with
Timeframe servers constantly holdover Rb holdover
3 months 50 ns + PTP time transfer < 1 ms < 2 ms
6 months 50 ns + PTP time transfer < 2 ms < 4 ms
12 months 50 ns + PTP time transfer < 4 ms < 8 ms
Perpetual 50 ns + PTP time transfer <10 us/day < 20 us/day
Where:
ms = milliseconds
us = microseconds
ns = nanoseconds
Scenario III: Optimizing Time Transfer to a Server Rack
Assumptions:
• GPS is available in the trading center
• A low traffic, highly symmetric, dedicated timing network can be established between the GPS master clock and the server
rack or the network is PTP enabled
• All servers are running Red Hat Enterprise Linux
www.spectracomcorp.com 5 | Timing & Synchronization White Paper

6. Spectracom Solution:
• Network Master Clock: SecureSync® PTP Grandmaster
• Server Rack Master Clock: SecureSync PTP Master/Slave
• Linux Time Client: TimeKeeper PTP Client
Similar to scenario II, a SecureSync GPS clock PTP grandmaster can accurately synchronize another SecureSync through a low PDV
network. This allows for physical separation of the GPS grandmaster and the server rack from a GPS accessible location to the “top-
of-rack”. Network infrastructure can be used to transfer time via PTP. The top-of-rack SecureSync acts as a slave to the GPS
grandmaster and a master for all servers in the rack. The rack becomes a “mini timing network” to eliminate the PDV in the time
transfer between PTP master and client/slaves.
Scenario IV: Improving NTP time synchronization within a local trading center
Assumptions:
• NTP is preferred over PTP
• There is a desire to improve the accuracy of time transfer
• All servers are running Red Hat Enterprise Linux
Spectracom Solution:
• Network Master Clock: Existing GPS NTP Server
• Linux NTP Time Server: TimeKeeper NTP Server Software
• Linux NTP Time Client: TimeKeeper NTP Time Client
NTP was designed for wide time distribution across many devices and networks in a cascading hierarchy. Accuracy of NTP time
transfer can be greatly improved by optimizing the algorithm for point-to-point synchronization. A version of TimeKeeper does just
that. Install this software application in any Linux server. Synchronization to a traceable time source is achieved from a timing signal
through hardware such as a 1PPS signal from existing GPS timing hardware. The TimeKeeper NTP client synchronizes to the NTP
server in a similar way as the PTP client. TimeKeeper is offered in a combined NTP/PTP client, so a new PTP network can be overlaid
on an NTP network. The same principals for managing packet delay variation apply to the NTP configuration as well as the PTP
configuration.
www.spectracomcorp.com 6 | Timing & Synchronization White Paper

7. Conclusion
Financial institutions, hedge funds and market-makers need to address a variety of challenges with accurate, traceable time for their
critical networks and network elements. High frequency trading requires real-time decisions based on microsecond time accuracy in
order to remain competitive. This level of accuracy is available today through a combination of a GPS timing, new network
synchronization protocol (PTP), PTP masters and slaves, time-transfer Linux software, time calibration for GPS-denied environments
and/or carefully managed networks, and measurement solutions.
USA | 1565 Jefferson Road, Suite 460 | Rochester, NY 14623 | +1.585.321.5800 | sales@spectracomcorp.com
FRANCE | 3 Avenue du Canada | 91974 Les Ulis, Cedex | +33 (0)1 64 53 39 80 | sales@spectracom.fr
UK | 6A Beechwood | Chineham Park | Basingstoke, Hants, RG24 8WA | +44 (0)1256 303630 | info@spectracom.co.uk
June 8, 2011 - WP06-101 (A)
www.spectracomcorp.com 7 | Timing & Synchronization White Paper