1. c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233
journal homepage: www.intl.elsevierhealth.com/journals/cmpb
CYCLOPS: A mobile robotic platform for testing and
validating image processing and autonomous navigation
algorithms in support of artificial vision prostheses
Wolfgang Fink ∗ , Mark A. Tarbell
Visual and Autonomous Exploration Systems Research Laboratory, Division of Physics, Mathematics, and Astronomy, California Institute
of Technology, Pasadena, CA 91125, USA
a r t i c l e i n f o a b s t r a c t
Article history: While artificial vision prostheses are quickly becoming a reality, actual testing time with
Received 22 February 2009 visual prosthesis carriers is at a premium. Moreover, it is helpful to have a more realis-
Received in revised form tic functional approximation of a blind subject. Instead of a normal subject with a healthy
10 June 2009 retina looking at a low-resolution (pixelated) image on a computer monitor or head-mounted
Accepted 26 June 2009 display, a more realistic approximation is achieved by employing a subject-independent
mobile robotic platform that uses a pixelated view as its sole visual input for navigation
Keywords: purposes. We introduce CYCLOPS: an AWD, remote controllable, mobile robotic platform
Artificial vision prostheses that serves as a testbed for real-time image processing and autonomous navigation systems
Retinal implants for the purpose of enhancing the visual experience afforded by visual prosthesis carriers.
Image processing Complete with wireless Internet connectivity and a fully articulated digital camera with
Autonomous navigation wireless video link, CYCLOPS supports both interactive tele-commanding via joystick, and
Robotics autonomous self-commanding. Due to its onboard computing capabilities and extended
Tele-commanding battery life, CYCLOPS can perform complex and numerically intensive calculations, such as
Self-commanding image processing and autonomous navigation algorithms, in addition to interfacing to addi-
Cloud computing tional sensors. Its Internet connectivity renders CYCLOPS a worldwide accessible testbed for
Worldwide accessibility researchers in the field of artificial vision systems. CYCLOPS enables subject-independent
evaluation and validation of image processing and autonomous navigation systems with
respect to the utility and efficiency of supporting and enhancing visual prostheses, while
potentially reducing to a necessary minimum the need for valuable testing time with actual
visual prosthesis carriers.
© 2009 Elsevier Ireland Ltd. All rights reserved.
1. Introduction “experiencing” the visual perception of a blind person with
a vision implant is emulated by having normal subjects with
While artificial vision prostheses are quickly becoming a real- a healthy retina look at a low-resolution (pixelated) image
ity, actual testing time with visual prosthesis carriers is at a on a computer monitor or head-mounted display. This is a
premium. Moreover, it is helpful to have a realistic functional rather inadequate emulation as a healthy retina with 109
approximation of a blind subject. Commonly, the process of photoreceptors can glean more information from a pixelated
∗
Corresponding author at: Visual and Autonomous Exploration Systems Research Laboratory, California Institute of Technology, 1200 East
California Blvd, Mail Code 103-33, Pasadena, CA 91125, USA. Tel.: +1 626 395 4587.
E-mail address: wfink@autonomy.caltech.edu (W. Fink).
URL: http://autonomy.caltech.edu (W. Fink).
0169-2607/$ – see front matter © 2009 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.cmpb.2009.06.009

2. c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 227
image (e.g., edges, edge transitions, grayscale information, image streams before they enter the visual prosthesis (Fig. 3).
and spatial frequencies) than an impaired retina in which the Moreover, such image processing systems must provide the
photoreceptor layer is dysfunctional due to diseases such as flexibility of repeated application of image manipulation and
retinitis pigmentosa and age-related macular degeneration. processing modules in a user-defined order. Thus, current
A more realistic approximation is achieved by employing a and future artificial vision implant carriers can customize
subject-independent mobile robotic platform that uses a pix- the individual visual perception generated by their visual
elated view as its sole visual input for navigation purposes. prostheses, by actively manipulating parameters of individ-
Such a mobile robotic platform, described in the follow- ual image processing filters or altering the sequence of these
ing, represents not only a constantly available testbed for filters.
real-time image processing systems, but even more so pro-
vides a subject-independent means for testing and validating
the efficiency and utility of real-time image processing and 2. Hardware description
autonomous navigation algorithms for enhanced visual per-
ception and independent mobility for the blind and visually For the purpose of creating a subject-independent mobile
impaired using artificial vision prostheses. testbed for image processing and autonomous navigation
The current state-of-the-art and near future artificial algorithms for artificial vision prostheses, we have cre-
vision implants, such as epi-retinal and sub-retinal implants ated CYCLOPS, an All-Wheel Drive (AWD) remote-controllable
[1–9] (Fig. 1), provide only tens of stimulating electrodes, robotic platform testbed with wireless Internet connectivity
thereby allowing only for limited visual perception (pixela- and a fully articulated digital camera with wireless video link
tion). Usually these implants are driven by extraocular [8,9] (Fig. 4) [13]. For the basic robotic hardware we utilized a WiFi-
or intraocular [10] high-resolution digital cameras that ulti- BoT [14]. The WiFiBoT has a 4G Access Cube, which serves as
mately result in orders of magnitude smaller numbers of the central onboard processor, controlling four electric motors.
pixels that are relayed to the respective implant in use. Hence, We custom-built CYCLOPS by equipping it with:
real-time image processing and enhancement will afford a
critical opportunity to improve on the limited vision afforded • Bi-level metal chassis and sensor trays.
by these implants for the benefit of blind subjects. • General-purpose, high-performance mini Unix workstation
Since tens of pixels/electrodes allow only for a very crude (i.e., Mac mini).
approximation of the roughly 10,000 times higher optical reso- • Rechargeable battery for the Unix workstation.
lution of the external camera image feed, the preservation and • Two rechargeable batteries for the wheel motors.
enhancement of contrast differences and transitions, such as • Gimbaled IP camera that is user-controllable.
edges, become very important as opposed to picture details • IEEE 1394 navigation camera with wide-angle field of view.
such as object texture. Image processing systems (Fig. 2), such • Two forward-looking IR proximity sensors.
as the Artificial Vision Simulator (AVS) [11,12], perform real-time • Real-time voice synthesizer interface.
(i.e., 30 fps) image processing and enhancement of camera • Wireless Internet capability.
Fig. 1 – One instantiation of an artificial vision prosthesis. An intraocular retinal prosthesis using an external
microelectronic system to capture and process image data and transmit the information to an implanted microelectronic
system. The implanted system decodes the data and stimulates via an electrode array the retina with a pattern of electrical
impulses to generate a visual perception.

3. 228 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233
Fig. 2 – Schematic diagram of a real-time image processing system for artificial vision prostheses (e.g., AVS [11,12]) that are
driven by extraocular [8,9] or intraocular [10] camera systems.

4. c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 229
Fig. 3 – Typical palette of image processing modules (e.g., employed by AVS [11,12]) that can be applied in real time to a
video camera stream driving an artificial visual prosthesis.
connects to one or more known “Com Servers” (Fig. 5) . The
3. Testbed implementation Com Servers are known, established Internet entities to which
both the mobile platform and the controlling computer sys-
CYCLOPS supports both interactive tele-commanding via joy- tem connect, acting as a go-between and buffer. In this way,
stick, and autonomous self-commanding. Due to its onboard neither end need know the actual IP address of the other, yet
computing capabilities and battery life (i.e., 4.5 h for the an Internet connection is still established between them, with
onboard mini Unix workstation and 2 h for the electric auto-reconnect in case of connection dropouts.
motors), CYCLOPS can perform complex and numerically The cloud computing approach affords a great deal of archi-
intensive calculations, such as: tectural flexibility; many operational modes are supported,
from fully synchronous to fully autonomous. In the case of
• Testing and validation of image processing systems, such as the joystick operation of CYCLOPS, the mobile platform is in
AVS [11,12], to further the experience of visual prostheses synchronous connection to the Com Server. However, this is
users. not strictly required for other modes of operation. Minimally,
• Testing of navigation algorithms and strategies to improve constant connectivity is not required, as a temporary connec-
the degree of unaided mobility. tion would be sufficient to upload from the mobile platform
• Testing of additional sensors (e.g., infrared) to further the video and sensor data resulting from prior command sets,
utility of visual prostheses. and to download to the mobile platform further operational
commands (e.g., navigation commands), which would be exe-
3.1. Cloud computing
cuted autonomously. A fully “offline” autonomous operational
capability, i.e., independent real-time onboard processing, is
To enable testing of real-time image processing modules, indi-
currently in development for CYCLOPS.
vidually or in sequence, and to enable the transmission of
the resulting remote control navigation sequences, the mobile
platform must establish a connection between itself and the 3.2. Interconnectivity
computer hosting the control software. A standard, direct one-
to-one connection could be established, but this is fragile, as An Internet TCP/IP connection is established between the CPU
either system may not be reachable or known at the moment aboard the mobile platform testbed (via its wireless LAN) and
the connection attempt is made. Instead, a “cloud comput- the computer hosting the image processing and the front-
ing” concept is utilized [15,16], wherein the mobile platform end control software via a Com Server. For the purpose of

5. 230 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233
reliability, this connection is instantiated by the creation of
a temporary AF INET stream socket at the transport layer, uti-
lizing a three-way handshake. Once this full-duplex transport
layer is accomplished, the mobile platform is able to trans-
mit video frames and other sensor data in a packetized and
compressed format over the layer. The mobile platform also
transmits its housekeeping data (battery level, hardware sen-
sor data, etc.) and awaits sensor and drive commands from
the front-end software that are triggered by the results of the
real-time image processing of the video frames (e.g., via AVS).
3.3. Video and sensor data processing
The video and sensor data are treated similarly, however, the
video data are first preprocessed into a suitable data format.
This is accomplished by packetizing the video. Each non-
interlaced stream frame of video data is compressed and
inserted into the payload portion of a header packet, tagging
the data as to type, length, timestamp, and sequence. This has
the advantage over time-division multiplexing of allowing for
real-time synchronization to occur on the receiving end with
Fig. 4 – CYCLOPS, an AWD remote-controllable robotic minimal reconstruction processing. The network connection
platform testbed with wireless Internet connectivity and a is thus used as a virtual N-receiver broadcast channel, each
fully articulated (user-controllable) digital camera with channel being a Q-ary data channel, providing the same mech-
wireless video link, as well as an IEEE 1394 navigation anism for video, sensor, and hardware housekeeping data (e.g.,
camera with wide-angle field of view. It is equipped with a battery charge levels).
general-purpose mini Unix workstation. CYCLOPS is
powered by rechargeable batteries. Furthermore, CYCLOPS 3.4. Navigation commanding
supports a sensor bay for additional sensors (e.g., infrared).
Navigation and camera command transmittal to the mobile
platform testbed is accomplished as follows: The platform
Fig. 5 – Principle of “cloud computing” [15,16]. A mobile platform connects to one or more known “Com Servers” on the
Internet in lieu of direct end-to-end connection with the control system. In this way, neither end need know the actual IP
address of the other.

6. c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 231
utilizes two independent pairs of CPU-controlled electric dependent command sets in the pipeline to be invalidated de
motors, each using a 3-gear reduction module, providing a 4:1 facto, thus returning the mobile platform to a “known good”
reduction ratio for the increase of applied torque to the wheels state.
and reduced power consumption [14]. Each pair of wheel
motors is individually addressable via a command packet that 3.5. User interactivity
specifies motor speed, direction, duration, and speed limiter.
Transmittal of the command packet to the mobile platform For the purpose of commanding CYCLOPS interactively (later
is analogous to the preceding, as the existing transport layer autonomously), the front-end software has an integrated
is utilized in a full-duplex mode, albeit strictly sequenced video panel (Fig. 6) for displaying the transmitted video frames
for simplicity of processing and performing the commands from the mobile platform’s on-board camera (Fig. 4); it is also
in order. Commanding the onboard camera follows a similar outfitted with a USB-based joystick device. The user, con-
procedure. trolling the joystick, is in the loop for the purpose of the
It should be pointed out that the current input com- development of automated software and algorithms to con-
mand set (e.g., set of navigation commands) for the mobile trol CYCLOPS. Once developed, such automated software can
platform differs in form and function from the video and be “plugged in” in lieu of the user for automatic control, with
sensor data, which are received. Input commands require manual user override always available. The user’s movements
comparatively few bytes for expression, thus a simpler, more of the joystick are translated into camera orientation and
expedient architecture is utilized. The input command set wheel rotation commands, and are sent to the mobile plat-
for the mobile platform forms a Strictly Ordered Command form. As the mobile platform begins to move, it also sends back
Pipeline (SOCP) set. Such sets form conditional pipeline branch video, sensor, and housekeeping data, which are displayed on
maps, with sequencing precluding the need for individual the front-end. With this feedback information, a user (or auto-
command prioritization. For example, the mobile platform mated control software for self-commanding) is able to control
may be instructed to perform a certain overall movement, e.g., CYCLOPS interactively (or automatically) from anywhere in
move to the other side of the room. This is translated into the world, in near real-time.
a SOCP set resembling a binary tree; it comprises individual CYCLOPS uses only the pixelated camera images to move
robotic movements (turns, motor commands, etc.) to accom- about an environment (e.g., room/corridor with obstacles),
plish the overall goal of moving to the other side of the room. If thus more realistically emulating the visual perception of a
any individual command in the SOCP set cannot be executed, blind subject. It processes and enhances the pixelated imagery
that particular SOCP set is invalidated at that point, causing to result in new motion and navigation commands, such as
Fig. 6 – CYCLOPS Commanding Interface, controlling the AWD remote robotic platform testbed in near real-time. The
interface displays the current status of CYCLOPS including battery charge levels, heading, velocity, obstacle proximity, and
the high-resolution gimbaled camera view.

7. 232 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233
igation systems (Figs. 2 and 3) with respect to the utility
and efficiency of supporting and enhancing visual prosthe-
ses, while potentially reducing to a necessary minimum the
need for valuable testing time with actual visual prosthesis
carriers.
It is difficult to predict exactly what a blind subject with
a camera-driven visual prosthesis may be able to perceive.
Therefore, it is advantageous to offer a wide variety of image
processing modules and the capability and flexibility for
repeated application of these modules in a user-defined order.
AVS [11,12], in particular, comprises numerous efficient image
processing modules, such as pixelation, contrast/brightness
enhancement, grayscale equalization for luminance control
under severe contrast/brightness conditions, grayscale levels
for reduction of data volume transmitted to the visual pros-
thesis, blur algorithms, and edge detection (e.g., [17,18]). With
the development of CYCLOPS it is now possible to empirically
determine, in the absence of a blind subject, which particular
sequences of image processing modules may work best for the
blind subject in real world scenarios (e.g., Fig. 7).
One of the goals is to get CYCLOPS to “behave” simi-
lar to a blind subject (especially motionwise) by developing,
implementing, testing, and fine-tuning/customizing onboard
algorithms for image processing and analysis as well as auto-
navigation. Once a certain degree of similarity in a behavioral
pattern is achieved, such as navigating safely through a cor-
ridor with obstacles or guideline following (e.g., Fig. 7), the
underlying image processing and analysis algorithms as well
as the sequences of image processing modules that enabled
this successful behavior may be used to establish a practical
initial configuration for blind subjects when implemented in
their respective visual prosthesis systems. Furthermore, test-
ing with CYCLOPS may contribute to improving the design of
environments that provide suitable access for the blind (e.g.,
rooms, corridors, entrances) by choosing those that CYCLOPS
performed best in.
Its Internet connectivity renders CYCLOPS a worldwide
accessible testbed for researchers in the field of artificial
vision systems and machine vision. We have provided a
commanding interface that allows the research community
to easily interface their respective image processing and
autonomous navigation software packages to CYCLOPS by
merely using high-level commands, such as “turn right by
Fig. 7 – Navigation camera view of CYCLOPS at different
25 degrees” or “move forward one meter”. Additionally, we
visual resolutions (i.e., degrees of pixelation), mimicking
have provided numerous interfaces for onboard cameras (Eth-
the view afforded by artificial vision prostheses. Each
ernet, IEEE 1394, USB). The direction and orientation of the
column from top to bottom: 64 × 64, 32 × 32, 16 × 16, 8 × 8.
gimbaled camera can be user-controlled, allowing for the
Left column shows navigating a corridor while avoiding an
emulation of head/eye-motion of a blind subject wearing an
obstacle (i.e., a chair). Right column shows the following of
artificial vision prosthesis. The onboard real-time voice syn-
a high-contrast guideline on the floor of a corridor.
thesizer can be used as a means to communicate audio cues
(e.g., “Door 2 meters ahead.”) resulting from autonomous
navigation and obstacle recognition/avoidance systems (e.g.,
navigating a corridor while avoiding obstacles, and guideline [19]).
following (Fig. 7). Researchers can interface their software packages either by
remotely issuing high-level commands over the Internet, or by
integrating and running their software packages locally on the
4. Conclusion onboard Unix workstation, thereby bypassing the Internet for
command transmittal. Regardless, researchers will be able to
CYCLOPS enables subject-independent testing, evaluation, monitor remotely the actions and camera views of CYCLOPS
and validation of image processing and autonomous nav- via its commanding interface (Fig. 6).

8. c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 6 ( 2 0 0 9 ) 226–233 233
CYCLOPS is directly and immediately applicable to any [6] S.C. DeMarco, The architecture, design, and electromagnetic
(artificial) vision-providing system that is based on an imag- and thermal modeling of a retinal prosthesis to benefit the
ing modality (e.g., video cameras, infrared sensors, sound, visually impaired, PhD Thesis, North Carolina State
University, 2001.
ultrasound, microwave, radar, etc.) as the first step in the gen-
[7] P.R. Singh, W. Liu, M. Sivaprakasam, M.S. Humayun, J.D.
eration of visual perception. Weiland, A matched biphasic microstimulator for an
implantable retinal prosthetic device, in: Proceedings of the
IEEE International Symposium on Circuits and Systems, vol.
Conflict of interest statement
4, 2004.
[8] J.D. Weiland, W. Fink, M. Humayun, W. Liu, D.C. Rodger, Y.C.
Fink and Tarbell may have proprietary interest in the technol- Tai, M. Tarbell, Progress towards a high-resolution retinal
ogy presented here as a provisional patent has been filed on prosthesis, Conf. Proc. IEEE Eng. Med. Biol. Soc. 7 (2005)
behalf of the California Institute of Technology. 7373–7375.
[9] J.D. Weiland, W. Fink, M.S. Humayun, W. Liu, W. Li, M.
Sivaprakasam, Y.C. Tai, M.A. Tarbell, System design of a high
Acknowledgment resolution retinal prosthesis, Conf. Proc. IEEE IEDM (2008),
doi:10.1109/IEDM.2008.4796682.
The work described in this publication was carried out at [10] C.-Q. Zhou, X.-Y. Chai, K.-J. Wu, C. Tao, Q. Ren, In vivo
evaluation of implantable micro-camera for visual
the California Institute of Technology under support of the
prosthesis, Invest. Ophthalmol. Vis. Sci. 48 (2007) 668
National Science Foundation Grant EEC-0310723. (E-Abstract).
[11] W. Fink, M. Tarbell, Artificial vision simulator (AVS) for
references enhancing and optimizing visual perception of retinal
implant carriers, Invest. Ophthalmol. Vis. Sci. 46 (2005) 1145
(E-Abstract).
[12] W. Liu, W. Fink, M. Tarbell, M. Sivaprakasam, Image
[1] W. Liu, M.S. Humayun, Retinal prosthesis, in: IEEE
processing and interface for retinal visual prostheses, in:
International Solid-State Circuits Conference Digest of
ISCAS 2005 Conference Proceedings, vol. 3, 2005, pp.
Technical Papers, 2004, pp. 218–219.
2927–2930.
[2] E. Zrenner, K.-D. Miliczek, V.P. Gabel, H.G. Graf, E. Guenther,
[13] M.A. Tarbell, W. Fink, CYCLOPS: A mobile robotic platform
H. Haemmerle, B. Hoefflinger, K. Kohler, W. Nisch, M.
for testing and validating image processing algorithms in
Schubert, A. Stett, S. Weiss, The development of subretinal
support of visual prostheses, Invest. Ophthalmol. Vis. Sci. 50
microphotodiodes for replacement of degenerated
(2009) 4218 (E-Abstract).
photoreceptors, Ophthalmic Res. 29 (1997) 269–328.
[14] Robosoft, http://www.robosoft.fr/eng/.
[3] J.F. Rizzo, J.L. Wyatt, Prospects for a visual prosthesis,
[15] R. Chellappa, Cloud computing—emerging paradigm for
Neuroscientist 3 (1997) 251–262.
computing, INFORMS, 1997.
[4] E. Zrenner, Will retinal implants restore vision? Science 295
[16] B. Hayes, Cloud computing, Commun. ACM 51 (2008).
(2002) 1022–1025.
[17] J.C. Russ, The Image Processing Handbook, CRC Press, 2002.
[5] M.S. Humayun, J. Weiland, G. Fujii, R.J. Greenberg, R.
[18] H.R. Myler, A.R. Weeks, The Pocket Handbook of Image
Williamson, J. Little, B. Mech, V. Cimmarusti, G. van Boemel,
Processing Algorithms in C, Prentice Hall PTR, 1993.
G. Dagnelie, E. de Juan Jr., Visual perception in a blind
[19] W. Fink, M. Tarbell, J. Weiland, M. Humayun, DORA: digital
subject with a chronic microelectronic retinal prosthesis,
object recognition audio-assistant for the visually impaired,
Vision Res. 43 (2003) 2573–2581.
Invest. Ophthalmol. Vis. Sci. 45 (2004) 4201 (E-Abstract).