1. National Aeronautics and Space
Administration
Jet Propulsion Laboratory
California Institute of Technology
Machine Learning and Instrument Autonomy Group (398E)
Contact (Supervisor): Tara Estlin
Presenter: Lukas Mandrake
Tara.a.estlin@jpl.nasa.gov
Lukas.mandrake@jpl.nasa.gov
© 2015 California Institute of Technology
Government sponsorship acknowledged

2. • Machine Learning Definition
• Amenable Data Types
• Curse of Dimensionality
• Common Techniques
• Example Applications
2

3. 3

4. Notional
Graphs
Only
%ThinkingSpent
kB
MB
GB
TB
Available Data
Electrical sensors for data taking
Computers for analytics
ML for building models
Leaves humans to interpret
4

5. Linear regressions
Correlation coeff’s, R2
Higher curve fits
Separate & identify sub-populations
Make & Compare Models
Throw out outliers
Wonder if you picked the best X-axis
Pick 2 dimensions and plot
Potentially interesting
Importantvariable
What you hope for
Scatter Data
What you get What you do
Spectral Data
Fit slopes
Subtract / divide backgrounds
Take ratios of known frequencies
Look for familiar peaks
Make & Compare Models
Hope you know which frequencies
are most informative
Does it work?
YES! It’s how we got here.
Takes a LOT of human time
That’s the currency of merit
Bins (λ,ν)
Counts
5

6. • Replicate what humans do while analyzing data
– Builds and auto-tunes models
– Forces analyst to make metrics clear(er)
• Subjective decisions become principled (or at least
repeatable) and FAST!
• Higher dimensions than humans can visualize
• Replicate over huge datasets
• In the end, extends human analysis.
No one gets replaced, they get augmented!
6

7. 7

8. Hyperspectral imagers: pictures with high-res spectra at each pixel
4000 frequencies x 5e6 pixels x 3000 images
A human can select a pixel to study the spectra, or make an image
of a particular frequency (or ratio). Maybe take a dot product with a
desired spectra.
Most of image(s) remain unexamined! Needle in haystack problem.
Image databases: How to search?
Martian crater expert only wants images with craters, but would take hundreds of
grad students to label them all. Even if you do, each student’s criteria will be
different (fatigue).
This classification problem is ubiquitous in image and video processing
Maybe want to find list of “things that aren’t anything you taught me” to look for
interesting new landforms not expected.
Compare nearly overlapping images at different times and find “what’s changed”
for dynamics study.
Just too large for even a team to tackle.8

9. Metadata-Rich Datasets:
Observations aren’t just numbers: whole vector of associated data
- Estimates of T, P, aerosol estimates, cloudiness, H2O content…
- 10 to 1000 such parameters
Look for correlated trends, sub-category description / separation,
anomaly discovery, and primary correlates to unwanted behavior
Fundamentally >2D relationships will be missed by simple plotting
Object databases: What’s in there?
Martian rock database records dozens of properties of each rock
examined… thousands or millions of them
What rock types group together?
What stands out as unique? Why was it unique?
Given we now know that, what’s the next most unique thing?
9

10. • Sphere vols increase until dim=5 then go to zero as d->inf
• Sphere within cube eventually removes no volume from cube
• A smaller sphere within a larger removes no volume
• Zero prob within 1 stdev in Gaussian (“Losing the Middle”)
…
10

11. Example: Analyze a genetics dataset
• Data: 1e6 samples, each is gene snippet 1e5 base-pair long
• “Find the gene locations that correlate with an observed condition”
• Already processed dataset with only 1e2 base-pairs just fine
Each base-pair location has ~10 samples. Dbig makes data coverage -> 0
Regressions detonate as (ever more likely) correlated genes cause singularities
Even without correlations, distance function diluted. Everybody is very far.
Dilutes meaning of all regressions, nearest neighbor comparisons
Space is too large to be searched. Requires exponential sampling.
Errors! Singular matrices! Meaningless results!
D1
2
+ D2
2
+D3
2
+ D4
2
+ D5
2
+ D6
2
+D7
2
+×××
11

12. • Identify most informative dimensions or Mixtures
“feature selection”
• Requires a search over number of dimensions D
Can take time
• But once informative features recognized,
everything else is faster and easier
• Fundamental to Machine Learning,
usually as a first step
12

13. 13

14. • Operational Scenarios• Data > Transmission
– Autonomous prioritize by “interest”
– “interest” can be specified, calculated in-situ, or anomalous
• Comm Delay > Decision Time
– Autonomous decision making
– Pre-defined or anomalous triggers
– Plan/schedule response and follow-up
• Volume > Analysis Time
– Identify uninteresting / interesting data and sub-populations
– Identify anomalies, test models
• Data collection capability > data storage capability
– Autonomous decision what to retain
14

15. 15

16. Unsupervised
“Data Mining”
Supervised
“Learning”
Other Methods
• Algorithm studies data independently
• User does not “help” algorithm understand
• No user assumptions to corrupt results
• No human expertise either
• Human provides labeled examples to “learn”
• Human selects algorithm / model for generalization
• Algorithm figures out how to generalize labels
• Resulting tuned model reveals structure of data
• Produces useful system for replicating the labeling
• Might involve humans as part of the learning cycle
• Might seek feedback to make new labeled data
• Might use evolution to figure out best ML parameters
• Might maintain multi-goal output space
16

17. 17

18. Finding Hidden Structure In Unlabeled Data
Clustering: “Are there sub-populations in my data?”
Defines n clusters to which all observations are members
Easy to see in 2D, harder in Dbig
Sub-populations can guide analyst to independent analysis
May correspond to physically meaningful populations
Must provide distance metric, algorithm, parameters, data
filtration, parameter n
PCA: “What combination of dims explains my data Var?”
New axes based on linear combination of original dims
Axes ordered in terms of data variance they explain
Works if data variance is all “interesting” (rare)
Dimension reduction: take first n axes until 99% Var explained
Can’t handle dimensional correlations
HMM: “What statistical model produced my data?”
Pertains to time-series or sequential data
Assumes prob model that only depends on last state
Constructs most likely model that would explain dataset
Can reveal hidden relations and driving processes
Principle Component Analysis
Hidden Markov Model
18

19. Finding Hidden Structure In Unlabeled Data
Rules Learning: “What events tend to co-occur?”
Discovers strong, simple rules between dimensions
Useful to figure out hidden relations in large datasets
Can also be helpful to remove correlated features
Gives potential rules for interpretation and investigation
Segmentation: “What regions best describe image?”
Groups pixels/samples into larger regions of similar nature
Expensive, slow analysis may then be done per region
Averaging across region may reduce noise in “super pixel”
Helps image recognition and classification tasks
Focuses analysis on complex areas vs boring stretches
DOGO: “Order my data by its quality / utility”
Specify a metric to max/minimize
Finds features that, via filtration, optimize metric
Constructs sliding filter that monotonically reduces metric
Inverts filter to produce data ordering of most to least trusted
Useful when data isn’t merely “good” or “bad”
Data Ordering through Genetic Optimization
19

20. 20

21. Algorithms Learning from Humans
LDA: “What hyperplane would best separate my labels?”
Like PCA, but works to separate labels, not explain Variance
Returns vector of how useful each dimension was to separate
Vulnerable to correlated features
Surprisingly powerful for simple classification
SVM: “What set of samples best define label separation?”
Same idea of LDA. Make a separating hyperplane
Pays attention only to the most confusing examples
Creates a “basis” set of support vectors, most informative samples
Gives idea of data importance: which samples change answer
Neural Network: “Predict my labels, I don’t care how”
Define layer geometry: # hidden layers, input types, output types
Train on input data and user labels. Defines weights.
Black-box predictor now online
Monte Carlo stimulate inputs to maximize output concept signal
Hard to get insight from network itself
Linear Discriminant Analysis
Support Vector Machine
21

22. Algorithms Learning from Humans
Decision Tree: “Play 20 questions to separate my labels.”
Set # of tree branches allowed
Learns best series of questions that isolate provided labels
Directly interpretable by domain experts… no black box
Extremely fast to evaluate once trained
Naïve Bayes: “What’s probability of belong to any label?”
Assumes non-correlated dimensions (often works despite this)
Needs relatively small number of input labels
Learns distribution of training labels independently
Predicts probability of sample being in all label categories
Can easily have “I don’t know” response added
Nearest Neighbor: “Use comparables to predict label”
User picks how many neighbors to consider
For each sample to predict, scans all input training data
Finds k neighbors by distance metric, then averages labels
Learns no structure, uses no models, just distance metric
Can be slow to evaluate if lots of labeled data22

23. Algorithms Learning from Humans
Random Forest: “Make lots of little trees and take vote.”
Trains hundreds of small trees on label data subsets
In final prediction, let them vote for final output
Overcomes Decision Tree tendency to overfit data
Removes Decision Tree strength of interpretability
Boosting / Ensemble: “Combine algorithms to improve.”
Iteratively train lots of weak / simple methods (any mix will do)
Larger optimization (genetic) twiddles with all their parameters
Larger optimization learns weights to combine answers together
Takes a lot of processing power and input data, but great results
Can learn about data based on which predictors were selected
23

24. 24

25. Algorithms that work with or without labels
or that are interactive and can generate them
Genetic Algorithm: “Iteratively optimize my (gene) model”
Define a “gene” of all parameters you want GA to optimize
Define goal metric(s) GA should try to max/minimize
GA’s handle mixed input, arbitrary goal metrics
Not really learning, but useful in similar situations. Slow.
Active Learning: “What should I have labeled to help?”
Initialize system with supervised or unsupervised method
Have system predict and display to user
User corrects errors and addresses confusing examples
Iterate between prediction and feedback until results look good
Can easily over-fit, so hold-out tests are important here
25

26. Some concepts to save you from harm
Over-Fitting
• Samples is not >> degrees of freedom
• Gets you great results… on your training set
• Can’t generalize! Predictor fails on new data.
• Use cross-validation and/or simpler model
• Train on 20% data, test on 80%? Vice versa? 50/50?
• Depends on data volume and algorithm need
• Data structure also determines: how heterogeneous
Train / Test Split
• Automated way to explore all possible train/test splits
• Second level withholds data from test & train
• Takes lots of data and time
• Actually tests generalization
Cross-Validation
Label Imbalance
• If 5% of your labels are “yes” and 95% are “no”…
• Just guess no all the time, and you’re right 95%!
• This can imbalance some training algorithms
26
Normalization
• Should you normalize all inputs between 0-1?
• Perhaps they should have mean 0, STD=1 instead?
• Not if spectrum where relative intensity matters

27. • 4 metrics: TPOS, TNEG, FPOS, FNEG
• Application Specific
• Make trade-off curve by running algorithm w/ different parameters
• Receiver Operating Characteristic (ROC)
• Just means “How often do you miss” vs. “How often do you hallucinate?”
• Curse says what your options are. You pick what you can tolerate most.
False Positives ->
TruePositives->
27

28. 28

29. • Automatically Identifies and Classifies Rocks
Image Data Classification Random Forest
Dr. David Thompson29

30. March 11, 2011 0500 UTC March 11, 2011 0530 UTC March 11, 2011 0600 UTC
March 11, 2011 0630 UTC March 11, 2011 0700 UTC March 13, 2011 1300 UTC
Before the rupture
Nominal states
Rupture Initiation Propagation
~1.5 hour timescale
propagation of
state changes
Rupture completion
Nominal states
Two days later
Growth of feature
near triple junction
(near Tokyo)
• Looks for Earthquake behavior people miss
Time Series Data Anomaly Detection Hidden Markov Models
Dr. Robert Granat30

31. Full Pancam View
AEGIS Autonomously
Delivers 13F Pancam image
• Notices “interesting” objects while driving/scanning
• Takes higher resolution images for later analysis
• Operational on MER
Real Time Image Data Anomaly Detection Segmentation
Dr. Tara Estlin31

32. • Prioritizes incoming soundings on usefulness for further analysis
• Lets retrieval algorithms initially work only on cleanest data
• Will be operational in OCO-2 DAC to meet Level 1 requirements
• Advises scientists on which data to include in their analysis
Real-Time Soundings Data Prioritization Genetic Algorithm
Dr. Lukas Mandrake32

33. • Automatically recognize, outline, and classify Martian landmarks
• Hi-Rise database = tens of thousands of huge resolution images
• How to search for your field of interest?
• What are statistics on various landforms?
Image Database Anomaly Detection & Classification Boost + SVM
Dr. Kiri Wagstaff33

34. 0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
2.2
Noise
RFI
RFI
Transients
Quadratic
Sum
Robust
ECDF
! (x1
,d)
!(x2
,d)
Decision boundaries for multistation transient detectors
A transient signal Separating signals in the
parameter space
• Recognizes brand new Supernova in a few seconds
Fast, Real-Time Series Data Anomaly Detection Random Forests
Supernova &
Pulsars
Dr. Umaa Rebbapragada34

35. • Machine Learning is for everyone!
• Relatively simple algorithms lying around for use
• Can help researcher understand their data initially
• Can help drill-down into sub-populations
• Can automate monotonous labeling tasks
• Available in
– Python (Scikit-learn, Orange, Weka)
– Matlab (Statistics, Neural Net, Fuzzy Logic Toolboxes)
– Most languages (OpenCV)
Or just jot us an email! We love to collaborate.35