半教師あり学習

Introduction to Semi-
Supervised Learning

Thursday, January 21, 2010

Outline
•
•
• Self Traininga
• Multiview Algorithms
• Generative Models
• S3VMs(TSVMs)
• Graph-Based Algorithms
•
•

?


•
•
• costly...
•
•
• :


•

• :

? => SSL


•
•

http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

SSL

• Self Training
• Multiview Algorithms
• Generative Models
• S3VMs(TSVMs)
• Graph-Based Algorithms


Self Training


Self Training

• :

• : p(y=1|x) 0.99
1
!

•


Self Training

•
f

• f

•( )

•

Self Training
•
• !

•
•
•
•
•

Multiview Algorithms


Co-training

• Self Training

=>
• ?
•


: Web


•
• )
(BOF

•
• Co-training idea

•
•


Co-training

•
•

•


Co-training


Co-training
•
•
• Self Training
•
•

•

Co-training
• Co-EM
• EM

• Fake feature split
•
• Multiview
•
• =>

Generative Models


Generative Models

•

•


Generative Models
•

•

•


Generative Models

•


Generative Models

• :

• ( )

•


Generative Models
•
•
• EM

•
•
• EM

•
•
• Baum-Welch algorithm

•
•
• : PRML
=> EM MLE


• (
)

• y 2

• EM
MLE


E-step and M-step


Generative Models
•
•
•
•
•
• EM ( )

•
=> page


Generative Models



•

•
λ(λ < 1)


S3VMs(TSVMs)


TSVMs
•
( )



(RKHS)

• SVM
•
=>
(RKHS:reproducing kernel Hilbert space)
•


k

k

f


•
•
• (
)
•


•
(F )

• x
F (F
!)


•f F

•
• f f( x
) => f
=> F =>
!!


SVM

RKHS !

hw RKHS


hinge loss

hinge loss !!



•
• ?

• f(x)>1 or f(x)<-1

•

!
=>


• Joachims(1999)
• branch-
and-bound search( )

• 10000


S3VM

•

•
•


CCCP
(Concave-Convex procedure)
• Yuille (2003)

•
• )
(

• Difference of Convex (DC )

• Update


CCCP
(Concave-Convex procedure)
• update
•
• (update )

• !!


TSVMs + CCCP
• CCCP TSVMs

•
• Iteration 2

http://www.stat.umn.edu/~xshen/paper/tsvm.pdf


L. Wang (2007)
• SVM
TSMVs CCCP
TSMVs

http://www.stat.umn.edu/~xshen/paper/tsvm.pdf

Graph-Based
Algorithms


:
•
•


• !!
•
• etc


• sparseness



https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf

Graph-based SSL
• : edge node

• :

• (instance )

• kNN : k 1
0 => sparse

• :

• dense


Graph-based SSL

• mincut
• harmonic
• manifold regularization


•
• 0 1
ﬁx

•


• =>
• sink
source

• :

http://john.blitzer.com/tutorial/ssl_tutorial.pdf

source sink

: source ∞ : sink -∞


harmonic => Laplacian
• 0-1 y

• Zhu (2003a)

Goldberg (2006)

•


harmonic => Laplacian
(SVM )

(
L
a U b f ...
) → !!


http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf

y_i
hat{y}_i


Laplacian

•f
0
! =>


Goldberg (2006)

• SVMR Metric labeling

• SSL

•



Manifold regularization
• Harmonic
• ...
•
• RKHS
• :)


Manifold regularization
•
•
• SVM

• !


:
•

• 2
• ( )RKHS

•

•
f

• f


•

•

•


•

• Iterated Laplacians
• Heat semigroup
•
• Squared norm of the Hessian
• Belkin (2004)
http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf


Laplacian Regularized
Least Squres(LapRLS)
•
+
0
Regularized Least Squres

• closed


Laplacian Support
Vector Machines
• SVM

•

0 SVM


?
• (Harmonic
)
•
•


mixture model, EM

TSMVs

Co-training
i.i.d
(
Graph-based
)

• no pain, no gain
• no model assumption, no gain
• wrong model assumption, no gain, a lot of
pain


( | )
•
• Co-boosting

• bootstrap

• Directed graphs

• Information Regularization

• Structural Learning

• Large Data

•
• PAC Statistical Learning Theory

(Tutorial etc)
•
• NAACL 2006 Tutorial: Inductive Semi-supervised Learning
with Applicability to NLP, A. Sarkar and G. Haffari.
• ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu.
• Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi-
Supervised learning. http://ssl-acl08.wikidot.com/.
•
• X. Zhu. Semi-supervised learning literature survey. Technical
report, Computer Sciences, University of Wisconsin-
Madison, 2007.
• Zhu, X. (2005). Semi-supervised learning with graphs.
Doctoral dissertation, Carnegie Mellon University. CMU-
LTI-05-192.

(Generative model)
• Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text
classification from labeled and unlabeled documents using EM.
Machine Learning, 39, 103–134.

• Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised
classification of text documents. Proceedings of the Nineteenth
International Conference on Machine Learning (ICML).
• Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled
examples using weighted logistic regression. Proceedings of the
Twentieth International Conference on Machine Learning (ICML).

• Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from
positive and unlabeled examples. The 9th International Conference
on Information Processing and Management of Uncertainty in
Knowledge-Based Systems(IPMU).


(TSVMs)
• Joachims, T. (1999). Transductive inference for text classiﬁcation
using support vector machines. Proc. 16th International Conf.
on Machine Learning (pp. 200– 209). Morgan Kaufmann, San
Francisco, CA.

• Yuille, A.L., Rangara jan, A. The concave-convex procedure.
Neural Computation 15(4) (2003) 915–936.

• L. Wang, X. Shen, and W. Pan. On transductive support vector
machines. In J. Verducci, X. Shen, and J. Lafferty, editors,
Prediction and Discovery. American Mathematical Society, 2007.

• R. Collobert, et al. (2006). Large Scale Transductive SVMs.
Journal of Machine Learning Research 7:1687-1712.


(Graph-based)
• Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph
mincuts. Proc. 18th International Conf. on Machine Learning.

• Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using
Gaussian fields and harmonic functions. The 20th International Conference on
Machine Learning (ICML).

• Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 22, 888–905.

• Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using
subjectivity summarization based on minimum cuts. Proceedings of the Association
for Computational Linguistics (pp. 271–278).

• Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph-
based semi-supervised learning for sentiment categorization. HLT-NAACL 2006
Workshop on Textgraphs: Graph-based Algorithms for Natural Language
Processing. New York, NY.

• Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric
framework for learning from examples (Technical Report TR-2004-06). University
of Chicago.
• M. Belkin & P. Niyogi (2002). Ùsing Manifold Structure for Partially Labelled
Classification'. In NIPS, pp. 929+.

( )
• Seeger, M. (2001). Learning with labeled and unlabeled
data (Technical Report). University of Edinburgh.

• François Denis, Bat M, Universit'e De Lille I. PAC
Learning from Positive Statistical Queries. Proc. 9th
International Conference on Algorithmic Learning
Theory - ALT '98

•
•


半教師あり学習

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Más de syou6162

Más de syou6162 (20)

半教師あり学習