Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Extend and Build on	
Kubernetes
Dr.	Stefan	Schimanski
sttts@redhat.com
@the1stein
Kubernetes Meetup Frankfurt,	Mar	10	2017
Nune Isabekyan,	https://x-team.com/blog/introduction-kubernetes-architecture/
Disclaimer:
Kubernetes happens to be able to launch pods.	It‘s even quite good at	it.
We will	not	launch pods today.
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
$	kube-apiserver
--secure-port 0	
--etcd-servers	http://127.0.0.1:2379
--service-cluster-ip-range	10.0.0.0/16
--storage-ba...
$	http	GET http://127.0.0.1:8080/api/v1/namespaces/default
{
"apiVersion":	"v1",				
"kind": "Namespace",
"metadata": {
"a...
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$(date...
/apis/batch/v2alpha1/jobs
Apigroup Version			Resource
HTTP	paths:
{
“apiVersion“:	“v2alpha1“,
“kind“:	“Job“,
“metadata“:	{...
{				
"apiVersion":	"v1",				
"kind":	"Status",				
"metadata":	{},				
"code":	409,			
"message":	“...",				
"status":	"F...
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“m...
/apis/extensions/v1alpha1/jobs
/apis/batch/v2alpha1/jobs
/apis/batch/v1/jobs {
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“m...
/apis/extensions/v1alpha1/jobs/nightly
/apis/batch/v2alpha1/jobs/nightly
/apis/batch/v1/jobs/nightly
{
“apiVersion“:	“v1al...
{
“apiVersion“:	“v1alpha1“,
“kind“:	“Job“,
“metadata“:	{
“name“:	“nightly“
},
“spec“:	{	 ...	}
}
{
“apiVersion“:	“v2alpha1...
I	want my own kinds
and store them in	the apiserver and use kubectl.
Discovery
how kubectl knows which kinds/resources exist
$	http	127.0.0.1:8080/apis/	
{
"groups":	[{
"name": "batch",
"preferredVersion": {"groupVersion": "batch/v1", "version": "...
Third	Party	Resources
Third	Party	Resources
*	they are marked as beta in	the API,	but	became beta before we even had alpha.
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	databases.example.com
description:	"A	specificat...
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	databases.example.com
description:	"A	specificat...
apiVersion:	extensions/v1beta1
kind:	ThirdPartyResource
metadata:		
name:	database.example.com
description:	"A	specificati...
$ kubectl get databases –w --no-headers
wordpress <none>				{"apiVersion":"example.com/v1","kind":"Databases",...
wordpres...
https://github.com/kubernetes/features/issues/95
ThirdPartyResources are limited
• no version conversion
• no defaulting
• no validation
• no subresources (scale,	status)
...
Controllers
where is the business logic?
$	while true;	do
http	GET	http://127.0.0.1:8080/api/v1/namespaces/default |	
jq ".metadata.annotations["meetup"]	=	"$(date...
$	kubectl get namespaces -w --no-headers	|	
while read NS	STATUS	TIME	;	do
#	do	whatever you like	here,	e.g.	change the na...
API	Aggregation
Why
• ThirdPartyResources are limited
• no version conversion
• no defaulting
• no validation
• no subresources (scale,	st...
Alpha	in	v1.6:	k8s.io/apiserver
• generic apiserver library in	Go
• today used inside
• kube-apiserver
• federation apiser...
Nune Isabekyan,	https://x-team.com/blog/introduction-kubernetes-architecture/
kube-apiserver service catalog PaaS
kube-aggregator
kube-apiserver service catalog PaaS
kube-aggregator
API API API
API
Pods Jobs ... announcement ... build test project
kube-apiserver kube-apiserver kube-apiserver
federation
apiserver
API API API
API
Pods replicasets Pods replicasets Pods r...
kube-apiserver service catalog PaaS
kube-aggregator
discoverydiscovery
kube-apiserver service catalog PaaS
kube-aggregator
GET
GET	/apis/servicecatalog/subscription/database-prod-wordpress
kube-apiserver service catalog PaaS
kube-aggregator
GET	/apis/servicecatalog/subscription/database-prod-wordpress
GET
RBAC...
kube-apiserver service catalog PaaS
kube-aggregator
GET
Vision:	
$ helm install service-catalog
$ kubectl create service-a...
Status
• will	be part of Kubernetes 1.6	as an	alpha
• https://github.com/kubernetes/sample-apiserver
• potentially kube-ag...
Links
• https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPA
Wj8vAgNFrdTZeCincmI/
Two Ways to Extend the K8s	API -...
https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md
Backup
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
Restful
http	API
/
/version
/api
/api/v1/pods
/api/v1/pods/status
/apis
/apis/batch
/apis/batch/v2alpha1
/apis/batch/v2alp...
Restful
http	API
mux
k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST
AddSupportedResourcesWebService – /apis/ba...
Restful
http	API
mux
pkg/apis/batch
type Jobs	struct
pkg/apis/batch/v2alp
ha1
type Jobs	struct
api.Sche
me api.Scheme.Conv...
type Scheme struct
• AddKnownTypes(gv, obj Object)
• Default(src Object)
• Copy(src Object) Object
• Convert(in, out inter...
Extend and build on Kubernetes
Extend and build on Kubernetes
Próxima SlideShare
Cargando en…5
×

Extend and build on Kubernetes

4.110 visualizaciones

Publicado el

API basics, kinds, resource, apigroups, etcd storage, watches, discovery, ThirdPartyResources, custom apiservers, API aggregation

Publicado en: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Extend and build on Kubernetes

  1. 1. Extend and Build on Kubernetes Dr. Stefan Schimanski sttts@redhat.com @the1stein Kubernetes Meetup Frankfurt, Mar 10 2017
  2. 2. Nune Isabekyan, https://x-team.com/blog/introduction-kubernetes-architecture/
  3. 3. Disclaimer: Kubernetes happens to be able to launch pods. It‘s even quite good at it. We will not launch pods today.
  4. 4. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs /apis/batch/v1 /apis/batch/v1/jobs kube-apiserver kubeletproxy Node: $ kubectl create -f foo.yaml User: scheduler controller manager apiserver ingress controller Master: cloud native apps Pods:
  5. 5. $ kube-apiserver --secure-port 0 --etcd-servers http://127.0.0.1:2379 --service-cluster-ip-range 10.0.0.0/16 --storage-backend etcd2 --storage-media-type application/json $ etcd $ kubectl config set-cluster local --server=http://127.0.0.1:8080 $ kubectl config set-context local --cluster=local $ kubectl config use-context local $ kubectl get namespaces –v=7 $ kubectl get namespace default -o json $ kubectl annotate namespace default meetup=hello $ curl http://127.0.0.1:8080/api/v1/namespaces/default $ etcdctl get / --recursive $ etcdctl get /registry/namespaces/default $ etcdctl -o extended get /registry/namespaces/default
  6. 6. $ http GET http://127.0.0.1:8080/api/v1/namespaces/default { "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "meetup": “hallo" }, "creationTimestamp": "2017-03-10T07:51:39Z", "name": "default", "resourceVersion": “73", }, "spec": { "finalizers": ["kubernetes“] }, "status": { "phase": "Active“ } } $ http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http PUT http://127.0.0.1:8080/api/v1/namespaces/default HTTP/1.1 200 OK Content-Length: 341 Content-Type: application/json Date: Fri, 10 Mar 2017 08:28:01 GMT
  7. 7. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done HTTP/1.1 409 Conflict Content-Length: 310 Content-Type: application/json Date: Fri, 10 Mar 2017 08:27:58 GMT { "apiVersion": "v1", "code": 409, "details": { "kind": "namespaces", "name": "default“ }, "kind": "Status", "message": "Operation cannot be fulfilled on namespaces "default": the object has been modified; please apply your changes to the latest version and try again", "metadata": {}, "reason": "Conflict", "status": "Failure“
  8. 8. /apis/batch/v2alpha1/jobs Apigroup Version Resource HTTP paths: { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  9. 9. { "apiVersion": "v1", "kind": "Status", "metadata": {}, "code": 409, "message": “...", "status": "Failure“ } /apis/batch/v2alpha1/jobs { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } Resource vs. Kind http path vs. logical object
  10. 10. /apis/extensions/v1alpha1/jobs /apis/batch/v2alpha1/jobs /apis/batch/v1/jobs { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  11. 11. /apis/extensions/v1alpha1/jobs /apis/batch/v2alpha1/jobs /apis/batch/v1/jobs { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } }
  12. 12. /apis/extensions/v1alpha1/jobs/nightly /apis/batch/v2alpha1/jobs/nightly /apis/batch/v1/jobs/nightly { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1beta1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } * I omitted the namespace in /apis/batch/v1/jobs/namespaces/default/nightly
  13. 13. { “apiVersion“: “v1alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v2alpha1“, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } { “apiVersion“: “v1 “, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } etcd { “apiVersion“: “v1 “, “kind“: “Job“, “metadata“: { “name“: “nightly“ }, “spec“: { ... } } Protobuf or JSON storage version encoding
  14. 14. I want my own kinds and store them in the apiserver and use kubectl.
  15. 15. Discovery how kubectl knows which kinds/resources exist
  16. 16. $ http 127.0.0.1:8080/apis/ { "groups": [{ "name": "batch", "preferredVersion": {"groupVersion": "batch/v1", "version": "v1“}, "versions": [{"groupVersion": "batch/v1", "version": "v1"}] }, ...] } $ http 127.0.0.1:8080/apis/batch/v1 { "apiVersion": "v1", "groupVersion": "batch/v1", "kind": "APIResourceList", "resources": [{ "kind": "Job", "name": "jobs", "namespaced": true, "verbs": ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch“ ] }, ...] } resource name ⇒ /apis/batch/v1/jobs
  17. 17. Third Party Resources
  18. 18. Third Party Resources * they are marked as beta in the API, but became beta before we even had alpha.
  19. 19. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: databases.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml
  20. 20. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: databases.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml $ kubectl create –f wordpress-databases.yaml apiVersion: example.com/v1 kind: Databases metadata: name: wordpress spec: user: wp password: secret encoding: unicode
  21. 21. apiVersion: extensions/v1beta1 kind: ThirdPartyResource metadata: name: database.example.com description: "A specification of a SQL database.“ versions: - name: v1 $ kubectl create –f databases-tpr.yaml $ kubectl create –f wordpress-database.yaml apiVersion: example.com/v1 kind: Database metadata: name: wordpress spec: user: wp password: secret encoding: unicode
  22. 22. $ kubectl get databases –w --no-headers wordpress <none> {"apiVersion":"example.com/v1","kind":"Databases",... wordpress <none> {"apiVersion":"example.com/v1","kind":"Databases",...
  23. 23. https://github.com/kubernetes/features/issues/95
  24. 24. ThirdPartyResources are limited • no version conversion • no defaulting • no validation • no subresources (scale, status) • no admission • alpha ⇒ API might change • but: demand is high, expect improvements in 1.7+ • Today‘s users of TPRs: https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c
  25. 25. Controllers where is the business logic?
  26. 26. $ while true; do http GET http://127.0.0.1:8080/api/v1/namespaces/default | jq ".metadata.annotations["meetup"] = "$(date)"" | http --check-status PUT http://127.0.0.1:8080/api/v1/namespaces/default || break done ⟲
  27. 27. $ kubectl get namespaces -w --no-headers | while read NS STATUS TIME ; do # do whatever you like here, e.g. change the namespace echo "$NS changed“ done ⟲ $ curl -f 'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘ {"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ... {"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...
  28. 28. API Aggregation
  29. 29. Why • ThirdPartyResources are limited • no version conversion • no defaulting • no validation • no subresources (scale, status) • no admission • Some things need full power of Go • Service catalog • OpenShift PaaS • other powerful APIs ≫ Goal: allow powerful extensions without modifying Kubernetes itself
  30. 30. Alpha in v1.6: k8s.io/apiserver • generic apiserver library in Go • today used inside • kube-apiserver • federation apiserver • service catalog • allows creation of custom apiservers in a couple hundred lines of code • each custom apiserver is its own process, communicating via HTTPS • delegates authentication/authorization to kube-apiserver • uses etcd storage (possibly shared with kube)
  31. 31. Nune Isabekyan, https://x-team.com/blog/introduction-kubernetes-architecture/
  32. 32. kube-apiserver service catalog PaaS kube-aggregator
  33. 33. kube-apiserver service catalog PaaS kube-aggregator API API API API Pods Jobs ... announcement ... build test project
  34. 34. kube-apiserver kube-apiserver kube-apiserver federation apiserver API API API API Pods replicasets Pods replicasets Pods replicasets deployment service Europe US Asia Not this: federation controllers⟲ federated resources: availability zones + regions:
  35. 35. kube-apiserver service catalog PaaS kube-aggregator discoverydiscovery
  36. 36. kube-apiserver service catalog PaaS kube-aggregator GET GET /apis/servicecatalog/subscription/database-prod-wordpress
  37. 37. kube-apiserver service catalog PaaS kube-aggregator GET /apis/servicecatalog/subscription/database-prod-wordpress GET RBAC Namespace
  38. 38. kube-apiserver service catalog PaaS kube-aggregator GET Vision: $ helm install service-catalog $ kubectl create service-announcement .... GET /apis/servicecatalog/subscription/database-prod-wordpress
  39. 39. Status • will be part of Kubernetes 1.6 as an alpha • https://github.com/kubernetes/sample-apiserver • potentially kube-aggregator integrated into kube-apiserver in 1.7
  40. 40. Links • https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPA Wj8vAgNFrdTZeCincmI/ Two Ways to Extend the K8s API - Add resources to a Kubernetes API with TPR or AA • https://github.com/kubernetes/community/blob/master/contributor s/design-proposals/aggregated-api-servers.md • https://gist.github.com/philips/a97a143546c87b86b870a82a753db1 4c - Kubernetes Third-Party Resource Users
  41. 41. https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md
  42. 42. Backup
  43. 43. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjo /apis/batch/v1beta1 /apis/batch/v1beta1/jobs /apis/batch/v2alpha1/jobs Group Version ResourceHTTP paths: In Go: gvk := schema.GroupVersionKind{Group: “batch“, Version: “v2alpha1“, Kind: “Job“} obj := api.Scheme.New(gvk) codec := api.Codecs.LegacyCodec(gvk.GroupVersion()) codec.Decode(reqBody, gvk, obj) type Job struct { metav1.TypeMeta metav1.ObjectMeta Spec JobSpec Status JobStatus } pkg/apis/batch/v2alpha1/types.go type TypeMeta struct { Kind string APIVersion string } type ObjectMeta struct { Name string ... }
  44. 44. Restful http API / /version /api /api/v1/pods /api/v1/pods/status /apis /apis/batch /apis/batch/v2alpha1 /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjo /apis/batch/v1beta1 /apis/batch/v1beta1/jobs MaxInFlightLimit TimeoutForNonLongRunningRequests Panic Recovery CORS Authentication Audit Impersonation Authorization k8s.io/apiserver/pkg/server. DefaultBuildHandlerChain „Filters“ k8s.io/apiserver/pkg/server/routes/index.go – / k8s.io/apiserver/pkg/server/routes/version.go – /version k8s.io/apiserver/pkg/server/routes/swagger.go – /swaggerapi k8s.io/apiserver/pkg/server/routes/openapi.go – /swagger.json „Routes“ mux k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST AddSupportedResourcesWebService – /apis/batch/v2alpha1 k8s.io/apiserver/pkg/endpoints.APIInstaller.Install /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs ... WithRequestInfo ctx.RequestInfo
  45. 45. Restful http API mux k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST AddSupportedResourcesWebService – /apis/batch/v2alpha1 k8s.io/apiserver/pkg/endpoints.APIInstaller.Install /apis/batch/v2alpha1/jobs /apis/batch/v2alpha1/cronjobs ... pkg/apis/batch type Jobs struct pkg/apis/batch/v2alpha1 type Jobs struct api.Scheme k8s.io/apiserver pkg/api api.Scheme.Convert(&internalJob, &v2alohaJob) /apis/batch/v2alpha1/jobs GET PUT POST DELETE ... /status /scale /proxy ... subresources
  46. 46. Restful http API mux pkg/apis/batch type Jobs struct pkg/apis/batch/v2alp ha1 type Jobs struct api.Sche me api.Scheme.Convert(&job, &v1job) POST /apis/batch/v2alpha1/jobs k8s.io/apiserver pkg/endpoints/handlers.CreateNamedResource binary JSON payload Go struct v2alpha1.Job HTTP Request Go struct internal.Job Store k8s.io/apiserver pkg/registry/generic Storage k8s.io/apiserver pkg/storage/etcd3 ProtoBuf Job Go struct v2alpha1.Job etcd
  47. 47. type Scheme struct • AddKnownTypes(gv, obj Object) • Default(src Object) • Copy(src Object) Object • Convert(in, out interface{}) • New(gvk) Object ApiGroup pkg/apis/batch pkg/apis/batch/v1 pkg/apis/batch/v2alpha1 pkg/apis/batch/register.go pgk/apis/batch/install Group Version Kind Resource type Object interface • GetObjectKind() string client-go/pkg/api.Scheme client-go/pkg/api.Codecs Discovery type APIGroupList struct type APIVersions struct type APIResourceList struct GroupVersionKind „gvk“ GroupVersionResource Unversioned types Unstructured List Registry / Storage type Storage interface type Lister interface type Updater interface type Getter interface type Deleter interface .... deepcopy-gen conversion-gen defaulting-gen Code Generation type OwnerReference struct type ObjectReference struct type TypeMeta struct type ObjectMeta struct Meta api.Scheme api.Codecs api.Registry api.GroupFactoryRegistry Globals

×