HyperShift API Reference

Spec is the desired behavior of the HostedCluster.

Status is the latest observed status of the HostedCluster.

Spec is the desired behavior of the NodePool.

Status is the latest observed status of the NodePool.

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

AdvertiseAddress is the address that nodes will use to talk to the API server. This is an address associated with the loopback adapter of each node. If not specified, 172.20.0.1 is used.

Port is the port at which the APIServer is exposed inside a node. Other pods using host networking cannot listen on this port. If not specified, 6443 is used.

Subnet is the subnet to use for control plane cloud resources.

Zone is the availability zone where control plane cloud resources are created.

VPC is the VPC to use for control plane cloud resources.

"Private"

Private endpoint access allows only private API server access and private node communication with the control plane.

"Public"

Public endpoint access allows public API server access and public node communication with the control plane.

"PublicAndPrivate"

PublicAndPrivate endpoint access allows public API server access and private node communication with the control plane.

Credentials contains the name of the secret that holds the aws credentials that can be used to make the necessary KMS calls. It should at key AWSCredentialsFileSecretKey contain the aws credentials file that can be used to configure AWS SDKs

ARN is the Amazon Resource Name for the encryption key

Region contains the AWS region

ActiveKey defines the active key used to encrypt new secrets

BackupKey defines the old key during the rotation process so previously created secrets can continue to be decrypted until they are all re-encrypted with the active key.

Auth defines metadata about the management of credentials used to interact with AWS KMS

InstanceType is an ec2 instance type for node instances (e.g. m5.large).

InstanceProfile is the AWS EC2 instance profile, which is a container for an IAM role that the EC2 instance uses.

Subnet is the subnet to use for node instances.

AMI is the image id to use for node instances. If unspecified, the default is chosen based on the NodePool release payload image.

SecurityGroups is an optional set of security groups to associate with node instances.

RootVolume specifies configuration for the root volume of node instances.

ResourceTags is an optional list of additional tags to apply to AWS node instances.

These will be merged with HostedCluster scoped tags, and HostedCluster tags take precedence in case of conflicts.

See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

Region is the AWS region in which the cluster resides. This configures the OCP control plane cloud integrations, and is used by NodePool to resolve the correct boot AMI for a given release.

CloudProviderConfig specifies AWS networking configuration for the control plane.

TODO(dan): should this be named AWSNetworkConfig?

ServiceEndpoints specifies optional custom endpoints which will override the default service endpoint of specific AWS Services.

There must be only one ServiceEndpoint for a given service name.

Roles must contain exactly 4 entries representing the locators for roles supporting the following OCP services:

• openshift-ingress-operator/cloud-credentials
• openshift-image-registry/installer-cloud-credentials
• openshift-cluster-csi-drivers/ebs-cloud-credentials
• cloud-network-config-controller/cloud-credentials

Each role has unique permission requirements whose documentation is TBD.

TODO(dan): revisit this field; it’s really 3 required fields with specific content requirements

KubeCloudControllerCreds is a reference to a secret containing cloud credentials with permissions matching the cloud controller policy. The secret should have exactly one key, credentials, whose value is an AWS credentials file.

TODO(dan): document the “cloud controller policy”

NodePoolManagementCreds is a reference to a secret containing cloud credentials with permissions matching the node pool management policy. The secret should have exactly one key, credentials, whose value is an AWS credentials file.

TODO(dan): document the “node pool management policy”

ControlPlaneOperatorCreds is a reference to a secret containing cloud credentials with permissions matching the control-plane-operator policy. The secret should have exactly one key, credentials, whose value is an AWS credentials file.

TODO(dan): document the “control plane operator policy”

ResourceTags is a list of additional tags to apply to AWS resources created for the cluster. See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html for information on tagging AWS resources. AWS supports a maximum of 50 tags per resource. OpenShift reserves 25 tags for its use, leaving 25 tags available for the user.

EndpointAccess specifies the publishing scope of cluster endpoints. The default is Public.

Value must be one of: "Private", "Public", "PublicAndPrivate"

ID of resource

ARN of resource

Filters is a set of key/value pairs used to identify a resource They are applied according to the rules defined by the AWS API: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

Key is the key of the tag.

Value is the value of the tag.

Some AWS service do not support empty values. Since tags are added to resources in many services, the length of the tag value must meet the requirements of all services.

Name is the name of the AWS service. This must be provided and cannot be empty.

URL is fully qualified URI with scheme https, that overrides the default generated endpoint for a client. This must be provided and cannot be empty.

MinCPUs specifies the minimum number of CPU cores required.

MinMemoryMiB specifies the minimum amount of RAM required, in MiB.

AgentLabelSelector contains labels that must be set on an Agent in order to be selected for a Machine.

AgentNamespace is the namespace where to search for Agents for this cluster

"HighlyAvailable"

HighlyAvailable means components should be resilient to problems across fault boundaries as defined by the component to which the policy is attached. This usually means running critical workloads with 3 replicas and with little or no toleration of disruption of the component.

"SingleReplica"

SingleReplica means components are not expected to be resilient to problems across most fault boundaries associated with high availability. This usually means running critical workloads with just 1 replica and with toleration of full disruption of the component.

MaxNodesTotal is the maximum allowable number of nodes across all NodePools for a HostedCluster. The autoscaler will not grow the cluster beyond this number.

MaxPodGracePeriod is the maximum seconds to wait for graceful pod termination before scaling down a NodePool. The default is 600 seconds.

MaxNodeProvisionTime is the maximum time to wait for node provisioning before considering the provisioning to be unsuccessful, expressed as a Go duration string. The default is 15 minutes.

PodPriorityThreshold enables users to schedule “best-effort” pods, which shouldn’t trigger autoscaler actions, but only run when there are spare resources available. The default is -10.

See the following for more details: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-cluster-autoscaler-work-with-pod-priority-and-preemption

SecretRefs holds references to any secrets referenced by configuration entries. Entries can reference the secrets using local object references.

ConfigMapRefs holds references to any configmaps referenced by configuration entries. Entries can reference the configmaps using local object references.

Items embeds the serialized configuration resources.

ServiceCIDR is…

TODO(dan): document it

PodCIDR is…

TODO(dan): document it

MachineCIDR is…

TODO(dan): document it

NetworkType specifies the SDN provider used for cluster networking.

Value must be one of: "Calico", "OpenShiftSDN"

APIServer contains advanced network settings for the API server that affect how the APIServer is exposed inside a cluster node.

desired is the version that the cluster is reconciling towards. If the cluster is not yet fully initialized desired will be set with the information available, which may be an image or a tag.

history contains a list of the most recent versions applied to the cluster. This value may be empty during cluster startup, and then will be updated when a new update is being applied. The newest update is first in the list and it is ordered by recency. Updates in the history have state Completed if the rollout completed - if an update was failing or halfway applied the state will be Partial. Only a limited amount of update history is preserved.

observedGeneration reports which version of the spec is being synced. If this value is not equal to metadata.generation, then the desired and conditions fields may represent a previous version.

"EndpointAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint has been created in the guest VPC

"EndpointServiceAvailable"

AWSEndpointServiceAvailable indicates whether the AWS Endpoint Service has been created for the specified NLB in the management VPC

"ClusterVersionFailing"

"ClusterVersionSucceeding"

ClusterVersionSucceeding indicates the current status of the desired release version of the HostedCluster as indicated by the Failing condition in the underlying cluster’s ClusterVersion.

"EtcdAvailable"

"Available"

HostedClusterAvailable indicates whether the HostedCluster has a healthy control plane.

"Available"

"IgnitionEndpointAvailable"

IgnitionEndpointAvailable indicates whether the ignition server for the HostedCluster is available to handle ignition requests.

"InfrastructureReady"

"KubeAPIServerAvailable"

"SupportedHostedCluster"

SupportedHostedCluster indicates whether a HostedCluster is supported by the current configuration of the hypershift-operator. e.g. If HostedCluster requests endpointAcess Private but the hypershift-operator is running on a management cluster outside AWS or is not configured with AWS credentials, the HostedCluster is not supported.

"UnmanagedEtcdAvailable"

UnmanagedEtcdAvailable indicates whether a user-managed etcd cluster is healthy.

"ValidConfiguration"

ValidHostedClusterConfiguration indicates (if status is true) that the ClusterConfiguration specified for the HostedCluster is valid.

"ValidHostedControlPlaneConfiguration"

BaseDomain is the base domain of the cluster.

PublicZoneID is the Hosted Zone ID where all the DNS records that are publicly accessible to the internet exist.

PrivateZoneID is the Hosted Zone ID where all the DNS records that are only available internally to the cluster exist.

"Managed"

Managed means HyperShift should provision and operator the etcd cluster automatically.

"Unmanaged"

Unmanaged means HyperShift will not provision or manage the etcd cluster, and the user is responsible for doing so.

ManagementType defines how the etcd cluster is managed.

Value must be one of: "Managed", "Unmanaged"

Managed specifies the behavior of an etcd cluster managed by HyperShift.

Unmanaged specifies configuration which enables the control plane to integrate with an eternally managed etcd cluster.

ClientSecret refers to a secret for client mTLS authentication with the etcd cluster. It may have the following key/value pairs:

etcd-client-ca.crt: Certificate Authority value
etcd-client.crt: Client certificate value
etcd-client.key: Client certificate key value

Name of the filter. Filter names are case-sensitive.

Values includes one or more filter values. Filter values are case-sensitive.

Release specifies the desired OCP release payload for the hosted cluster.

Updating this field will trigger a rollout of the control plane. The behavior of the rollout will be driven by the ControllerAvailabilityPolicy and InfrastructureAvailabilityPolicy.

InfraID is a globally unique identifier for the cluster. This identifier will be used to associate various cloud resources with the HostedCluster and its associated NodePools.

TODO(dan): consider moving this to .platform.aws.infraID

Platform specifies the underlying infrastructure provider for the cluster and is used to configure platform specific behavior.

ControllerAvailabilityPolicy specifies the availability policy applied to critical control plane components. The default value is SingleReplica.

Value must be one of: "HighlyAvailable", "SingleReplica"

InfrastructureAvailabilityPolicy specifies the availability policy applied to infrastructure services which run on cluster nodes. The default value is HighlyAvailable.

Value must be one of: "HighlyAvailable", "SingleReplica"

DNS specifies DNS configuration for the cluster.

Networking specifies network configuration for the cluster.

Autoscaling specifies auto-scaling behavior that applies to all NodePools associated with the control plane.

Etcd specifies configuration for the control plane etcd cluster. The default ManagementType is Managed. Once set, the ManagementType cannot be changed.

Services specifies how individual control plane services are published from the hosting cluster of the control plane.

If a given service is not present in this list, it will be exposed publicly by default.

PullSecret references a pull secret to be injected into the container runtime of all cluster nodes. The secret must have a key named “.dockerconfigjson” whose value is the pull secret JSON.

SSHKey references an SSH key to be injected into all cluster node sshd servers. The secret must have a single key “id_rsa.pub” whose value is the public part of an SSH key.

IssuerURL is an OIDC issuer URL which is used as the issuer in all ServiceAccount tokens generated by the control plane API server. The default value is kubernetes.default.svc, which only works for in-cluster validation.

Configuration specifies configuration for individual OCP components in the cluster, represented as embedded resources that correspond to the openshift configuration API.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has mTLS the kubeconfig will contain client keys. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

This field is currently only supported on the IBMCloud platform.

ImageContentSources specifies image mirrors that can be used by cluster nodes to pull content.

SecretEncryption specifies a Kubernetes secret encryption strategy for the control plane.

FIPS indicates whether this cluster’s nodes will be running in FIPS mode. If set to true, the control plane’s ignition server will be configured to expect that nodes joining the cluster will be FIPS-enabled.

Version is the status of the release version applied to the HostedCluster.

KubeConfig is a reference to the secret containing the default kubeconfig for the cluster.

KubeadminPassword is a reference to the secret that contains the initial kubeadmin user password for the guest cluster.

IgnitionEndpoint is the endpoint injected in the ign config userdata. It exposes the config for instances to become kubernetes nodes.

Conditions represents the latest available observations of a control plane’s current state.

NetworkType specifies the SDN provider used for cluster networking.

Value must be one of: "Calico", "OpenShiftSDN"

APIPort is the port at which the APIServer listens inside a worker

APIAdvertiseAddress is the address at which the APIServer listens inside a worker.

ControllerAvailabilityPolicy specifies whether to run control plane controllers in HA mode Defaults to SingleReplica when not set

Value must be one of: "HighlyAvailable", "SingleReplica"

InfrastructureAvailabilityPolicy specifies whether to run infrastructure services that run on the guest cluster nodes in HA mode Defaults to HighlyAvailable when not set

Value must be one of: "HighlyAvailable", "SingleReplica"

FIPS specifies if the nodes for the cluster will be running in FIPS mode

KubeConfig specifies the name and key for the kubeconfig secret

Services defines metadata about how control plane services are published in the management cluster.

AuditWebhook contains metadata for configuring an audit webhook endpoint for a cluster to process cluster audit events. It references a secret that contains the webhook information for the audit webhook endpoint. It is a secret because if the endpoint has MTLS the kubeconfig will contain client keys. This is currently only supported in IBM Cloud. The kubeconfig needs to be stored in the secret with a secret key name that corresponds to the constant AuditWebhookKubeconfigKey.

Etcd contains metadata about the etcd cluster the hypershift managed Openshift control plane components use to store data.

Configuration embeds resources that correspond to the openshift configuration API: https://docs.openshift.com/container-platform/4.7/rest_api/config_apis/config-apis-index.html

ImageContentSources lists sources/repositories for the release-image content.

SecretEncryption contains metadata about the kubernetes secret encryption strategy being used for the cluster when applicable.

Ready denotes that the HostedControlPlane API Server is ready to receive requests This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L226-L230

Initialized denotes whether or not the control plane has provided a kubeadm-config. Once this condition is marked true, its value is never changed. See the Ready condition for an indication of the current readiness of the cluster’s control plane. This satisfies CAPI contract https://github.com/kubernetes-sigs/cluster-api/blob/cd3a694deac89d5ebeb888307deaa61487207aa0/controllers/cluster_controller_phases.go#L238-L252

ExternalManagedControlPlane indicates to cluster-api that the control plane is managed by an external service. https://github.com/kubernetes-sigs/cluster-api/blob/65e5385bffd71bf4aad3cf34a537f11b217c7fab/controllers/machine_controller.go#L468

ControlPlaneEndpoint contains the endpoint information by which external clients can access the control plane. This is populated after the infrastructure is ready.

Version is the semantic version of the release applied by the hosted control plane operator

ReleaseImage is the release image applied to the hosted control plane.

lastReleaseImageTransitionTime is the time of the last update to the current releaseImage property.

KubeConfig is a reference to the secret containing the default kubeconfig for this control plane.

KubeadminPassword is a reference to the secret containing the initial kubeadmin password for the guest cluster.

Condition contains details for one aspect of the current state of the HostedControlPlane. Current condition types are: “Available”

Type defines the IBM Cloud KMS authentication strategy

Value must be one of: "Managed", "Unmanaged"

Unmanaged defines the auth metadata the customer provides to interact with IBM Cloud KMS

Managed defines metadata around the service to service authentication strategy for the IBM Cloud KMS system (all provider managed).

"Managed"

IBMCloudKMSManagedAuth defines the KMS authentication strategy where the IKS/ROKS platform uses service to service auth to call IBM Cloud KMS APIs (no customer credentials requried)

"Unmanaged"

IBMCloudKMSUnmanagedAuth defines the KMS authentication strategy where a customer supplies IBM Cloud authentication to interact with IBM Cloud KMS APIs

CRKID is the customer rook key id

InstanceID is the id for the key protect instance

CorrelationID is an identifier used to track all api call usage from hypershift

URL is the url to call key protect apis over

KeyVersion is a unique number associated with the key. The number increments whenever a new key is enabled for data encryption.

Region is the IBM Cloud region

Auth defines metadata for how authentication is done with IBM Cloud KMS

KeyList defines the list of keys used for data encryption

Credentials should reference a secret with a key field of IBMCloudIAMAPIKeySecretKey that contains a apikey to call IBM Cloud KMS APIs

ProviderType is a specific supported infrastructure provider within IBM Cloud.

Source is the repository that users refer to, e.g. in image pull specifications.

Mirrors are one or more repositories that may also contain the same images.

"AWS"

"IBMCloud"

Provider defines the KMS provider

Value must be one of: "AWS", "IBMCloud"

IBMCloud defines metadata for the IBM Cloud KMS encryption strategy

AWS defines metadata about the configuration of the AWS KMS Secret Encryption provider

NodeTemplate Spec contains the VirtualMachineInstance specification.

Storage specifies how etcd data is persisted.

Type is the kind of persistent storage implementation to use for etcd.

Value must be one of: "PersistentVolume"

PersistentVolume is the configuration for PersistentVolume etcd storage. With this implementation, a PersistentVolume will be allocated for every etcd member (either 1 or 3 depending on the HostedCluster control plane availability configuration).

"PersistentVolume"

PersistentVolumeEtcdStorage uses PersistentVolumes for etcd storage.

"Calico"

Calico specifies Calico as the SDN provider

"OpenShiftSDN"

OpenShiftSDN specifies OpenshiftSDN as the SDN provider

Min is the minimum number of nodes to maintain in the pool. Must be >= 1.

Max is the maximum number of nodes allowed in the pool. Must be >= 1.

UpgradeType specifies the type of strategy for handling upgrades.

Value must be one of: "InPlace", "Replace"

Replace is the configuration for rolling upgrades.

InPlace is the configuration for in-place upgrades.

AutoRepair specifies whether health checks should be enabled for machines in the NodePool. The default is false.

Type specifies the platform name.

Value must be one of: "AWS", "Agent", "IBMCloud", "KubeVirt", "None"

AWS specifies the configuration used when operating on AWS.

IBMCloud defines IBMCloud specific settings for components

Kubevirt specifies the configuration used when operating on KubeVirt platform.

Agent specifies the configuration used when using Agent platform.

ClusterName is the name of the HostedCluster this NodePool belongs to.

TODO(dan): Should this be a LocalObjectReference?

Release specifies the OCP release used for the NodePool. This informs the ignition configuration for machines, as well as other platform specific machine properties (e.g. an AMI on the AWS platform).

Platform specifies the underlying infrastructure provider for the NodePool and is used to configure platform specific behavior.

NodeCount is the desired number of nodes the pool should maintain. If unset, the default value is 0.

Management specifies behavior for managing nodes in the pool, such as upgrade strategies and auto-repair behaviors.

Autoscaling specifies auto-scaling behavior for the NodePool.

Config is a list of references to ConfigMaps containing serialized MachineConfig resources to be injected into the ignition configurations of nodes in the NodePool. The MachineConfig API schema is defined here:

https://github.com/openshift/machine-config-operator/blob/master/pkg/apis/machineconfiguration.openshift.io/v1/types.go#L172

Each ConfigMap must have a single key named “config” whose value is the JSON or YAML of a serialized MachineConfig.

TODO (alberto): this ConfigMaps are meant to contain MachineConfig, KubeletConfig and ContainerRuntimeConfig but MCO only supports MachineConfig in bootstrap mode atm. See: https://github.com/openshift/machine-config-operator/blob/9c6c2bfd7ed498bfbc296d530d1839bd6a177b0b/pkg/controller/bootstrap/bootstrap.go#L104-L119

NodeCount is the latest observed number of nodes in the pool.

Version is the semantic version of the latest applied release specified by the NodePool.

Conditions represents the latest available observations of the node pool’s current state.

Address is the host/ip that the NodePort service is exposed over.

Port is the port of the NodePort service. If <=0, the port is dynamically assigned when the service is created.

StorageClassName is the StorageClass of the data volume for each etcd member.

See https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1.

Size is the minimum size of the data volume for each etcd member.

Type is the type of infrastructure provider for the cluster.

Value must be one of: "AWS", "Agent", "IBMCloud", "KubeVirt", "None"

AWS specifies configuration for clusters running on Amazon Web Services.

Agent specifies configuration for agent-based installations.

IBMCloud defines IBMCloud specific settings for components

"AWS"

AWSPlatform represents Amazon Web Services infrastructure.

"Agent"

AgentPlatform represents user supplied insfrastructure booted with agents.

"IBMCloud"

IBMCloudPlatform represents IBM Cloud infrastructure.

"KubeVirt"

KubevirtPlatform represents Kubevirt infrastructure.

"None"

NonePlatform represents user supplied (e.g. bare metal) infrastructure.

Image is the image pullspec of an OCP release payload image.

Strategy is the node replacement strategy for nodes in the pool.

Value must be one of: "OnDelete", "RollingUpdate"

RollingUpdate specifies a rolling update strategy which upgrades nodes by creating new nodes and deleting the old ones.

MaxUnavailable is the maximum number of nodes that can be unavailable during the update.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding down.

This can not be 0 if MaxSurge is 0.

Defaults to 0.

Example: when this is set to 30%, old nodes can be deleted down to 70% of desired nodes immediately when the rolling update starts. Once new nodes are ready, more old nodes be deleted, followed by provisioning new nodes, ensuring that the total number of nodes available at all times during the update is at least 70% of desired nodes.

MaxSurge is the maximum number of nodes that can be provisioned above the desired number of nodes.

Value can be an absolute number (ex: 5) or a percentage of desired nodes (ex: 10%).

Absolute number is calculated from percentage by rounding up.

This can not be 0 if MaxUnavailable is 0.

Defaults to 1.

Example: when this is set to 30%, new nodes can be provisioned immediately when the rolling update starts, such that the total number of old and new nodes do not exceed 130% of desired nodes. Once old nodes have been deleted, new nodes can be provisioned, ensuring that total number of nodes running at any time during the update is at most 130% of desired nodes.

Type defines the type of kube secret encryption being used

Value must be one of: "aescbc", "kms"

KMS defines metadata about the kms secret encryption strategy

AESCBC defines metadata about the AESCBC secret encryption strategy

"aescbc"

AESCBC uses AES-CBC with PKCS#7 padding to do secret encryption

"kms"

KMS integrates with a cloud provider’s key management service to do secret encryption

Type is the publishing strategy used for the service.

NodePort configures exposing a service using a NodePort.

Service identifies the type of service being published.

ServicePublishingStrategy specifies how to publish Service.

Endpoint is the full etcd cluster client endpoint URL. For example:

https://etcd-client:2379

If the URL uses an HTTPS scheme, the TLS field is required.

TLS specifies TLS configuration for HTTPS etcd client endpoints.

"OnDelete"

UpgradeStrategyOnDelete replaces old nodes when the deletion of the associated node instances are completed.

"RollingUpdate"

UpgradeStrategyRollingUpdate means use a rolling update for nodes.

"InPlace"

UpgradeTypeInPlace is a strategy which replaces nodes in-place with no additional node capacity requirements.

"Replace"

UpgradeTypeReplace is a strategy which replaces nodes using surge node capacity.

Size specifies size (in Gi) of the storage device.

Must be greater than the image snapshot size or 8 (whichever is greater).

Type is the type of the volume.

IOPS is the number of IOPS requested for the disk. This is only valid for type io1.