Packages:
Package v1beta2 is the v1beta2 version of the API.
Resource Types:
| Field | Description | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion</br>
string |
sparkoperator.k8s.io/v1beta2
|
||||||||||||
kind</br>
string
|
ScheduledSparkApplication |
||||||||||||
metadata</br>
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||
spec</br>
ScheduledSparkApplicationSpec
|
|
||||||||||||
status</br>
ScheduledSparkApplicationStatus
|
SparkApplication represents a Spark application running on and using Kubernetes as a cluster manager.
| Field | Description | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
apiVersion</br>
string |
sparkoperator.k8s.io/v1beta2
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
kind</br>
string
|
SparkApplication |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
metadata</br>
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
spec</br>
SparkApplicationSpec
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
status</br>
SparkApplicationStatus
|
(Appears on: SparkApplicationStatus)
ApplicationState tells the current state of the application and an error message in case of failures.
| Field | Description |
|---|---|
state</br>
ApplicationStateType
|
|
errorMessage</br>
string
|
string alias)</p>(Appears on: ApplicationState)
ApplicationStateType represents the type of the current state of an application.
(Appears on: SparkApplicationSpec)
BatchSchedulerConfiguration used to configure how to batch scheduling Spark Application
| Field | Description |
|---|---|
queue</br>
string
|
(Optional)
Queue stands for the resource queue which the application belongs to, it’s being used in Volcano batch scheduler. |
priorityClassName</br>
string
|
(Optional)
PriorityClassName stands for the name of k8s PriorityClass resource, it’s being used for job scheduling and preemption, whether with the Volcano batch scheduler or the Kubernetes default scheduler. |
resources</br>
Kubernetes core/v1.ResourceList
|
(Optional)
Resources stands for the resource list custom request for. Usually it is used to define the lower-bound limit. If specified, volcano scheduler will consider it as the resources requested. |
string alias)</p>(Appears on: ScheduledSparkApplicationSpec)
(Appears on: SparkApplicationSpec)
Dependencies specifies all possible types of dependencies of a Spark application.
| Field | Description |
|---|---|
jars</br>
[]string
|
(Optional)
Jars is a list of JAR files the Spark application depends on. |
files</br>
[]string
|
(Optional)
Files is a list of files the Spark application depends on. |
pyFiles</br>
[]string
|
(Optional)
PyFiles is a list of Python files the Spark application depends on. |
packages</br>
[]string
|
(Optional)
Packages is a list of maven coordinates of jars to include on the driver and executor classpaths. This will search the local maven repo, then maven central and any additional remote repositories given by the “repositories” option. Each papckage should be of the form “groupId:artifactId:version”. |
excludePackages</br>
[]string
|
(Optional)
ExcludePackages is a list of “groupId:artifactId”, to exclude while resolving the dependencies provided in Packages to avoid dependency conflicts. |
repositories</br>
[]string
|
(Optional)
Repositories is a list of additional remote repositories to search for the maven coordinate given with the “packages” option. |
string alias)</p>(Appears on: SparkApplicationSpec)
DeployMode describes the type of deployment of a Spark application.
(Appears on: SparkApplicationStatus)
DriverInfo captures information about the driver.
| Field | Description |
|---|---|
webUIServiceName</br>
string
|
|
webUIPort</br>
int32
|
UI Details for the UI created via ClusterIP service accessible from within the cluster. |
webUIAddress</br>
string
|
|
webUIIngressName</br>
string
|
Ingress Details if an ingress for the UI was created. |
webUIIngressAddress</br>
string
|
|
podName</br>
string
|
(Appears on: SparkApplicationSpec)
DriverSpec is specification of the driver.
| Field | Description |
|---|---|
SparkPodSpec</br>
SparkPodSpec
|
(Members of |
podName</br>
string
|
(Optional)
PodName is the name of the driver pod that the user creates. This is used for the in-cluster client mode in which the user creates a client pod where the driver of the user application runs. It’s an error to set this field if Mode is not in-cluster-client. |
coreRequest</br>
string
|
(Optional)
CoreRequest is the physical CPU core request for the driver.
Maps to |
javaOptions</br>
string
|
(Optional)
JavaOptions is a string of extra JVM options to pass to the driver. For instance, GC settings or other logging. |
lifecycle</br>
Kubernetes core/v1.Lifecycle
|
(Optional)
Lifecycle for running preStop or postStart commands |
kubernetesMaster</br>
string
|
(Optional)
KubernetesMaster is the URL of the Kubernetes master used by the driver to manage executor pods and other Kubernetes resources. Default to https://kubernetes.default.svc. |
serviceAnnotations</br>
map[string]string
|
(Optional)
ServiceAnnotations defines the annoations to be added to the Kubernetes headless service used by executors to connect to the driver. |
string alias)</p>
DriverState tells the current state of a spark driver.
(Appears on: SparkApplicationSpec)
DynamicAllocation contains configuration options for dynamic allocation.
| Field | Description |
|---|---|
enabled</br>
bool
|
Enabled controls whether dynamic allocation is enabled or not. |
initialExecutors</br>
int32
|
(Optional)
InitialExecutors is the initial number of executors to request. If .spec.executor.instances is also set, the initial number of executors is set to the bigger of that and this option. |
minExecutors</br>
int32
|
(Optional)
MinExecutors is the lower bound for the number of executors if dynamic allocation is enabled. |
maxExecutors</br>
int32
|
(Optional)
MaxExecutors is the upper bound for the number of executors if dynamic allocation is enabled. |
shuffleTrackingTimeout</br>
int64
|
(Optional)
ShuffleTrackingTimeout controls the timeout in milliseconds for executors that are holding shuffle data if shuffle tracking is enabled (true by default if dynamic allocation is enabled). |
(Appears on: SparkApplicationSpec)
ExecutorSpec is specification of the executor.
| Field | Description |
|---|---|
SparkPodSpec</br>
SparkPodSpec
|
(Members of |
instances</br>
int32
|
(Optional)
Instances is the number of executor instances. |
coreRequest</br>
string
|
(Optional)
CoreRequest is the physical CPU core request for the executors.
Maps to |
javaOptions</br>
string
|
(Optional)
JavaOptions is a string of extra JVM options to pass to the executors. For instance, GC settings or other logging. |
deleteOnTermination</br>
bool
|
(Optional)
DeleteOnTermination specify whether executor pods should be deleted in case of failure or normal termination.
Maps to |
string alias)</p>(Appears on: SparkApplicationStatus)
ExecutorState tells the current state of an executor.
(Appears on: SparkPodSpec)
| Field | Description |
|---|---|
name</br>
string
|
Name is GPU resource name, such as: nvidia.com/gpu or amd.com/gpu |
quantity</br>
int64
|
Quantity is the number of GPUs to request for driver or executor. |
(Appears on: SparkApplicationSpec)
MonitoringSpec defines the monitoring specification.
| Field | Description |
|---|---|
exposeDriverMetrics</br>
bool
|
ExposeDriverMetrics specifies whether to expose metrics on the driver. |
exposeExecutorMetrics</br>
bool
|
ExposeExecutorMetrics specifies whether to expose metrics on the executors. |
metricsProperties</br>
string
|
(Optional)
MetricsProperties is the content of a custom metrics.properties for configuring the Spark metric system. If not specified, the content in spark-docker/conf/metrics.properties will be used. |
metricsPropertiesFile</br>
string
|
(Optional)
MetricsPropertiesFile is the container local path of file metrics.properties for configuring the Spark metric system. If not specified, value /etc/metrics/conf/metrics.properties will be used. |
prometheus</br>
PrometheusSpec
|
(Optional)
Prometheus is for configuring the Prometheus JMX exporter. |
(Appears on: SparkPodSpec)
NameKey represents the name and key of a SecretKeyRef.
| Field | Description |
|---|---|
name</br>
string
|
|
key</br>
string
|
(Appears on: SparkPodSpec)
NamePath is a pair of a name and a path to which the named objects should be mounted to.
| Field | Description |
|---|---|
name</br>
string
|
|
path</br>
string
|
(Appears on: MonitoringSpec)
PrometheusSpec defines the Prometheus specification when Prometheus is to be used for collecting and exposing metrics.
| Field | Description |
|---|---|
jmxExporterJar</br>
string
|
JmxExporterJar is the path to the Prometheus JMX exporter jar in the container. |
port</br>
int32
|
(Optional)
Port is the port of the HTTP server run by the Prometheus JMX exporter. If not specified, 8090 will be used as the default. |
configFile</br>
string
|
(Optional)
ConfigFile is the path to the custom Prometheus configuration file provided in the Spark image. ConfigFile takes precedence over Configuration, which is shown below. |
configuration</br>
string
|
(Optional)
Configuration is the content of the Prometheus configuration needed by the Prometheus JMX exporter. If not specified, the content in spark-docker/conf/prometheus.yaml will be used. Configuration has no effect if ConfigFile is set. |
(Appears on: SparkApplicationSpec)
RestartPolicy is the policy of if and in which conditions the controller should restart a terminated application. This completely defines actions to be taken on any kind of Failures during an application run.
| Field | Description |
|---|---|
type</br>
RestartPolicyType
|
Type specifies the RestartPolicyType. |
onSubmissionFailureRetries</br>
int32
|
(Optional)
OnSubmissionFailureRetries is the number of times to retry submitting an application before giving up. This is best effort and actual retry attempts can be >= the value specified due to caching. These are required if RestartPolicy is OnFailure. |
onFailureRetries</br>
int32
|
(Optional)
OnFailureRetries the number of times to retry running an application before giving up. |
onSubmissionFailureRetryInterval</br>
int64
|
(Optional)
OnSubmissionFailureRetryInterval is the interval in seconds between retries on failed submissions. |
onFailureRetryInterval</br>
int64
|
(Optional)
OnFailureRetryInterval is the interval in seconds between retries on failed runs. |
string alias)</p>(Appears on: RestartPolicy)
string alias)</p>(Appears on: ScheduledSparkApplicationStatus)
(Appears on: ScheduledSparkApplication)
| Field | Description |
|---|---|
schedule</br>
string
|
Schedule is a cron schedule on which the application should run. |
template</br>
SparkApplicationSpec
|
Template is a template from which SparkApplication instances can be created. |
suspend</br>
bool
|
(Optional)
Suspend is a flag telling the controller to suspend subsequent runs of the application if set to true. Defaults to false. |
concurrencyPolicy</br>
ConcurrencyPolicy
|
ConcurrencyPolicy is the policy governing concurrent SparkApplication runs. |
successfulRunHistoryLimit</br>
int32
|
(Optional)
SuccessfulRunHistoryLimit is the number of past successful runs of the application to keep. Defaults to 1. |
failedRunHistoryLimit</br>
int32
|
(Optional)
FailedRunHistoryLimit is the number of past failed runs of the application to keep. Defaults to 1. |
(Appears on: ScheduledSparkApplication)
| Field | Description |
|---|---|
lastRun</br>
Kubernetes meta/v1.Time
|
LastRun is the time when the last run of the application started. |
nextRun</br>
Kubernetes meta/v1.Time
|
NextRun is the time when the next run of the application will start. |
lastRunName</br>
string
|
LastRunName is the name of the SparkApplication for the most recent run of the application. |
pastSuccessfulRunNames</br>
[]string
|
PastSuccessfulRunNames keeps the names of SparkApplications for past successful runs. |
pastFailedRunNames</br>
[]string
|
PastFailedRunNames keeps the names of SparkApplications for past failed runs. |
scheduleState</br>
ScheduleState
|
ScheduleState is the current scheduling state of the application. |
reason</br>
string
|
Reason tells why the ScheduledSparkApplication is in the particular ScheduleState. |
(Appears on: SparkPodSpec)
SecretInfo captures information of a secret.
| Field | Description |
|---|---|
name</br>
string
|
|
path</br>
string
|
|
secretType</br>
SecretType
|
string alias)</p>(Appears on: SecretInfo)
SecretType tells the type of a secret.
(Appears on: SparkApplication, ScheduledSparkApplicationSpec)
SparkApplicationSpec describes the specification of a Spark application using Kubernetes as a cluster manager. It carries every pieces of information a spark-submit command takes and recognizes.
| Field | Description |
|---|---|
type</br>
SparkApplicationType
|
Type tells the type of the Spark application. |
sparkVersion</br>
string
|
SparkVersion is the version of Spark the application uses. |
mode</br>
DeployMode
|
Mode is the deployment mode of the Spark application. |
proxyUser</br>
string
|
(Optional)
ProxyUser specifies the user to impersonate when submitting the application. It maps to the command-line flag “–proxy-user” in spark-submit. |
image</br>
string
|
(Optional)
Image is the container image for the driver, executor, and init-container. Any custom container images for the driver, executor, or init-container takes precedence over this. |
imagePullPolicy</br>
string
|
(Optional)
ImagePullPolicy is the image pull policy for the driver, executor, and init-container. |
imagePullSecrets</br>
[]string
|
(Optional)
ImagePullSecrets is the list of image-pull secrets. |
mainClass</br>
string
|
(Optional)
MainClass is the fully-qualified main class of the Spark application. This only applies to Java/Scala Spark applications. |
mainApplicationFile</br>
string
|
(Optional)
MainFile is the path to a bundled JAR, Python, or R file of the application. |
arguments</br>
[]string
|
(Optional)
Arguments is a list of arguments to be passed to the application. |
sparkConf</br>
map[string]string
|
(Optional)
SparkConf carries user-specified Spark configuration properties as they would use the “–conf” option in spark-submit. |
hadoopConf</br>
map[string]string
|
(Optional)
HadoopConf carries user-specified Hadoop configuration properties as they would use the the “–conf” option in spark-submit. The SparkApplication controller automatically adds prefix “spark.hadoop.” to Hadoop configuration properties. |
sparkConfigMap</br>
string
|
(Optional)
SparkConfigMap carries the name of the ConfigMap containing Spark configuration files such as log4j.properties. The controller will add environment variable SPARK_CONF_DIR to the path where the ConfigMap is mounted to. |
hadoopConfigMap</br>
string
|
(Optional)
HadoopConfigMap carries the name of the ConfigMap containing Hadoop configuration files such as core-site.xml. The controller will add environment variable HADOOP_CONF_DIR to the path where the ConfigMap is mounted to. |
volumes</br>
[]Kubernetes core/v1.Volume
|
(Optional)
Volumes is the list of Kubernetes volumes that can be mounted by the driver and/or executors. |
driver</br>
DriverSpec
|
Driver is the driver specification. |
executor</br>
ExecutorSpec
|
Executor is the executor specification. |
deps</br>
Dependencies
|
(Optional)
Deps captures all possible types of dependencies of a Spark application. |
restartPolicy</br>
RestartPolicy
|
RestartPolicy defines the policy on if and in which conditions the controller should restart an application. |
nodeSelector</br>
map[string]string
|
(Optional)
NodeSelector is the Kubernetes node selector to be added to the driver and executor pods. This field is mutually exclusive with nodeSelector at podSpec level (driver or executor). This field will be deprecated in future versions (at SparkApplicationSpec level). |
failureRetries</br>
int32
|
(Optional)
FailureRetries is the number of times to retry a failed application before giving up. This is best effort and actual retry attempts can be >= the value specified. |
retryInterval</br>
int64
|
(Optional)
RetryInterval is the unit of intervals in seconds between submission retries. |
pythonVersion</br>
string
|
(Optional)
This sets the major Python version of the docker image used to run the driver and executor containers. Can either be 2 or 3, default 2. |
memoryOverheadFactor</br>
string
|
(Optional)
This sets the Memory Overhead Factor that will allocate memory to non-JVM memory.
For JVM-based jobs this value will default to 0.10, for non-JVM jobs 0.40. Value of this field will
be overridden by |
monitoring</br>
MonitoringSpec
|
(Optional)
Monitoring configures how monitoring is handled. |
batchScheduler</br>
string
|
(Optional)
BatchScheduler configures which batch scheduler will be used for scheduling |
timeToLiveSeconds</br>
int64
|
(Optional)
TimeToLiveSeconds defines the Time-To-Live (TTL) duration in seconds for this SparkAplication after its termination. The SparkApplication object will be garbage collected if the current time is more than the TimeToLiveSeconds since its termination. |
batchSchedulerOptions</br>
BatchSchedulerConfiguration
|
(Optional)
BatchSchedulerOptions provides fine-grained control on how to batch scheduling. |
sparkUIOptions</br>
SparkUIConfiguration
|
(Optional)
SparkUIOptions allows configuring the Service and the Ingress to expose the sparkUI |
dynamicAllocation</br>
DynamicAllocation
|
(Optional)
DynamicAllocation configures dynamic allocation that becomes available for the Kubernetes scheduleer backend since Spark 3.0. |
(Appears on: SparkApplication)
SparkApplicationStatus describes the current status of a Spark application.
| Field | Description |
|---|---|
sparkApplicationId</br>
string
|
SparkApplicationID is set by the spark-distribution(via spark.app.id config) on the driver and executor pods |
submissionID</br>
string
|
SubmissionID is a unique ID of the current submission of the application. |
lastSubmissionAttemptTime</br>
Kubernetes meta/v1.Time
|
LastSubmissionAttemptTime is the time for the last application submission attempt. |
terminationTime</br>
Kubernetes meta/v1.Time
|
CompletionTime is the time when the application runs to completion if it does. |
driverInfo</br>
DriverInfo
|
DriverInfo has information about the driver. |
applicationState</br>
ApplicationState
|
AppState tells the overall application state. |
executorState</br>
map[string]github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/apis/sparkoperator.k8s.io/v1beta2.ExecutorState
|
ExecutorState records the state of executors by executor Pod names. |
executionAttempts</br>
int32
|
ExecutionAttempts is the total number of attempts to run a submitted application to completion. Incremented upon each attempted run of the application and reset upon invalidation. |
submissionAttempts</br>
int32
|
SubmissionAttempts is the total number of attempts to submit an application to run. Incremented upon each attempted submission of the application and reset upon invalidation and rerun. |
string alias)</p>(Appears on: SparkApplicationSpec)
SparkApplicationType describes the type of a Spark application.
(Appears on: DriverSpec, ExecutorSpec)
SparkPodSpec defines common things that can be customized for a Spark driver or executor pod. TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
| Field | Description |
|---|---|
cores</br>
int32
|
(Optional)
Cores maps to |
coreLimit</br>
string
|
CoreLimit specifies a hard limit on CPU cores for the pod. Optional |
memory</br>
string
|
(Optional)
Memory is the amount of memory to request for the pod. |
memoryOverhead</br>
string
|
(Optional)
MemoryOverhead is the amount of off-heap memory to allocate in cluster mode, in MiB unless otherwise specified. |
gpu</br>
GPUSpec
|
(Optional)
GPU specifies GPU requirement for the pod. |
image</br>
string
|
(Optional)
Image is the container image to use. Overrides Spec.Image if set. |
configMaps</br>
[]NamePath
|
(Optional)
ConfigMaps carries information of other ConfigMaps to add to the pod. |
secrets</br>
[]SecretInfo
|
(Optional)
Secrets carries information of secrets to add to the pod. |
env</br>
[]Kubernetes core/v1.EnvVar
|
(Optional)
Env carries the environment variables to add to the pod. |
envVars</br>
map[string]string
|
(Optional)
EnvVars carries the environment variables to add to the pod.
Deprecated. Consider using |
envFrom</br>
[]Kubernetes core/v1.EnvFromSource
|
(Optional)
EnvFrom is a list of sources to populate environment variables in the container. |
envSecretKeyRefs</br>
map[string]github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg/apis/sparkoperator.k8s.io/v1beta2.NameKey
|
(Optional)
EnvSecretKeyRefs holds a mapping from environment variable names to SecretKeyRefs.
Deprecated. Consider using |
labels</br>
map[string]string
|
(Optional)
Labels are the Kubernetes labels to be added to the pod. |
annotations</br>
map[string]string
|
(Optional)
Annotations are the Kubernetes annotations to be added to the pod. |
volumeMounts</br>
[]Kubernetes core/v1.VolumeMount
|
(Optional)
VolumeMounts specifies the volumes listed in “.spec.volumes” to mount into the main container’s filesystem. |
affinity</br>
Kubernetes core/v1.Affinity
|
(Optional)
Affinity specifies the affinity/anti-affinity settings for the pod. |
tolerations</br>
[]Kubernetes core/v1.Toleration
|
(Optional)
Tolerations specifies the tolerations listed in “.spec.tolerations” to be applied to the pod. |
podSecurityContext</br>
Kubernetes core/v1.PodSecurityContext
|
(Optional)
PodSecurityContext specifies the PodSecurityContext to apply. |
securityContext</br>
Kubernetes core/v1.SecurityContext
|
(Optional)
SecurityContext specifies the container’s SecurityContext to apply. |
schedulerName</br>
string
|
(Optional)
SchedulerName specifies the scheduler that will be used for scheduling |
sidecars</br>
[]Kubernetes core/v1.Container
|
(Optional)
Sidecars is a list of sidecar containers that run along side the main Spark container. |
initContainers</br>
[]Kubernetes core/v1.Container
|
(Optional)
InitContainers is a list of init-containers that run to completion before the main Spark container. |
hostNetwork</br>
bool
|
(Optional)
HostNetwork indicates whether to request host networking for the pod or not. |
nodeSelector</br>
map[string]string
|
(Optional)
NodeSelector is the Kubernetes node selector to be added to the driver and executor pods. This field is mutually exclusive with nodeSelector at SparkApplication level (which will be deprecated). |
dnsConfig</br>
Kubernetes core/v1.PodDNSConfig
|
(Optional)
DnsConfig dns settings for the pod, following the Kubernetes specifications. |
terminationGracePeriodSeconds</br>
int64
|
(Optional)
Termination grace periond seconds for the pod |
serviceAccount</br>
string
|
(Optional)
ServiceAccount is the name of the custom Kubernetes service account used by the pod. |
(Appears on: SparkApplicationSpec)
SparkUIConfiguration is for driver UI specific configuration parameters.
| Field | Description |
|---|---|
servicePort</br>
int32
|
(Optional)
ServicePort allows configuring the port at service level that might be different from the targetPort. TargetPort should be the same as the one defined in spark.ui.port |
ingressAnnotations</br>
map[string]string
|
(Optional)
IngressAnnotations is a map of key,value pairs of annotations that might be added to the ingress object. i.e. specify nginx as ingress.class |
ingressTLS</br>
[]Kubernetes extensions/v1beta1.IngressTLS
|
(Optional)
TlsHosts is useful If we need to declare SSL certificates to the ingress object |
Generated with gen-crd-api-reference-docs
on git commit cf34460.