How to Assign a Parameter Value in a Spark Command?

Question

I do not like the idea to assign parameter values on user interface or in configuration files. How to do it with Spark commands?

Answer

Spark configuration options can be defined either in a configuration file or in Spark commands.

To assign a parameter value, run the --conf Parameter name=Parameter value command on a Spark client. Add the parameter name and its value after --conf. The following is an example.

--conf spark.eventQueue.size=50000