aws: error: The step type spark_application is not supported.

[hongbo-miao]

Describe the feature

I have an EMR cluster with step "Spark application" which created from EMR UI.

Now I am hoping to create it by CLI.

I got the command by clicking "View command for cloning cluster" button

It gives me command like

aws emr create-cluster \
    # ...
    --steps '[{"Name":"xxx","ActionOnFailure":"TERMINATE_CLUSTER","Jar":"command-runner.jar","Properties":"","Args":["spark-submit","--deploy-mode","cluster","s3://my-bucket/MySparkApp-assembly-1.0.jar"],"Type":"SPARK_APPLICATION"}]'

However, when I run this command, I got error

aws: error: The step type spark_application is not supported.

Based on CLI document, currently --steps only supports

"Type": "CUSTOM_JAR"|"STREAMING"|"HIVE"|"PIG"|"IMPALA"

It would be great to support SPARK_APPLICATION, thanks! 😃

Use Case

It would be convenient to use CLI to create EMR cluster with step "Spark application".

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CLI version used

aws-cli/2.8.12 Python/3.9.11 Darwin/22.5.0 exe/x86_64 prompt/off

Environment details (OS name and version, etc.)

macOS

[RyanFitzSimmonsAK]

Hi @hongbo-miao, thanks for reaching out. In the developer guide for EMR, it looks like the examples use Type=Spark rather than SPARK_APPLICATION. Could you try formatting it similarly to the examples, and letting me know if that works? Thanks!

[hongbo-miao]

Hi @RyanFitzSimmonsAK hmm the error is something else now

An error occurred (ValidationException) when calling the RunJobFlow operation: An instance group must have at least one instance.

Although this new error also should not exist as I exported, when I rerun, it should create same EMR cluster.

Back to this issue, I think SPARK works!

I guess just need

• Update the document at https://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html
  • I opened a pull request at fix(emr): add missing SPARK type #7973
• Make this "View command for cloning cluster" button use SPARK instead of SPARK_APPLICATION
  

Feel free to use this ticket to track, or create other tickets and close this one. Thanks! 😃

[RyanFitzSimmonsAK]

Back to this issue, I think SPARK works!

Glad to hear it. Did you change any other part of your command? And do you need help with the new error, or were you able to get this feature working?

[hongbo-miao]

That is all changes I have. 😃
The new issue is a different one, I will create a new ticket later when I got time, thanks!

[RyanFitzSimmonsAK]

I'd like to make sure we get it working before I raise the issue with the service team about the button in the console. Let's follow up in this issue, and I'll bring everything up with them once we've figured out what all needs to be fixed.

[hongbo-miao]

Sounds good! I will provide more info regarding the new error. Hopefully this week!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.