feat: Merge changes on main to spark workshop branch

[alanty]

What does this PR do?

Pulling in latest updates from main to the workshop branch and resolving merge conflicts.

Motivation

When merging main into the workshop branch we hit some merge conflicts (see #782).

More

• Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
• Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
• Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
• Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

• E2E Test successfully complete before merge?

Additional Notes

This merge hasn't been tested, I've just tried to unwind the changes. I may have missed something.