[batch] hailctl batch submit improvements and testing

[ehigham]

Potentially backwards incompatible hailctl batch submit rewrite and various other fixes. Originally authored by @danking in #14186 and later @jigold in #14351. It's had some tweaks since then as well as a liberal sprinkling of my biases.

Issues resolved:

1. build.yaml tests must not use exit 0 as it exits the test early.

2. Always prefer orjson to json.

3. The --files option has been changed significantly, allowing for special characters, environment variables and glob patterns in SRC paths. DST paths may be absolute or relative to the job's working directory.

4. Contra to @danking's approach, I'm not giving special dispensation to trailing slashes in DST paths. Instead I'm mimicking the behaviour of gnu tools cp -RT SRC DST.

Fixes #14274

This change has low security impact.

[ehigham]

This change is @jigold's edits manually applied on main.
I couldn't figure out the merge conflicts when rebasing #14351.
Mixing merges and rebases causes havoc!

[chrisvittal]

Looks good. Thanks for pushing this over the line.

[cjllanwarne]

This seems well implemented and the test cases give me good confidence. My only minor concern is relying on a trailing '/' to decide whether something is a directory or not. If you can make me feel better about that, I'll happily 👍 . Oh, and I think a minor documentation improvement is possible too 😄

[ehigham]

This was mostly implemented by @danking - I just tweaked the tests to not drool invalid current working directories.
I hate trialing slashes to denote directories. If we're doing that anywhere here I'll gladly kill it.