Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added --no-run option for rustdoc #83857

Merged
merged 11 commits into from
May 1, 2021
10 changes: 10 additions & 0 deletions src/librustdoc/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ crate struct Options {
/// For example, using ignore-foo to ignore running the doctest on any target that
/// contains "foo" as a substring
crate enable_per_target_ignores: bool,
/// Do not run doctests, compile them if should_test is active.
crate no_run: bool,

/// The path to a rustc-like binary to build tests with. If not set, we
/// default to loading from `$sysroot/bin/rustc`.
Expand Down Expand Up @@ -198,6 +200,7 @@ impl fmt::Debug for Options {
.field("runtool_args", &self.runtool_args)
.field("enable-per-target-ignores", &self.enable_per_target_ignores)
.field("run_check", &self.run_check)
.field("no_run", &self.no_run)
.finish()
}
}
Expand Down Expand Up @@ -456,6 +459,12 @@ impl Options {
test_args.iter().flat_map(|s| s.split_whitespace()).map(|s| s.to_string()).collect();

let should_test = matches.opt_present("test");
let no_run = matches.opt_present("no-run");

if !should_test && no_run {
diag.err("the `--test` flag must be passed to enable `--no-run`");
return Err(1);
}

let output =
matches.opt_str("o").map(|s| PathBuf::from(&s)).unwrap_or_else(|| PathBuf::from("doc"));
Expand Down Expand Up @@ -662,6 +671,7 @@ impl Options {
enable_per_target_ignores,
test_builder,
run_check,
no_run,
render_options: RenderOptions {
output,
external_html,
Expand Down
3 changes: 2 additions & 1 deletion src/librustdoc/doctest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -940,13 +940,14 @@ impl Tester for Collector {
let report_unused_externs = |uext| {
unused_externs.lock().unwrap().push(uext);
};
let no_run = config.no_run || options.no_run;
let res = run_test(
&test,
&cratename,
line,
options,
config.should_panic,
config.no_run,
no_run,
config.test_harness,
runtool,
runtool_args,
Expand Down
1 change: 1 addition & 0 deletions src/librustdoc/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,7 @@ fn opts() -> Vec<RustcOptGroup> {
"[unversioned-shared-resources,toolchain-shared-resources,invocation-specific]",
)
}),
unstable("no-run", |o| o.optflag("", "no-run", "Compile doctests without running them")),
]
}

Expand Down
6 changes: 6 additions & 0 deletions src/test/rustdoc-ui/no-run-flag-error.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// test the behavior of the --no-run flag without the --test flag

// compile-flags:-Z unstable-options --no-run --test-args=--test-threads=1
// error-pattern: the `--test` flag must be passed

pub fn f() {}
2 changes: 2 additions & 0 deletions src/test/rustdoc-ui/no-run-flag-error.stderr
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
error: the `--test` flag must be passed to enable `--no-run`

38 changes: 38 additions & 0 deletions src/test/rustdoc-ui/no-run-flag.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// test the behavior of the --no-run flag

// check-pass
// compile-flags:-Z unstable-options --test --no-run --test-args=--test-threads=1
// normalize-stdout-test: "src/test/rustdoc-ui" -> "$$DIR"
// normalize-stdout-test "finished in \d+\.\d+s" -> "finished in $$TIME"

/// ```
/// let a = true;
/// ```
/// ```should_panic
/// panic!()
/// ```
/// ```ignore (incomplete-code)
/// fn foo() {
/// ```
/// ```no_run
/// loop {
/// println!("Hello, world");
/// }
/// ```
/// fails to compile
/// ```compile_fail
/// let x = 5;
/// x += 2; // shouldn't compile!
/// ```
/// Ok the test does not run
/// ```
/// panic!()
/// ```
/// Ok the test does not run
/// ```should_panic
/// loop {
/// println!("Hello, world");
/// panic!()
/// }
/// ```
pub fn f() {}
12 changes: 12 additions & 0 deletions src/test/rustdoc-ui/no-run-flag.stdout
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@

running 7 tests
test $DIR/no-run-flag.rs - f (line 11) ... ok
test $DIR/no-run-flag.rs - f (line 14) ... ignored
test $DIR/no-run-flag.rs - f (line 17) ... ok
test $DIR/no-run-flag.rs - f (line 23) ... ok
test $DIR/no-run-flag.rs - f (line 28) ... ok
test $DIR/no-run-flag.rs - f (line 32) ... ok
test $DIR/no-run-flag.rs - f (line 8) ... ok
Comment on lines +3 to +9
Copy link
Contributor

@pickfire pickfire Apr 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it didn't run why is it "ok"? IIRC that is the same as test passing right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so adding the --no-run flag change the behavior of the test like it was annotated as ```no_run. Maybe this is unwanted and something else should be displayed.

Copy link
Contributor

@pickfire pickfire Apr 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping @CraftSpider @jyn514 should we have a different text? "ok" may be confusing since it didn't run. Also note that we may also want a different color or the same color as "ignored".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm. I don't think 'ignored' is quite right, but 'ok' is also a bit confusing. If we can make it something new, maybe something like 'built' or 'checked'? If not, I think 'ok' is more accurate than 'ignored'.
Color-wise, I'm fine with it being ok colored, as it is a requested change to behavior, not implicit, so it's requested to treat it as success. If that makes sense.

Copy link
Contributor Author

@ABouttefeux ABouttefeux Apr 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After investigations I think that in order to do so, I have to modify the lib test. This will also affect all tests and not just doctests. Do you think this is a worthwhile investment and should this be in a different PR ? Anyway I would be interested in implementing that but the scope is a bit bigger than I anticipated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no_run by itself is useless.
To summarize:

  • if you run rustdoc -Z unstable-options --no-run it just build the documentation, it has the same behavior as just rustdoc
  • rustdoc -Z unstable-options --no-run --test run the doctest set. It behave as if every test had the no_run meaning they won't run. Meaning that test like
    ```
    panic!()
    ```
    will print as ok like
    ```no_run
    panic!()
    ```
    without the --no-run flag

I agree that is confusing. Maybe a error should be raised if no_run is active and not should_test. However I think that if you run rustdoc -Z unstable-options --no-run your intention was probably to run the test instead of documenting your project.

I assume that in the original issue the final goal would be to be able to generate bin (with --persist-doctests) but without running them.

Or maybe a another possible case would be to unify the behavior of cargo test --no-run (provided that down the line cargo is also modified). For instance cargo test --all test both regular test and doctest but cargo test --all --no-run skips doctest cargo test --doc --no-run give an error.

Now if the explicit goal is to harmonize with regular test the output should just be empty.
Another option would for me to change the test lib such that no_run test display something like test foo ... checked, provided that such change is desirable (to discuss thought the RFC process ?).
Moreover test with compile_fail should also print something different maybe checked is also desirable.

I am new contributing for rust so I am not quite sure of what to do to solve this issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that is confusing. Maybe a error should be raised if no_run is active and not should_test. However I think that if you run rustdoc -Z unstable-options --no-run your intention was probably to run the test instead of documenting your project.

I think we should give a hard error here; if people meant to run the test they can add --test. Could you suggest adding --test in the error message?

See

} else if !out_fmt.is_json() && show_coverage {
diag.struct_err(
"html output format isn't supported for the --show-coverage option",
)
.emit();
return Err(1);
}
for an example of how to give an error if the options are inconsistent.

I think all your other changes can go in a follow-up PR, I don't think we should be changing libtest here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thank you. I have not change libtest yet I will work on that too :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not change libtest in this pr. I am not a maintainer and don't know who to ask to approve; finding out will delay landing this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I was not clear. I will do another PR for libtest


test result: ok. 6 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in $TIME