Crates.io

[nuxeh]

https://crates.io/crates/webpage
https://crates.io/crates/openssl
and probably others...

Produce 404s:

ERROR Inner { kind: ClientError(404), url: Some("https://crates.io/crates/openssl") }

Curling the page produces what looks like json:

$ curl -s https://crates.io/crates/openssl
{"errors":[{"detail":"Not Found"}]}

[benjamb]

Upstream bug: rust-lang/crates.io#788

[benjamb]

tl;dr specify the content we want as (text/)html, if <noscript> tag is present, it's due to not supporting javascript. Could maybe make use of https://github.com/jonhoo/fantoccini?

[nuxeh]

Interesting stuff, but seems like it relies on a running (possibly headless) instance of a web browser? Could be heavy.

[benjamb]

@nuxeh Hrm, didn't escape the tag I was referring to. Aye, could be heavy, though it would only be in the case of noscript. I think just specifying the context we want as html should at least cause the bot to output something.