Crate-level proc-macro attributes have issues

[Rantanen]

#![my_attribute]

While normal proc-macro attribute work, crate level proc-macro attributes seem to have several issues. The clearest of these is the order of the attributes and extern crate statements: #41430. #![feature(extern_absolute_paths)] feature can work around this specific issue; However this alone doesn't make proc macros work.

Some of these issues are due to lack of library support (syn), but some are caused by rustc itself.

Identity macro ✔️

First of all: The following proc-macro works just fine with nightly 29c8276:

#[proc_macro_attribute]
fn attribute(_: TokenStream, input: TokenStream) -> TokenStream {
    input
}

#![feature(extern_absolute_paths)]
#![::my_crate::attribute]
fn main() {}

Round trip through FromIterator ❌

As long as the macro doesn't touch the input, everything works just fine. However this isn't really that useful for a macro. Once we try doing anything with the input token stream (short of a .clone()), things start breaking.

#[proc_macro_attribute]
fn attribute(_: TokenStream, input: TokenStream) -> TokenStream {
    TokenStream::from_iter( input.into_iter() )
}

#![feature(extern_absolute_paths)]
#![::my_crate::attribute]
fn main() {}

This results in the following error message

error: expected identifier, found `{`
 --> <macro expansion>:1:1
  |
1 | pub mod  {
  | ^^^ expected identifier

The reason seems to be that the TokenStream the attribute receives looks like:

pub mod {
    ...
}

This is most likely caused by the ast::Crate modeling the items as a Mod with no Ident: https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs#L447-L451

Stripping the pub mod away ❌

Implementing a Crate type for Syn and having its to_tokens skip the pub mod { .. } bits allows the parser to process the output correctly. I'm also hoping this would be the way to go, since the parse_crate_mod method just parses inner attributes and mod items without seemingly caring about the mod keywords or braces.

However now we encounter the following ICE:

thread 'rustc' panicked at 'internal error: entered unreachable code', libsyntax/ext/expand.rs:258:18

error: an inner attribute is not permitted in this context
  |
  = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like `#[test]`, annotate the item following them.

thread 'rustc' panicked at 'internal error: entered unreachable code', libsyntax/ext/expand.rs:258:18
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
   6: syntax::ext::expand::MacroExpander::expand_crate
   7: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
   8: rustc_driver::driver::phase_2_configure_and_expand_inner
   9: rustc_driver::driver::compile_input
  10: rustc_driver::run_compiler

error: internal compiler error: unexpected panic

As far as I can tell, the only way for this ICE to happen is for the MacroExpander to successfully expand the attribute. However the code expects the expansion to result in ast::Item(Mod)

I tried compiling a debug version of rustc for this, however then I end up with...

syn::parse(input) panics ❌

At this point my attribute fn does nothing but parses and quotes the input using syn using a custom Crate type that handles pub mod without an ident:

#[proc_macro_attribute]
pub fn attribute(_attr: TokenStream, input: TokenStream) -> TokenStream {
    let file : Crate = parse( input ).unwrap();
    quote!( #file ).into()
}

This results in the following panic:

thread '' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!',

   Compiling test_crate v0.1.0 (file:///C:/Dev/Projects/crate_level_proc_macro/test_crate)
thread '<unnamed>' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!', libproc_macro\lib.rs:851:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at C:\Dev\Projects\rust\src\libstd\sys\windows\backtrace\mod.rs:65
   1: std::sys_common::backtrace::_print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:71
   2: std::sys_common::backtrace::print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:58
   3: std::panicking::default_hook::{{closure}}
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:380
   4: std::panicking::default_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:396
   5: std::panicking::rust_panic_with_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:576
   6: std::panicking::begin_panic<str*>
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:537
   7: proc_macro::__internal::with_sess
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:851
   8: proc_macro::TokenTree::from_internal
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:682
   9: proc_macro::{{impl}}::next
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:570
  10: proc_macro2::imp::{{impl}}::next
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\proc-macro2-0.2.2\src\unstable.rs:117
  11: proc_macro2::{{impl}}::next
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\proc-macro2-0.2.2\src\lib.rs:326
  12: syn::buffer::TokenBuffer::inner_new
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\syn-0.12.12\src\buffer.rs:176
  13: syn::buffer::TokenBuffer::new2
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\syn-0.12.12\src\buffer.rs:228
  14: syn::synom::{{impl}}::parse2<fn(syn::buffer::Cursor) -> core::result::Result<(crate_level_proc_macro::Crate, syn::buffer::Cursor), syn::error::ParseError>,crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\syn-0.12.12\src\synom.rs:221
  15: syn::parse2<crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\syn-0.12.12\src\lib.rs:601
  16: syn::parse<crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\garden.eu.org-1ecc6299db9ec823\syn-0.12.12\src\lib.rs:580
  17: crate_level_proc_macro::attribute
             at C:\Dev\Projects\crate_level_proc_macro\src\lib.rs:54
  18: std::panicking::try::do_call
  19: _rust_maybe_catch_panic
  20: <std::thread::local::LocalKey<T>>::with
  21: <syntax_ext::proc_macro_impl::AttrProcMacro as syntax::ext::base::AttrProcMacro>::expand
  22: syntax::ext::expand::MacroExpander::expand
  23: syntax::ext::expand::MacroExpander::expand
  24: syntax::ext::expand::MacroExpander::expand_crate
  25: rustc_driver::driver::count_nodes
  26: rustc_driver::driver::count_nodes
  27: rustc_driver::driver::compile_input
  28: rustc_driver::run_compiler
error: custom attribute panicked
 --> src\main.rs:2:1
  |
2 | #![::crate_level_proc_macro::attribute]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: proc_macro::__internal::with_sess() called before set_parse_sess()!

The fact that the same code breaks on my own stage-1 rustc build based on the commit 29c8276ce, while it "works" on nightly 29c8276ce 2018-02-07 is a bit weird. Shouldn't those two compilers have the same code base?

In any case, I can try working around that specific issue by not going through proc_macro2 crate and instead using Syn's parse_str. However...

.to_string() on TokenStream ❌

We can skip proc_macro2 by using the TokenStream::to_string() to turn the stream into rust-code that Syn can parse on its own. This gives us the following attribute function:

#[proc_macro_attribute]
pub fn attribute(_attr: TokenStream, input: TokenStream) -> TokenStream {
    let file : Crate = parse_str( &input.to_string() ).unwrap();
    quote!( #file ).into()
}

Running this results in the following panic

thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0',

   Compiling test_crate v0.1.0 (file:///C:/Dev/Projects/crate_level_proc_macro/test_crate)
thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0', C:\Dev\Projects\rust\src\liballoc\vec.rs:1551:10
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at C:\Dev\Projects\rust\src\libstd\sys\windows\backtrace\mod.rs:65
   1: std::sys_common::backtrace::_print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:71
   2: std::sys_common::backtrace::print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:58
   3: std::panicking::default_hook::{{closure}}
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:380
   4: std::panicking::default_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:396
   5: std::panicking::rust_panic_with_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:576
   6: std::panicking::begin_panic<alloc::string::String>
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:537
   7: std::panicking::begin_panic_fmt
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:521
   8: std::panicking::rust_begin_panic
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:497
   9: core::panicking::panic_fmt
             at C:\Dev\Projects\rust\src\libcore\panicking.rs:71
  10: core::panicking::panic_bounds_check
             at C:\Dev\Projects\rust\src\libcore\panicking.rs:58
  11: alloc::vec::{{impl}}::index
             at C:\Dev\Projects\rust\src\liballoc\vec.rs:1551
  12: syntax_pos::span_encoding::SpanInterner::get
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:134
  13: syntax_pos::span_encoding::decode::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:109
  14: syntax_pos::span_encoding::with_span_interner::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:144
  15: std::thread::local::LocalKey<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>>::try_with
             at C:\Dev\Projects\rust\src\libstd\thread\local.rs:377
  16: std::thread::local::LocalKey<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>>::with<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>,closure,syntax_pos::SpanData>
             at C:\Dev\Projects\rust\src\libstd\thread\local.rs:290
  17: syntax_pos::span_encoding::with_span_interner
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:144
  18: syntax_pos::span_encoding::decode
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:109
  19: syntax_pos::span_encoding::Span::data
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:47
  20: syntax_pos::span_encoding::Span::lo
             at C:\Dev\Projects\rust\src\libsyntax_pos\lib.rs:196
  21: syntax::print::pprust::State::print_item
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:1173
  22: syntax::print::pprust::item_to_string::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:332
  23: syntax::print::pprust::to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:165
  24: syntax::print::pprust::item_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:331
  25: syntax::print::pprust::token_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:269
  26: syntax::print::pprust::PrintState::print_tt<syntax::print::pprust::State>
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:794
  27: syntax::print::pprust::PrintState::print_tts<syntax::print::pprust::State>
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:818
  28: syntax::print::pprust::tokens_to_string::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:320
  29: syntax::print::pprust::to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:165
  30: syntax::print::pprust::tokens_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:320
  31: syntax::tokenstream::{{impl}}::fmt
             at C:\Dev\Projects\rust\src\libsyntax\tokenstream.rs:555
  32: core::fmt::{{impl}}::fmt<proc_macro::TokenStream>
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1566
  33: core::fmt::Formatter::run
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1084
  34: core::fmt::write
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1030
  35: core::fmt::Write::write_fmt<alloc::string::String>
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:226
  36: alloc::string::{{impl}}::to_string<proc_macro::TokenStream>
             at C:\Dev\Projects\rust\src\liballoc\string.rs:2054
  37: crate_level_proc_macro::attribute
             at C:\Dev\Projects\crate_level_proc_macro\src\lib.rs:54
  38: std::panicking::try::do_call
  39: _rust_maybe_catch_panic
  40: <std::thread::local::LocalKey<T>>::with
  41: <syntax_ext::proc_macro_impl::AttrProcMacro as syntax::ext::base::AttrProcMacro>::expand
  42: syntax::ext::expand::MacroExpander::expand
  43: syntax::ext::expand::MacroExpander::expand
  44: syntax::ext::expand::MacroExpander::expand_crate
  45: rustc_driver::driver::count_nodes
  46: rustc_driver::driver::count_nodes
  47: rustc_driver::driver::compile_input
  48: rustc_driver::run_compiler
error: custom attribute panicked
 --> src\main.rs:2:1
  |
2 | #![::crate_level_proc_macro::attribute]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `test_crate`.

I suspect this has something to do with the token stream including those pub mod bits, which do not have proper span information. The panic seems to result from the pretty printer trying to fetch SpanData for tokens that don't have any.

---

That's as far as I've gotten with this. I still believe crate-level macro attributes are a strong feature that is worth pursuing.

My own use case for these is to emit some FFI entry point functions, which enable the crate to be used as a COM library. While these could be done from a normal macro, an attribute would make it feel more declarative.

These attributes also came up in custom test frameworks. A while back I was playing around with a Catch-inspired ratcc test framework, where I would have loved to to use a crate-level attribute to alter the crate contents.

And continuing on the test framework theme, they also came up in the eRFC discussion for custom test frameworks: https://github.com/catchorg/Catch2/blob/master/examples/010-TestCase.cpp

Given the amount of different panics I encountered in trying to work around the issues hints that they are currently far from supported. However the fact that an identity attribute works makes me hopeful that if the parsing could be fixed so that a TokenStream could be parsed into an ast::Crate (or similar), then the rest of the macro pipeline would "just work".

[Rantanen]

@Manishearth I would love to hear what other issues came up in the further discussion mentioned in the eRFC comments.

[Manishearth]

This is a duplicate of #41430 , please move your comments there

[Manishearth]

cc @jseyfried