Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Election scheduler predicates #3296

Merged
merged 5 commits into from
May 27, 2021
Merged

Conversation

clemahieu
Copy link
Contributor

This change simplifies logic in the election scheduler. It extracts predicate functions so identical logic can be checked in the condition_variable wait function and also inside the election_scheduler processing loop. This allows new predicates to be easily added and also eliminates risk of disjoin checks between the loop and the condition variable.

SergiySW
SergiySW previously approved these changes May 21, 2021
@MajorChump
Copy link
Contributor

MajorChump commented May 22, 2021

Thread 44 "Request loop" received signal SIGSEGV, Segmentation fault.
boost::multi_index::detail::hashed_index_node_alg<boost::multi_index::detail::hashed_index_node_impl<std::allocator >, boost::multi_index::detail::hashed_unique_tag>::unlinkboost::multi_index::detail::default_assigner (x=0x7ef964e58fb0,
assign=...) at /usr/local/boost/include/boost/multi_index/detail/hash_index_node.hpp:287
287 assign(x->next()->prior()->prior(),x->prior());

Seem to get this every 10 - 11 minutes running a compiled version of this branch.

Backtrace:

#0 boost::multi_index::detail::hashed_index_node_alg<boost::multi_index::detail::hashed_index_node_impl<std::allocator<char> >, boost::multi_index::detail::hashed_unique_tag>::unlink<boost::multi_index::detail::default_assigner> (x=0x7edb44ab8d80, assign=...) at /root/boost_build/include/boost/multi_index/detail/hash_index_node.hpp:287
#1 0x0000563fa43178c7 in boost::multi_index::detail::hashed_index_node_alg<boost::multi_index::detail::hashed_index_node_impl<std::allocator<char> >, boost::multi_index::detail::hashed_unique_tag>::unlink (x=<optimized out>) at /root/boost_build/include/boost/multi_index/detail/hash_index_node.hpp:275 #2 boost::multi_index::detail::hashed_index<boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, boost::hash<nano::qualified_root>, std::equal_to<nano::qualified_root>, boost::multi_index::detail::nth_layer<2, nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >, boost::mpl::v_item<nano::active_transactions::tag_root, boost::mpl::vector0<mpl_::na>, 0>, boost::multi_index::detail::hashed_unique_tag>::unlink (this=0x563fa61bb040, x=0x7edb44ab8d10) at /root/boost_build/include/boost/multi_index/hashed_index.hpp:1280
#3 boost::multi_index::detail::hashed_index<boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, boost::hash<nano::qualified_root>, std::equal_to<nano::qualified_root>, boost::multi_index::detail::nth_layer<2, nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >, boost::mpl::v_item<nano::active_transactions::tag_root, boost::mpl::vector0<mpl_::na>, 0>, boost::multi_index::detail::hashed_unique_tag>::erase_ (this=0x563fa61bb040, x=0x7edb44ab8d10) at /root/boost_build/include/boost/multi_index/hashed_index.hpp:826 --Type <RET> for more, q to quit, c to continue without paging--c
#4 boost::multi_index::detail::random_access_index<boost::multi_index::detail::nth_layer<1, nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >, boost::mpl::v_item<nano::active_transactions::tag_random_access, boost::mpl::vector0<mpl_::na>, 0> >::erase_ (x=0x7edb44ab8d10, this=0x563fa61bb040) at /root/boost_build/include/boost/multi_index/random_access_index.hpp:801
#5 boost::multi_index::multi_index_container<nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >::erase_ (x=<optimized out>, this=<optimized out>) at /root/boost_build/include/boost/multi_index_container.hpp:790
#6 boost::multi_index::detail::index_base<nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >::final_erase_ (x=<optimized out>, this=0x563fa61bb040) at /root/boost_build/include/boost/multi_index/detail/index_base.hpp:255
#7 boost::multi_index::detail::hashed_index<boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, boost::hash<nano::qualified_root>, std::equal_to<nano::qualified_root>, boost::multi_index::detail::nth_layer<2, nano::active_transactions::conflict_info, boost::multi_index::indexed_by<boost::multi_index::random_access<boost::multi_index::tag<nano::active_transactions::tag_random_access, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::multi_index::hashed_unique<boost::multi_index::tag<nano::active_transactions::tag_root, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, boost::multi_index::member<nano::active_transactions::conflict_info, nano::qualified_root, &nano::active_transactions::conflict_info::root>, mpl_::na, mpl_::na>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, std::allocator<nano::active_transactions::conflict_info> >, boost::mpl::v_item<nano::active_transactions::tag_root, boost::mpl::vector0<mpl_::na>, 0>, boost::multi_index::detail::hashed_unique_tag>::erase (position=..., this=0x563fa61bb040) at /root/boost_build/include/boost/multi_index/hashed_index.hpp:317
#8 nano::active_transactions::erase (this=this@entry=0x563fa61bb030, root_a=...) at /root/nano-node/nano/node/active_transactions.cpp:1051
#9 0x0000563fa431a5d7 in nano::active_transactions::request_confirm (this=this@entry=0x563fa61bb030, lock_a=...) at /root/nano-node/nano/node/active_transactions.cpp:337
#10 0x0000563fa431ac80 in nano::active_transactions::request_loop (this=0x563fa61bb030) at /root/nano-node/nano/node/active_transactions.cpp:583
#11 0x0000563fa484a28a in thread_proxy ()
#12 0x00007f1c0e48b6db in start_thread () from target:/lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007f1c0dc1271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

@clemahieu
Copy link
Contributor Author

@m-king Is that running the correct branch? That line number 1051 seems empty and 337 isn't calling the correct function.


@MajorChump
Copy link
Contributor

@m-king Is that running the correct branch? That line number 1051 seems empty and 337 isn't calling the correct function.

@clemahieu

Sorry forgot to mention it's running my child branch with the additional stats so the numbers are slightly off, I haven't touched any of this logic, so I think its to do with the new erase_oldest?. As follows

https://github.com/m-king/nano-node/blob/election_overflow_stat_drop/nano/node/active_transactions.cpp#L1051
https://github.com/m-king/nano-node/blob/election_overflow_stat_drop/nano/node/active_transactions.cpp#L337

@clemahieu
Copy link
Contributor Author

The crash above was caused by the erase methods assuming iterators would be valid across mutex lock/unlock which was no longer guaranteed when the election scheduler became able to delete elections. This was fixed in #3302

clemahieu added 5 commits May 25, 2021 09:47
…redicate functions so identical logic can be checked in the condition_variable wait function and also inside the election_scheduler processing loop. This allows new predicates to be easily added and also eliminates risk of disjoin checks between the loop and the condition variable.
…scheduler::manual since the semantics of ::manual will be changed to not wait for vacancy before inserting.
… vacancy is not considered and elections are started for any blocks passed in.

- Moves responsibility for trimming down election count from active_transaction in to election_scheduler. This is done so that more advanced filling/spilling can be done by the election scheduler in the future.
- Erases from the active_transactions container by oldest transaction rather than by newest.
… vacancy is not considered and elections are started for any blocks passed in.

- Moves responsibility for trimming down election count from active_transaction in to election_scheduler. This is done so that more advanced filling/spilling can be done by the election scheduler in the future.
- Erases from the active_transactions container by oldest transaction rather than by newest.
@clemahieu clemahieu force-pushed the election_scheduler_predicates branch from 0cf134b to e1d7025 Compare May 25, 2021 09:35
@clemahieu clemahieu merged commit d5d9fdc into develop May 27, 2021
MajorChump pushed a commit to MajorChump/nano-node that referenced this pull request May 27, 2021
clemahieu added a commit that referenced this pull request May 28, 2021
@clemahieu clemahieu added this to the V22.1 milestone Jun 3, 2021
@zhyatt zhyatt deleted the election_scheduler_predicates branch September 21, 2021 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants