Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Analysis library crash on exit #908

Open
sdobbs opened this issue Mar 19, 2025 · 0 comments
Open

Analysis library crash on exit #908

sdobbs opened this issue Mar 19, 2025 · 0 comments

Comments

@sdobbs
Copy link
Contributor

sdobbs commented Mar 19, 2025

Rarely, in REST production test runs, there are segfaults as the process is terminating. It looks like this is in the ANALYSIS library.

Example output files are in this directory, and the stack trace is given below:
/volatile/halld/home/gxproj4/jobs/ver13-perl/job_info_120400_191

17:24:16.021 [fatal] Segfault detected!
17:24:16.022 [fatal] Hard exit due to segmentation fault! Backtrace:
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    0: DAnalysis::DSourceComboTimeHandler::Fill_Histograms()
17:24:16.022 [fatal]       hd_root:6076f0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    1: DAnalysis::DSourceComboer::~DSourceComboer()
17:24:16.022 [fatal]       hd_root:4127a7
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    2: DAnalysisResults_factory::~DAnalysisResults_factory()
17:24:16.022 [fatal]       hd_root:431132
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    3: DAnalysisResults_factory::~DAnalysisResults_factory()
17:24:16.022 [fatal]       hd_root:431799
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    4: JFactorySet::~JFactorySet()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9997
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    5: JFactorySet::~JFactorySet()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9a82
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    6: JEvent::~JEvent()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eb6f5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    7: void __gnu_cxx::new_allocator::destroy(JEvent*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d749
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    8: void std::allocator_traits >::destroy(std::allocator&, JEvent*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d715
17:24:16.022 [fatal] 
17:24:16.022 [fatal]    9: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d603
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   10: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   11: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   12: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4850
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   13: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4892
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   14: void std::_Destroy >(std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22b6d5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   15: void std::_Destroy_aux::__destroy*>(std::shared_ptr*, std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22a4c0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   16: void std::_Destroy*>(std::shared_ptr*, std::shared_ptr*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:228836
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   17: void std::_Destroy*, std::shared_ptr >(std::shared_ptr*, std::shared_ptr*, std::allocator >&)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:226b4d
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   18: std::vector, std::allocator > >::~vector()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:21fa07
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   19: JEventPool::~JEventPool()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d526
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   20: JEventPool::~JEventPool()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d54e
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   21: JTopologyBuilder::~JTopologyBuilder()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:218519
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   22: void __gnu_cxx::new_allocator::destroy(JTopologyBuilder*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eace1
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   23: void std::allocator_traits >::destroy(std::allocator&, JTopologyBuilder*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e9861
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   24: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e883d
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   25: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   26: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   27: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d580c
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   28: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d5828
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   29: jana::components::JComponent::Service::~Service()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d585a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   30: JExecutionEngine::~JExecutionEngine()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20bde3
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   31: JExecutionEngine::~JExecutionEngine()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20be0a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   32: std::default_delete::operator()(JExecutionEngine*) const
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd23e
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   33: std::_Sp_counted_deleter, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e8f76
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   34: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
17:24:16.022 [fatal]       hd_root:3c811a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   35: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   36: std::__shared_ptr::~__shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9e90
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   37: std::shared_ptr::~shared_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9ed2
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   38: std::pair >::~pair()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6818
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   39: void __gnu_cxx::new_allocator > > >::destroy > >(std::pair >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6838
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   40: void std::allocator_traits > > > >::destroy > >(std::allocator > > >&, std::pair >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e5459
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   41: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_destroy_node(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e3609
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   42: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_drop_node(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e18c9
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   43: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_erase(std::_Rb_tree_node > >*)
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dfca5
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   44: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::~_Rb_tree()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd068
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   45: std::map, std::less, std::allocator > > >::~map()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9098
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   46: JServiceLocator::~JServiceLocator()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcc7a
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   47: std::default_delete::operator()(JServiceLocator*) const
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcca0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   48: std::unique_ptr >::~unique_ptr()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d8dc4
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   49: JApplication::~JApplication()
17:24:16.022 [fatal]       /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d2524
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   50: main
17:24:16.022 [fatal]       hd_root:3a4505
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   51: ???
17:24:16.022 [fatal]       /lib64/libc.so.6:295d0
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   52: __libc_start_main
17:24:16.022 [fatal]       /lib64/libc.so.6:29680
17:24:16.022 [fatal] 
17:24:16.022 [fatal]   53: _start
17:24:16.022 [fatal]       hd_root:3b1c85
17:24:16.022 [fatal]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sdobbs