Describe the issue

onnxruntime_global_thread_pools_test and onnxruntime_shared_lib_test crash during tear-down in 32 bit Windows builds.

The crash is an internal Microsoft debug check Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call. This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

This happens in the std::invoke that std::thread does, but does not happen until main deletes the Ort::Env object in onnxruntime_global_thread_pools_test and in CApiTest.TestPerSessionCustomThreadPoolHooks in onnxruntime_shared_lib_test.

My initial idea was that there is a timing issue when the DLL is unloaded, but the thread pool seems to be only inside the dll so hopefully the DLL is not unloaded at this time as there are lots of threads running in it. Unfortunately I'm not knowledgeable in enough in this code base to give must more info than this call stack, which is for the main thread and shows that it is trying to destrot a thread pool. One of the worker threads is experienced the error, but its callstack is not useful.

onnxruntime_shared_lib_test.exe!std::thread::join() Line 130	C++
onnxruntime_shared_lib_test.exe!TestPerSessionCustomThreadHooks::JoinThreadCustomized(const OrtCustomHandleType * handle) Line 2051	C++
onnxruntime.dll!onnxruntime::`anonymous namespace'::WindowsThread::~WindowsThread() Line 125	C++
onnxruntime.dll!onnxruntime::`anonymous namespace'::WindowsThread::`scalar deleting destructor'(unsigned int)	C++
onnxruntime.dll!std::default_delete<onnxruntime::EnvThread>::operator()(onnxruntime::EnvThread * _Ptr) Line 3141	C++
onnxruntime.dll!std::unique_ptr<onnxruntime::EnvThread,std::default_delete<onnxruntime::EnvThread>>::reset(onnxruntime::EnvThread * _Ptr) Line 3289	C++
onnxruntime.dll!onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>::SignalAllAndWait() Line 693	C++
onnxruntime.dll!onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>::~ThreadPoolTempl<onnxruntime::Env>() Line 783	C++
onnxruntime.dll!onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>::`scalar deleting destructor'(unsigned int)	C++
onnxruntime.dll!std::default_delete<onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>>::operator()(onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env> * _Ptr) Line 3141	C++
onnxruntime.dll!std::unique_ptr<onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>,std::default_delete<onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>>>::~unique_ptr<onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>,std::default_delete<onnxruntime::concurrency::ThreadPoolTempl<onnxruntime::Env>>>() Line 3254	C++
onnxruntime.dll!onnxruntime::concurrency::ThreadPool::~ThreadPool() Line 402	C++
onnxruntime.dll!onnxruntime::concurrency::ThreadPool::`scalar deleting destructor'(unsigned int)	C++
onnxruntime.dll!std::default_delete<onnxruntime::concurrency::ThreadPool>::operator()(onnxruntime::concurrency::ThreadPool * _Ptr) Line 3141	C++
onnxruntime.dll!std::unique_ptr<onnxruntime::concurrency::ThreadPool,std::default_delete<onnxruntime::concurrency::ThreadPool>>::~unique_ptr<onnxruntime::concurrency::ThreadPool,std::default_delete<onnxruntime::concurrency::ThreadPool>>() Line 3254	C++
onnxruntime.dll!onnxruntime::InferenceSession::~InferenceSession() Line 467	C++
onnxruntime.dll!onnxruntime::InferenceSession::`scalar deleting destructor'(unsigned int)	C++
onnxruntime.dll!OrtApis::ReleaseSession(OrtSession * value) Line 2638	C++

onnxruntime_shared_lib_test.exe!Ort::OrtRelease(OrtSession * ptr) Line 109 C++
onnxruntime_shared_lib_test.exe!Ort::Base::~Base() Line 208 C++
onnxruntime_shared_lib_test.exe!Ort::Session::~Session() C++
onnxruntime_shared_lib_test.exe!TestPerSessionCustomThreadHooks::CApiTest_TestPerSessionCustomThreadPoolHooks_Test::TestBody() Line 2067 C++
onnxruntime_shared_lib_test.exe!testing::internal::HandleSehExceptionsInMethodIfSupportedtesting::Test,void(testing::Test * object, void(testing::Test::)() method, const char * location) Line 2592 C++
onnxruntime_shared_lib_test.exe!testing::internal::HandleExceptionsInMethodIfSupportedtesting::Test,void(testing::Test * object, void(testing::Test::)() method, const char * location) Line 2643 C++
onnxruntime_shared_lib_test.exe!testing::Test::Run() Line 2682 C++
onnxruntime_shared_lib_test.exe!testing::TestInfo::Run() Line 2864 C++
onnxruntime_shared_lib_test.exe!testing::TestSuite::Run() Line 3016 C++
onnxruntime_shared_lib_test.exe!testing::internal::UnitTestImpl::RunAllTests() Line 5856 C++
onnxruntime_shared_lib_test.exe!testing::internal::HandleSehExceptionsInMethodIfSupportedtesting::internal::UnitTestImpl,bool(testing::internal::UnitTestImpl * object, bool(testing::internal::UnitTestImpl::)() method, const char * location) Line 2592 C++
onnxruntime_shared_lib_test.exe!testing::internal::HandleExceptionsInMethodIfSupportedtesting::internal::UnitTestImpl,bool(testing::internal::UnitTestImpl * object, bool(testing::internal::UnitTestImpl::)() method, const char * location) Line 2643 C++
onnxruntime_shared_lib_test.exe!testing::UnitTest::Run() Line 5438 C++
onnxruntime_shared_lib_test.exe!RUN_ALL_TESTS() Line 2491 C++
onnxruntime_shared_lib_test.exe!main(int argc, char * * argv) Line 45 C++

Urgency

No response

Target platform

Windows 10/11

Build script

N/A

Error / output

Some test programs crash.

Visual Studio Version

VS 2022 17.3.6

GCC / Compiler Version

No response