Repository navigation
Conversation
|
Plausibly there's a second half missing, where the top-level restore of threadpoolctl, the one where it relinquishes control, should always be to the original level of the process, because when it grabs the original level it might grab the wrong one as a race condition. |
|
Or, another way of saying it, if thread A stores values to restore to at a time when thread B is the controlling thread, the restore value is wrong, and should instead be "all cores" (zero? or a level stored at startup). Will do that or equivalent tomorrow. |
|
The above is a real race condition (I have implemented a test and fix) but it is still not enough to fix the joblib race condition reproducer! |
|
Will continue trying to figure out the joblib race condition next week. |
|
@ogrisel ok I think this is ready for review. |
This comment was marked as resolved.
This comment was marked as resolved.
ogrisel
left a comment
There was a problem hiding this comment.
Some more feedback, this time about API design concerns.
|
The reproducer seems like basically the known limitation, that using the imperative API doesn't participate in automatic process? Last time it came up we decided to just document it. And it's why joblib has to increment/decrement manually itself. |
|
Yeah sorry, I forgot about that. Let's ignore this part of my review. The second part on the public API could be fixed though. What do you think? |
|
Yep, on it. |
|
Note that CI was failing because |
|
@ogrisel ok back to you |
ogrisel
left a comment
There was a problem hiding this comment.
Thanks @itamarst for the follow-up. The fix itself LGTM, an maybe is the correct one for a 3.8.1 bug fix release.
However, I have the feeling that our API design makes OpenBLASController leak its peculiarities in too many places:
- in the
_set_threadpool_limitsmethod (more details below); - in joblib itself (notably in
_get_threadpoolctl_limits) (see https://github.com/joblib/joblib/pull/1871/changes#r4524197578 in particular).
I think this is a sign we should consider exposing ownership control semantics more explicitly in controllers. WDYT?
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
975865d to
11eaef5
Compare
|
On the one hand it is getting more complicated, yeah. On the other hand this only makes a difference when:
Which isn't that common outside of scikit-learn, probably? And then add in the fact that half the time OpenBLAS parallelism slows things down and plausibly a lot of more sophisticated performance-sensitive users will just set the limit to 1 and call it a day (or use MKL if they have complex parallelism). |
|
Switching to Probably because with FlexiBLAS you actually set limits twice, once directly on OpenBLAS and once indirectly with FlexiBLAS and they now have conflicting semantics. So will think about it and hopefully fix that today. |
|
I made the failing test more constrained, so it should hopefully pass even on FlexiBLAS, and just left FlexiBLAS in its previous state of 🤷. |
|
My basic thought is, to summarize: none of this is perfect but it's probably better than what we have now, and the edge cases (which are real!) probably don't affect that many people. And edge cases are probably no worse than the status quo before 3.8.0. |
|
I added an extra commit with a proposed new API to address #307 / review comments in corresponding joblib PR. |
cakedev0
left a comment
There was a problem hiding this comment.
Thanks for the PR! Claude has two findings:
The first one is the snapshot race with two OpenBLAS copies loaded, see my inline comments. I didn't looked in details at the proposed fix, but the modified test that fails convinces me it's indeed worth fixing.
The second one is the bug of #299 in a sibling code path: restore_original_limits also restores the libraries that this limiter never set. A thread that only limits OpenMP (or passes limits=None) still holds an OpenBLAS snapshot, and on exit it takes control to restore it:
from threading import Event, Thread
import numpy # noqa: F401
from threadpoolctl import ThreadpoolController, threadpool_limits
lib = ThreadpoolController().select(internal_api="openblas").lib_controllers[0]
print(lib.get_num_threads()) # 14
a_in, b_in, a_out = Event(), Event(), Event()
def A():
with threadpool_limits(2):
a_in.set()
b_in.wait()
a_out.set()
def B():
a_in.wait()
with threadpool_limits(1, user_api="openmp"): # or threadpool_limits()
b_in.set()
a_out.wait()
threads = [Thread(target=A), Thread(target=B)]
for t in threads:
t.start()
for t in threads:
t.join()
print(lib.get_num_threads()) # 2It behaves the same on master, but I think the fix belongs here since it is the same bug. Keeping in _original_info only the libraries that _set_threadpool_limits did set (dropping the entry on the continue path and when num_threads is None) should cover it. WDYT?
| for t in ts: | ||
| t.join() | ||
|
|
||
| assert lib.get_num_threads() == orig_limit |
There was a problem hiding this comment.
This test only checks libs[0]. With SciPy imported there are two OpenBLAS copies, and the second one is not restored. Checking all of them:
libs, lib, orig_limit = _require_resizable_non_openmp_openblas()
request.addfinalizer(_restore_openblas_thread_counts(libs))
+ orig_limits = [lib.get_num_threads() for lib in libs]| assert lib.get_num_threads() == orig_limit | |
| assert [lib.get_num_threads() for lib in libs] == orig_limits |
gives on this branch:
E assert [14, 2] == [14, 14]
| elif isinstance(result, _THIS_THREAD_TOOK_CONTROL): | ||
| self._original_info[lib_controller]["num_threads"] = ( | ||
| result.previous_num_threads | ||
| ) |
There was a problem hiding this comment.
Only the OpenBLAS copy that takes control gets its snapshot fixed here. With NumPy and SciPy both loaded, the second copy goes through the controlling_thread is this_thread branch of _set_num_threads_with_extra_info, returns None, and keeps the snapshot read while the other thread was in control.
Something like this fixes it for me (the stricter test from my other comment passes, and so does the rest of the test suite):
+ took_control = False
for lib_controller in self._controller.lib_controllers:
...
if num_threads is not None:
+ if took_control and type(lib_controller) is OpenBLASController:
+ # The snapshot may have been read while another thread
+ # was in control: read it again now that we are.
+ self._original_info[lib_controller]["num_threads"] = (
+ lib_controller.get_num_threads()
+ )
if type(lib_controller) is OpenBLASController:
...
elif isinstance(result, _THIS_THREAD_TOOK_CONTROL):
self._original_info[lib_controller]["num_threads"] = (
result.previous_num_threads
)
+ took_control = True
Fixes #299
Fixes #307