ncdata.threadlock_sharing module#
Support for “unifying” the thread-safety mechanisms between ncdata and other packages.
Each of the data-format packages (ncdata, iris and xarray) uses its own locking mechanism to prevent overlapping calls into the netcdf library when called by multi-threaded code.
Most commonly, this occurs when netcdf file data is read to compute a Dask array, or written in a Dask delayed write operation.
All 3 data-format packages can map variable data into Dask lazy arrays. Iris and Xarray can also create delayed write operations (but ncdata currently does not).
However, those mechanisms cannot protect an operation of that package from overlapping with one in another package.
This module can ensure that all of the enabled packages use the same thread lock, so that any and all of them can safely co-operate in parallel operations.
sample code:
from ncdata.threadlock_sharing import enable_lockshare, disable_lockshare
from ncdata.xarray import from_xarray
from ncdata.iris import from_iris
from ncdata.netcdf4 import to_nc4
enable_lockshare(iris=True, xarray=True)
ds = from_xarray(xarray.open_dataset(file1))
ds2 = from_iris(iris.load(file2))
ds.variables['x'].data /= ds2.variables['acell'].data
to_nc4(ds, output_filepath)
disable_lockshare()
or:
with lockshare_context(iris=True):
ncdata = NcData(source_filepath)
ncdata.variables['x'].attributes['units'] = 'K'
cubes = ncdata.iris.to_iris(ncdata)
iris.save(cubes, output_filepath)
Begin lock-sharing between ncdata and the requested other package(s).
Does nothing if an existing sharing is already in place.
- Parameters:
Notes
If an ‘enable_lockshare’ call was already established, the function does nothing, i.e. it is not possible to modify an existing share. Instead, you must call
disable_lockshare()to cancel the current sharing, before you can establish a new one.While sharing with both iris and xarray, iris is modified to use the same netcdf lock as xarray.
Remove any enabled lock-sharing.
Does nothing if no lock share is in operation.