ncdata.threadlock_sharing module#

Support for “unifying” the thread-safety mechanisms between ncdata and other packages.

Each of the data-format packages (ncdata, iris and xarray) uses its own locking mechanism to prevent overlapping calls into the netcdf library when called by multi-threaded code.

Most commonly, this occurs when netcdf file data is read to compute a Dask array, or written in a Dask delayed write operation.

All 3 data-format packages (ncdata, Iris and xarray) can map variable data into Dask lazy arrays on file load. Iris and Xarray can also create delayed write operations (but ncdata currently does not).

However, those mechanisms cannot protect an operation of that package from overlapping with one in another package.

This module can ensure that all of the enabled packages use the same thread lock, so that any and all of them can safely co-operate in parallel operations.

sample usages:

from ncdata.threadlock_sharing import enable_lockshare, disable_lockshare
from ncdata.xarray import from_xarray
from ncdata.iris import from_iris, to_iris
from ncdata.netcdf4 import to_nc4, from_nc4

enable_lockshare(iris=True, xarray=True)

ds = from_xarray(xarray.open_dataset(file1))
ds2 = from_iris(iris.load(file2))
ds.variables['x'].data /= ds2.variables['acell'].data
to_nc4(ds, output_filepath)

disable_lockshare()

or:

with lockshare_context(iris=True):
    ncdata = from_nc4(source_filepath)
    my_adjust_process(ncdata)
    data_cube = to_iris(ncdata).extract("main_var")
    grid_cube = iris.load_cube(grid_path, "grid_cube")
    result_cube = data_cube.regrid(grid_cube)
    iris.save(result_cube, output_filepath)

Warning

The solution in this module is at present still experimental, and not itself thread-safe. So probably can only be applied at the outer level of an operation.

ncdata.threadlock_sharing.enable_lockshare(iris=False, xarray=False)#

Begin lock-sharing between ncdata and the requested other package(s).

Does nothing if an existing sharing is already in place.

Parameters:

  • iris (bool, default False) – make ncdata use the same netcdf lock as iris

  • xarray (bool, default False) – make ncdata use the same netcdf lock as xarray

Notes

If an enable_lockshare call was already established, the function does nothing, i.e. it is not possible to modify an existing share. Instead, you must call disable_lockshare() to cancel the current sharing, before you can establish a new one.

While sharing with both iris and xarray, iris is modified to use the same netcdf lock as xarray.

ncdata.threadlock_sharing.disable_lockshare()#

Remove any enabled lock-sharing.

Does nothing if no lock share is in operation.

ncdata.threadlock_sharing.lockshare_context(iris=False, xarray=False)#

Make a context with lock-sharing between the ncdata and the requested packages.

This allows safe netcdf access when using a combination of ncdata/iris/xarray packages.

Parameters: