Web sources internal API

These modules are abstracted by kadlu.load, which should generally be used instead for fetching and loading from web sources. These docs are provided for contributors wishing to add new web data source modules that can be accessed by kadlu.load.

kadlu.load works by mapping strings to function calls in kadlu.geospatial.data_sources.source_map as a convenience by passing the source and variable name; new data source modules should have a corresponding entry within the fetch and load maps there.

A general pattern for adding fetch/load modules is that fetching is an implicit action of the load function. each fetch and load function should accept a set of boundary arguments equal to or subsetting the following function signature, such that keyword args can be passed as a dict by fetch_handler:

(south=-90, west=-180, north=90, east=180, top=0, bottom=5000, start=datetime(2000,1,1), end=datetime(2000,1,1,1))

Source Map

used for generating fetch requests for web data sources

data fetching utils, function maps, and constant variables collection of objects

class kadlu.geospatial.data_sources.source_map.SourceMap

Bases: object

default_val()

load_map()

source_map()

var3d()

ERA5

API for Era5 dataset from Copernicus Climate Datastore

Metadata regarding the dataset can be found here:

https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview

class kadlu.geospatial.data_sources.era5.Era5

Bases: object

collection of module functions for fetching and loading

load_flux_ocean(**kwargs)

load_flux_waves(**kwargs)

load_precip_type(**kwargs)

load_precipitation(**kwargs)

load_snowfall(**kwargs)

load_stress_ocean(**kwargs)

load_wavedirection(**kwargs)

load_waveperiod(**kwargs)

load_wind_u(**kwargs)

load_wind_uv(**kwargs)

load_wind_v(**kwargs)

load_windwaveswellheight(**kwargs)

kadlu.geospatial.data_sources.era5.era5_cfg(key=None, url=None)

kadlu.geospatial.data_sources.era5.fetch_era5(var, *, west, east, south, north, start, end, **_)

fetch global era5 data for specified variable and time range

args:

var: string

the variable short name of desired wave parameter according to ERA5 docs. the complete list can be found here (table 7 for wave params): https://confluence.ecmwf.int/display/CKB/ERA5+data+documentation#ERA5datadocumentation-Temporalfrequency

kwargs: dict

keyword arguments passed from the Era5() class as a dictionary

return:

True if new data was fetched, else False

kadlu.geospatial.data_sources.era5.initdb()

kadlu.geospatial.data_sources.era5.load_era5(var, *, west, east, south, north, start, end, **_)

load era5 data from local database

args:

var:

variable to be fetched (string)

kwargs:

dictionary containing the keyword arguments used for the fetch request. must contain south, north, west, east keys as float values, and start, end keys as datetimes

return:

values:

values of the fetched var

lat:

y grid coordinates

lon:

x grid coordinates

epoch:

timestamps in epoch hours since jan 1 2000

GEBCO

class kadlu.geospatial.data_sources.gebco.Gebco

Bases: object

fetch_bathymetry_grid()

download netcdf archive and extract it

fetch_callback(south, north, west, east, top=None, bottom=None, start=None, end=None)

build data grid indexes from .nc file and insert into database

load_bathymetry(south, north, west, east, **_)

load gebco bathymetry data

kadlu.geospatial.data_sources.gebco.initdb()

HYCOM

data source:

https://www.hycom.org/data/glbv1pt08

web interface for manual hycom data retrieval:

https://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015.html

class kadlu.geospatial.data_sources.hycom.Hycom

Bases: object

collection of module functions for fetching and loading

attributes:

lat, lon: arrays

spatial grid arrays. used to convert grid index to coordinate point

epoch: dict

dictionary containing yearly time indexes. times formatted as epoch hours since 2000-01-01 00:00

depth: array

depth scale. used to convert depth index to metres

callback(var, **kwargs)

build indices for query, fetch from hycom, insert into database

args:

var: string

variable to be fetched. complete list of variables here https://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015.html

kwargs: dict

boundaries as keyword arguments

fetch_hycom(var, kwargs)

convert user query to grid index slices, handle edge cases

load_hycom(var, kwargs)

load hycom data from local database

args:

var:

variable to be fetched. complete list of variables here https://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/2015.html

south, north: float

ymin, ymax coordinate values. range: -90, 90

kwargs: dict (boundaries as keyword arguments)

west, east: float

xmin, xmax coordinate values. range: -180, 180

start, end: datetime

temporal boundaries in datetime format

return:

values: array

values of the fetched variable

lat: array

y grid coordinates

lon: array

x grid coordinates

epoch: array

timestamps in epoch hours since jan 1 2000

depth: array

measured in meters

load_salinity(**kwargs)

load_temp(**kwargs)

load_water_u(**kwargs)

load_water_uv(**kwargs)

load_water_v(**kwargs)

kadlu.geospatial.data_sources.hycom.fetch_grid(**_)

download lat/lon/time/depth arrays for grid indexing. times are formatted as epoch hours since 2000-01-01 00:00

kadlu.geospatial.data_sources.hycom.initdb()

kadlu.geospatial.data_sources.hycom.load_grid()

put spatial grid into memory

kadlu.geospatial.data_sources.hycom.slices_str(var, slices, steps=(1, 1, 1, 1))

IFREMER

WWIII

Kadlu API for the NOAA WaveWatch III Datastore

User guides:

https://github.com/NOAA-EMC/WW3/wiki/WAVEWATCH-III-User-Guide

Data model description (boundary definitions, map visualizations, etc)

https://polar.ncep.noaa.gov/waves/implementations.php

class kadlu.geospatial.data_sources.wwiii.Wwiii

Bases: object

collection of module functions for fetching and loading

load_wavedirection(**kwargs)

load_waveperiod(**kwargs)

load_wind_u(**kwargs)

load_wind_uv(**kwargs)

load_wind_v(**kwargs)

load_windwaveheight(**kwargs)

kadlu.geospatial.data_sources.wwiii.fetch_wwiii(var, **kwargs)

download wwiii data and return associated filepaths

args:

var: string

the variable name of desired parameter according to WWIII docs the complete list of variables can be found at the following URL under ‘model output’ https://polar.ncep.noaa.gov/waves/implementations.php

south, north: float

ymin, ymax coordinate boundaries (latitude). range: -90, 90

west, east: float

xmin, xmax coordinate boundaries (longitude). range: -180, 180

start: datetime

the start of the desired time range

end: datetime

the end of the desired time range

return:

True if new data was fetched, else False

kadlu.geospatial.data_sources.wwiii.initdb()

kadlu.geospatial.data_sources.wwiii.insert(table, agg)

insert parsed data into local database

kadlu.geospatial.data_sources.wwiii.load_wwiii(var, kwargs)

return downloaded wwiii data for specified wavevar according to given time, lat, lon boundaries

args:

var: string

the variable short name of desired wave parameter according to WWIII docs the complete list of variable short names can be found here (under ‘model output’) https://polar.ncep.noaa.gov/waves/implementations.php

south, north: float

ymin, ymax coordinate boundaries (latitude). range: -90, 90

west, east: float

xmin, xmax coordinate boundaries (longitude). range: -180, 180

start: datetime

the start of the desired time range

end: datetime

the end of the desired time range

return:

val, lat, lon, epoch as np arrays of floats

Data Utils

additional tools and utils used across data loading modules

class kadlu.geospatial.data_sources.data_util.Boundary(south, north, west, east, fetchvar='', **_)

Bases: object

compute intersecting boundaries with separating axis theorem

intersects(other)

kadlu.geospatial.data_sources.data_util.database_cfg()

configure and connect to sqlite database

time is stored as an integer in the database, where each value is epoch hours since 2000-01-01 00:00

returns:

conn:

database connection object

db:

connection cursor object

kadlu.geospatial.data_sources.data_util.dt_2_epoch(dt_arr, t0=datetime.datetime(2000, 1, 1, 0, 0))

convert datetimes to epoch hours

kadlu.geospatial.data_sources.data_util.epoch_2_dt(ep_arr, t0=datetime.datetime(2000, 1, 1, 0, 0), unit='hours')

convert epoch hours to datetimes

kadlu.geospatial.data_sources.data_util.ext(filepath, extensions)

kadlu.geospatial.data_sources.data_util.flatten(cols, frames)

dimensional reduction by taking average of time frames

kadlu.geospatial.data_sources.data_util.fmt_coords(kwargs)

kadlu.geospatial.data_sources.data_util.index_arr(val, sorted_arr)

converts value in coordinate array to grid index

kadlu.geospatial.data_sources.data_util.ll_2_regionstr(south, north, west, east, regions, default=[])

convert input bounds to region strings with seperating axis theorem

kadlu.geospatial.data_sources.data_util.logmsg(source, var, ntup=(), **kwargs)

kadlu.geospatial.data_sources.data_util.logmsg_nodata(source, var, **kwargs)

kadlu.geospatial.data_sources.data_util.reshape_2D(cols)

kadlu.geospatial.data_sources.data_util.reshape_3D(cols)

prepare loaded data for interpolation

args:

cols: flattened numpy array of shape (4, n)

cols[0]: values cols[1]: latitude cols[2]: longitude cols[3]: depth

return: gridded

dict(values=gridspace, lats=ygrid, lons=xgrid, depths=zgrid)

class kadlu.geospatial.data_sources.data_util.storage_cfg(setdir=None)

Bases: PathLike

return filepath containing storage configuration string

first checks the config.ini file in kadlu root folder, then defaults to kadlu/storage

cfg = <configparser.ConfigParser object>

default_storage(msg)

helper function for storage_cfg()

kadlu.geospatial.data_sources.data_util.str_def(self, info, args)

builds string definition for data source class objects

kadlu.geospatial.data_sources.data_util.verbosity(set_verbosity=None)

__file__ = ‘/home/matt/kadlu/kadlu/geospatial/data_sources/data_util.py’