ocean_data_gateway.readers.axds.region¶

class ocean_data_gateway.readers.axds.region(kwargs)[source]¶

Bases: ocean_data_gateway.readers.axds.AxdsReader

Inherits from AxdsReader to search over a region of space and time.

kw¶

Contains space and time search constraints: min_lon, max_lon, min_lat, max_lat, min_time, max_time.

Type: dict

variables¶

Variable names if you want to limit the search to those. There is different behavior depending on axds_type:

‘platform2’: the variable name or names must be from the list available in all_variables() and pass the check in check_variables().
‘layer_group’: the variable name or names will be searched for as a query so just do your best with the names and experiment.

Type: string or list

approach¶

approach is defined as ‘region’ for this class.

Type: string

Attributes

catalog: Write then open the catalog.
dataset_ids: Find dataset_ids for server.
meta: Rearrange the individual metadata into a dataframe.
search_results: Loop over self.urls to read in search results.
urls: Return a list of search urls.

Methods

`all_variables`()	Return a DataFrame of allowed variable names.
`check_variables`(variables[, verbose])	Checks variables for presence in database list.
`data`()	Read in data for all dataset_ids.
`data_by_dataset`(dataset_id)	Return the data for a single dataset_id.
`meta_by_dataset`(dataset_id)	Return the catalog metadata for a single dataset_id.
`save`()	Save datasets locally.
`search_variables`(variables)	Find valid variables names to use.
`url_builder`(url_base[, dataset_id, …])	Build an individual search url.
`url_dataset_id`(dataset_id)	url modification to search for known dataset_id.
`url_query`(query)	url modification to add query field.
`url_region`()	url modification to add spatial search box.
`url_time`()	url modification to add time filtering.
`url_variable`(variable)	url modification to add variable search.
`write_catalog`()	Write catalog file.
`write_catalog_layer_group_entry`(dataset, …)	Write part of catalog in case of layer_group.

__init__(kwargs)[source]¶

Parameters

kwargs (dict) –

Can contain arguments to pass onto the base AxdsReader class (catalog_name, parallel, axds_type). The dict entries to initialize this class are:

kw: dict Contains space and time search constraints: min_lon, max_lon, min_lat, max_lat, min_time, max_time.
variables: string or list, optional Variable names if you want to limit the search to those. There is different behavior depending on axds_type:
- ’platform2’: the variable name or names must be from the list available in all_variables() and pass the check in check_variables().
- ’layer_group’: the variable name or names will be searched for as a query so just do your best with the names and experiment.

Methods

`__init__`(kwargs)	param kwargs Can contain arguments to pass onto the base AxdsReader class
`all_variables`()	Return a DataFrame of allowed variable names.
`check_variables`(variables[, verbose])	Checks variables for presence in database list.
`data`()	Read in data for all dataset_ids.
`data_by_dataset`(dataset_id)	Return the data for a single dataset_id.
`meta_by_dataset`(dataset_id)	Return the catalog metadata for a single dataset_id.
`save`()	Save datasets locally.
`search_variables`(variables)	Find valid variables names to use.
`url_builder`(url_base[, dataset_id, …])	Build an individual search url.
`url_dataset_id`(dataset_id)	url modification to search for known dataset_id.
`url_query`(query)	url modification to add query field.
`url_region`()	url modification to add spatial search box.
`url_time`()	url modification to add time filtering.
`url_variable`(variable)	url modification to add variable search.
`write_catalog`()	Write catalog file.
`write_catalog_layer_group_entry`(dataset, …)	Write part of catalog in case of layer_group.

Attributes

`catalog`	Write then open the catalog.
`dataset_ids`	Find dataset_ids for server.
`meta`	Rearrange the individual metadata into a dataframe.
`search_results`	Loop over self.urls to read in search results.
`urls`	Return a list of search urls.

all_variables()¶

Return a DataFrame of allowed variable names.

Returns
Return type: DataFrame of variable names and count of how many times they are present in the database.

Notes

This list is only relevant for self.axds_type==’platform2’. It is not relevant for `self.axds_type==’layer_group’.

Example

>>> import ocean_data_gateway as odg
>>> odg.axds.AxdsReader().all_variables()
                                                 count
variable
Ammonium                                            23
Atmospheric Pressure: Air Pressure at Sea Level    362
Atmospheric Pressure: Barometric Pressure         4152
Backscatter Intensity                              286
Battery                                           2705
...                                                ...
Winds: Samples                                       1
Winds: Speed and Direction                        7091
Winds: Vertical Wind                                 4
Winds: at 10 m                                      18
pH                                                 965

property catalog¶: Write then open the catalog.

check_variables(variables, verbose=False)¶

Checks variables for presence in database list.

Parameters

variables (string, list) – String or list of strings to compare against list of valid variable names.
verbose (boolean, optional) – Print message if variables are matches instead of passing silently.

Returns

Nothing is returned. However, there are two types of behavior
if variables is not a valid variable name(s), an AssertionError is raised and search_variables(variables) is run on your behalf to suggest valid variable names to use.
if variables is a valid variable name(s), nothing happens.

Notes

This list is only relevant for self.axds_type==’platform2’. It is not relevant for `self.axds_type==’layer_group’.

Examples

Check if the variable name ‘sal’ is valid:

>>> odg.axds.AxdsReader().check_variables('sal')
AssertionError                            Traceback (most recent call last)
<ipython-input-11-454838d2e555> in <module>
----> 1 odg.axds.AxdsReader().check_variables('sal')
~/projects/ocean_data_gateway/ocean_data_gateway/readers/axds.py in check_variables(self, variables, verbose)
    878         CO2: PPM of Carbon Dioxide in Air in Dry Gas            1
    879         Evaporation Rate                                        1
--> 880         """
    881
    882         if not isinstance(variables, list):
AssertionError: The input variables are not exact matches to parameter groups.
Check all parameter group values with `AxdsReader().all_variables()`
or search parameter group values with `AxdsReader().search_variables(['sal'])`.
 Try some of the following variables:
               count
variable
Salinity        3204
Soil Salinity    622

Check if the variable name ‘Salinity’ is valid:

>>>  odg.axds.AxdsReader().check_variables('Salinity')

data()¶

Read in data for all dataset_ids.

Returns

A dictionary with keys of the dataset_ids and values the data of type
If `self.axds_type==’platform2’` (a pandas DataFrame)
If `self.axds_type==’layer_group’` (an xarray Dataset)

Notes

This is either done in parallel with the multiprocessing library or in serial.

data_by_dataset(dataset_id)¶

Return the data for a single dataset_id.

Returns

A tuple of (dataset_id, data), where data type depends on self.axds_type
If `self.axds_type==’platform2’` (a pandas DataFrame)
If `self.axds_type==’layer_group’` (an xarray Dataset)

Notes

Read behavior depends on axds_type:

If self.axds_type==’platform2’: data is read into memory with dask.
If self.axds_type==’layer_group’: data is pointed to with dask but nothing is read in except metadata associated with the xarray Dataset.

property dataset_ids¶

Find dataset_ids for server.

Notes

The dataset_ids are read from the catalog, so the catalog is created before this can happen, unless the dataset_ids were input from the beginning of the call via stations in which case they are simply saved to self._dataset_ids.

property meta¶: Rearrange the individual metadata into a dataframe.

meta_by_dataset(dataset_id)¶

Return the catalog metadata for a single dataset_id.

TO DO: Should this return intake-style or a row of the metadata dataframe?

save()¶: Save datasets locally.

property search_results¶

Loop over self.urls to read in search results.

Notes

The logic removes duplicate searches. This returns a dict of the datasets from the search results with the key of each entry being the dataset_id. For

self.axds_type == “platform2”: dataset_id is the uuid
self.axds_type == “layer_group”: dataset_id is the module_uuid since multiple layer_groups can be linked under one module_uuid

search_variables(variables)¶

Find valid variables names to use.

Parameters: variables (string, list) – String or list of strings to use in regex search to find valid variable names.
Returns
Return type: DataFrame of variable names and count of how many times they are present in the database, sorted by count.

Notes

This list is only relevant for self.axds_type==’platform2’. It is not relevant for `self.axds_type==’layer_group’.

Examples

Search for variables that contain the substring ‘sal’:

>>> odg.axds.AxdsReader().search_variables('sal')
               count
variable
Salinity        3204
Soil Salinity    622

Return all available variables, sorted by count (or could use all_variables() directly):

>>>  odg.axds.AxdsReader().search_variables('')
                                                    count
variable
Stream Height                                       19758
Water Surface above Datum                           19489
Stream Flow                                         15203
Temperature: Air Temperature                         8369
Precipitation                                        7364
...                                                   ...
Vent Fluid Temperature                                  1
Vent Fluid Thermocouple Temperature - Low               1
CO2: PPM of Carbon Dioxide in Sea Water in Wet Gas      1
CO2: PPM of Carbon Dioxide in Air in Dry Gas            1
Evaporation Rate                                        1

url_builder(url_base, dataset_id=None, add_region=False, add_time=False, variable=None, query=None)¶

Build an individual search url.

Parameters

url_base (string) – There are 2 possible bases for the url: * self.url_axds_type, for searching * self.url_docs_base, for selecting known dataset by dataset_id
dataset_id (string, optional) – dataset_id of station, if known.
add_region (boolean, optional) – True to filter the search by lon/lat box. Requires self.kw that contains keys min_lon, max_lon, min_lat, max_lat.
add_time (boolean, optional) – True to filter the search by time range. Requires self.kw that contains keys min_time and max_time.
variable (string, optional) – String of variable description to filter by, if desired. If axds_type==’platform2’, find the variable name options with class function all_variables(), search for variable names by string with search_variables(), and check your variable list with check_variables(). If axds_type==’layer_group’, there is no official variable list and you can instead just put in a basic variable name and hope the search works.
query (string, optional) – This could be any search query you want, but it is used in the code to search for station names (not dataset_ids).

Returns

Return type

Url for search.

url_dataset_id(dataset_id)¶

url modification to search for known dataset_id.

Parameters: dataset_id (string) – String of dataset_id to exactly match.
Returns
Return type: Modification for url to search for dataset_id.

url_query(query)¶

url modification to add query field.

Parameters: query (string) – String to query for. Can be multiple words.
Returns
Return type: Modification for url to add query field.

url_region()¶

url modification to add spatial search box.

Returns
Return type: Modification for url to add lon/lat filtering.

Notes

Uses the kw dictionary already stored in the class object to access the spatial limits of the box.

url_time()¶

url modification to add time filtering.

Returns
Return type: Modification for url to add time filtering.

Notes

Uses the kw dictionary already stored in the class object to access the time limits of the search.

url_variable(variable)¶

url modification to add variable search.

Parameters: variable (string) – String to search for.
Returns
Return type: Modification for url to add variable search.

Notes

This variable search is specifically by parameter group and only works for axds_type=’platform2’. For axds_type=’layer_group’, use url_query with the variable name.

property urls¶

Return a list of search urls.

Notes

Use this through the class methods region or stations to put together the search urls to represent the basic reader setup.

write_catalog()¶: Write catalog file.

write_catalog_layer_group_entry(dataset, dataset_id, urlpath, layer_groups)¶

Write part of catalog in case of layer_group.

Notes

This is used to manage the logic for axds_type=’layer_group’ in which the module is being linked to the set of layer_groups.