Loading and storing IMAS data¶

IMAS data is grouped together in Data Entries. A Data Entry is a collection of IDSs and their (potentially) multiple occurrences, which groups and stores data over multiple IDSs as a single dataset. The Data Entry concept is used whether the collection of IDSs is stored in a database or only exists temporarily (for example for communication in an integrated workflow).

Loading and storing IMAS data happens through an IMAS Database Entry. A Database Entry tracks the information required for locating where the Data Entry is (or will be) stored on disk. In MATLAB this object is modeled as an integer index.

You may Open an existing IMAS Database Entry, which you can use for loading data that was stored previously. Alternatively you can Create a new IMAS Database Entry to store IDS data.

Open an existing IMAS Database Entry¶

To open an IMAS Database Entry, you need to know the URI indicating where the Access Layer can find the data. IMAS URIs start with imas: and indicate the format and the location of the stored data. You can find a detailed description of the IMAS URI syntax on the IMAS Data Entry URIs page.

% Open the database entry by providing an IMAS URI
data_entry = imas_open( ...
    'imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3')

% Alternatively, we can open an IDS providing legacy arguments
data_entry2 = imas_open_env('ids', 131024, 41, 'public', 'ITER', '3')

Loading IMAS data¶

After you open a database entry, you can request to load data from disk.

Load an entire IDS¶

With ids_get() you can load (“get”) an entire IDS from the database entry.

Multiple occurrences of an IDS may be stored in a data entry. By default, if you don’t specify an occurrence number, occurrence 0 is loaded. By providing an occurrence number you can load a specific occurrence. How different occurrences are used depends on the experiment. They could, for example, correspond to:

• different methods for computing the physical quantities of the IDS, or

• different functionalities in a workflow (e.g. initial values, prescribed values, values at next time step, …), or

• multiple subsystems (e.g. diagnostics) of the same type in an experiment, etc.

% Open the database entry by providing an IMAS URI
data_entry = imas_open( ...
    'imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3')

% Load the core_profiles IDS, occurrence 0
cp = ids_get(data_entry, 'core_profiles')

% Load the core_profiles IDS, occurrence 1
cp1 = ids_get(data_entry, 'core_profiles/1')

Load a single time slice of an IDS¶

Instead of loading a full IDS from disk, the Access Layer allows you to load a specific time slice. This is often useful when you’re not interested in the full time evolution, but instead want data of a specific time. You can use ids_get_slice() for this.

Most of the time there are no entries at that specific time, so you also need to indicate an interpolation method. This determines what values the access layer returns when your requested time is in between available time points in the data. Three interpolation methods currently exist:

PREVIOUS_SAMPLE (2)

Returns the previous time slice if the requested time does not exactly exist in the original IDS.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t_r=2.1\) will give you the data at \(t=1\).

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r = t_i\) [1]	Return data at \(t_i\).

CLOSEST_SAMPLE (1)

Returns the closest time slice in the original IDS. This can also be after the requested time.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t=2.1\) will give you the data at \(t=3\).

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r = t_i\) [1]	Return data at \(t_i\).
\(t_r - t_i = t_{i+1} - t_r\) [1]	Return data at \(t_{i+1}\).

INTERPOLATION (3)

Returns a linear interpolation between the existing slices before and after the requested time.

For example, when data exists at \(t=\{1, 3, 4\}\), requesting \(t=2.1\) will give you a linear interpolation of the data at \(t=1\) and the data at \(t=3\).

Note that the linear interpolation will be successful only if, between the two time slices of an interpolated dynamic array of structure, the same leaves are populated and they have the same size. Otherwise ids_get_slice() will interpolate all fields with a compatible size and leave others empty.

Edge case behaviour. \(\{t_i\}, i=1..N\) represents the time series stored in the IDS.¶

Case	Behaviour
\(t_r \lt t_1\)	Return data at \(t_1\).
\(t_r \gt t_N\)	Return data at \(t_N\).

% Open the database entry by providing an IMAS URI
data_entry = imas_open( ...
    'imas:mdsplus?user=public;pulse=131024;run=41;database=ITER;version=3')

% Retrieve the time slice just before t=370 of the core_profiles IDS
cp = ids_get_slice(data_entry, 'core_profiles', 370, 2)

Create a new IMAS Database Entry¶

To create a new IMAS Database Entry, you need to provide the URI to indicate the format and the location where you want to store the data. You can find a detailed description of the IMAS URI syntax and the options available on the IMAS Data Entry URIs page.

% create the database entry by providing an IMAS URI
data_entry = imas_open('imas:hdf5?path=my-data', 43)

Store IMAS data¶

After you have created an IMAS Database Entry, you can use it for storing IDS data. There are two ways to do this:

Store an entire IDS¶

With ids_put() you can store (“put”) an entire IDS in a database entry. First you need to have an IDS with data: you can create a new one or load an IDS which you modify. See Use Interface Data Structures for more information on using and manipulating IDSs.

Multiple occurrences of an IDS may be stored in a data entry. By default, if you don’t specify an occurrence number, the IDS is stored as occurrence 0. By providing an occurrence number you can store the IDS as a specific occurrence.

% create the database entry by providing an IMAS URI
data_entry = imas_open('imas:hdf5?path=my-data', 43)

% Create an empty pf_active IDS
pf_active = ids_init('pf_active')
% Set the mandatory ids_properties.homogeneous_time field
pf_active.ids_properties.homogeneous_time = 1
% Continue filling the pf_active IDS here
% ...

% Store the pf_active IDS
ids_put(data_entry, 'pf_active', pf_active)

% Alternatively, store the pf_active IDS as occurrence 1
ids_put(data_entry, 'pf_active', 1, pf_active)

Append a time slice to an already-stored IDS¶

With ids_put_slice() you can append a time slice to an existing database entry. This is useful when you generate data inside a time loop (for example in simulations, or when taking measurements of an experiment).

It means you can put a time slice with every iteration of your loop such that you don’t have to keep track of the complete time evolution in memory. Instead, the Access Layer will keep appending the data to the Database Entry in the storage backend.

% create the database entry by providing an IMAS URI
data_entry = imas_open('imas:hdf5?path=my-data', 43)

% Create an empty core_profiles IDS
core_profiles = ids_init('core_profiles')

% Set the mandatory ids_properties.homogeneous_time field
core_profiles.ids_properties.homogeneous_time = 1
% Continue filling the core_profiles IDS with static values here
% ...

% Example time integration loop:
t = 0.0
t_stop = 60.0
dt = 0.1

while t < t_stop
    core_profiles.time = [t]
    % Fill your time-dependent data in the core_profiles IDS here
    % ...

    ids_put_slice(data_entry, 'core_profiles', core_profiles)

    t = t + dt
end

Listing all occurrences of an IDS from a backend¶

With imas_list_all_occurrences() you can List all non-empty occurrences of an IDS using its name in the dataset, and optionnally return the content of a descriptive node path.

% Opens an existing Data Entry object (dataset), using the hdf5 backend
idx = imas_open('imas:hdf5?path=my-data')

% Get all occurrences of the 'magnetics' IDS (integers) contained in the idx dataset 
% The argument 'ids_properties/comment' is optional; the function will retrieve the content of this path
% for each occurrence found in the idx dataset
[occurrence_list, node_content_list] = imas_list_all_occurrences(idx, 'magnetics', 'ids_properties/comment')

% Closing the dataset
imas_close(idx)