Skip to content

Commit 57fafe0

Browse files
VeckoTheGeckoVeckoTheGecko
authored
Merge pull request #1765 from OceanParcels/docs/metadata
DOC: particlefile metadata
2 parents d013cd3 + 7d6a143 commit 57fafe0

File tree

2 files changed

+64
-49
lines changed

2 files changed

+64
-49
lines changed

‎docs/examples/tutorial_output.ipynb‎

Lines changed: 63 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
"- [**Plotting**](#Plotting)\n",
2222
"- [**Animations**](#Animations)\n",
2323
"\n",
24-
"First we need to create some parcels output to analyze. We simulate a set of particles using the setup described in the [Delay start tutorial](https://docs.oceanparcels.org/en/latest/examples/tutorial_delaystart.html).\n"
24+
"First we need to create some parcels output to analyze. We simulate a set of particles using the setup described in the [Delay start tutorial](https://docs.oceanparcels.org/en/latest/examples/tutorial_delaystart.html). We will also add some user defined metadata to the output file.\n"
2525
]
2626
},
2727
{
@@ -30,7 +30,7 @@
3030
"metadata": {},
3131
"outputs": [],
3232
"source": [
33-
"from datetime import timedelta\n",
33+
"from datetime import datetime, timedelta\n",
3434
"\n",
3535
"import numpy as np\n",
3636
"\n",
@@ -41,16 +41,7 @@
4141
"cell_type": "code",
4242
"execution_count": 2,
4343
"metadata": {},
44-
"outputs": [
45-
{
46-
"name": "stdout",
47-
"output_type": "stream",
48-
"text": [
49-
"INFO: Output files are stored in Output.zarr.\n",
50-
"100%|██████████| 86400.0/86400.0 [00:00<00:00, 93792.23it/s] \n"
51-
]
52-
}
53-
],
44+
"outputs": [],
5445
"source": [
5546
"example_dataset_folder = parcels.download_example_dataset(\"Peninsula_data\")\n",
5647
"fieldset = parcels.FieldSet.from_parcels(\n",
@@ -68,8 +59,40 @@
6859
" fieldset=fieldset, pclass=parcels.JITParticle, lon=lon, lat=lat, time=time\n",
6960
")\n",
7061
"\n",
71-
"output_file = pset.ParticleFile(name=\"Output.zarr\", outputdt=timedelta(hours=2))\n",
72-
"\n",
62+
"output_file = pset.ParticleFile(name=\"Output.zarr\", outputdt=timedelta(hours=2))"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {},
68+
"source": [
69+
"Parcels saves some metadata in the output file with every simulation (Parcels version, CF convention information, etc.). This metadata is just a dictionary which is propogated to `xr.Dataset(attrs=...)` and is stored in the `.metadata` attribute. The user is free to manipulate this dictionary to add any custom, xarray-compatible metadata relevant to their simulation. Here we add a custom metadata field `date_created` to the output file."
70+
]
71+
},
72+
{
73+
"cell_type": "code",
74+
"execution_count": 3,
75+
"metadata": {},
76+
"outputs": [],
77+
"source": [
78+
"output_file.metadata[\"date_created\"] = datetime.now().isoformat()"
79+
]
80+
},
81+
{
82+
"cell_type": "code",
83+
"execution_count": 4,
84+
"metadata": {},
85+
"outputs": [
86+
{
87+
"name": "stdout",
88+
"output_type": "stream",
89+
"text": [
90+
"INFO: Output files are stored in Output.zarr.\n",
91+
"100%|██████████| 86400.0/86400.0 [00:01<00:00, 82356.55it/s]\n"
92+
]
93+
}
94+
],
95+
"source": [
7396
"pset.execute(\n",
7497
" parcels.AdvectionRK4,\n",
7598
" runtime=timedelta(hours=24),\n",
@@ -85,35 +108,36 @@
85108
"source": [
86109
"## Reading the output file\n",
87110
"\n",
88-
"Parcels exports output trajectories in `zarr` [format](https://zarr.readthedocs.io/en/stable/). Files in `zarr` are typically _much_ smaller in size than netcdf, although may be slightly more challenging to handle (but `xarray` has a fairly seamless `open_zarr()` method).\n"
111+
"Parcels exports output trajectories in `zarr` [format](https://zarr.readthedocs.io/en/stable/). Files in `zarr` are typically _much_ smaller in size than netcdf, although may be slightly more challenging to handle (but `xarray` has a fairly seamless `open_zarr()` method). Note when when we display the dataset we cam see our custom metadata field `date_created`.\n"
89112
]
90113
},
91114
{
92115
"cell_type": "code",
93-
"execution_count": 3,
116+
"execution_count": 5,
94117
"metadata": {},
95118
"outputs": [
96119
{
97120
"name": "stdout",
98121
"output_type": "stream",
99122
"text": [
100-
"<xarray.Dataset>\n",
123+
"<xarray.Dataset> Size: 3kB\n",
101124
"Dimensions: (trajectory: 10, obs: 12)\n",
102125
"Coordinates:\n",
103-
" * obs (obs) int32 0 1 2 3 4 5 6 7 8 9 10 11\n",
104-
" * trajectory (trajectory) int64 0 1 2 3 4 5 6 7 8 9\n",
126+
" * obs (obs) int32 48B 0 1 2 3 4 5 6 7 8 9 10 11\n",
127+
" * trajectory (trajectory) int64 80B 0 1 2 3 4 5 6 7 8 9\n",
105128
"Data variables:\n",
106-
" lat (trajectory, obs) float32 dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
107-
" lon (trajectory, obs) float32 dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
108-
" time (trajectory, obs) timedelta64[ns] dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
109-
" z (trajectory, obs) float32 dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
129+
" lat (trajectory, obs) float32 480B dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
130+
" lon (trajectory, obs) float32 480B dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
131+
" time (trajectory, obs) timedelta64[ns] 960B dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
132+
" z (trajectory, obs) float32 480B dask.array<chunksize=(1, 1), meta=np.ndarray>\n",
110133
"Attributes:\n",
111134
" Conventions: CF-1.6/CF-1.7\n",
135+
" date_created: 2024-11-20T11:07:47.494911\n",
112136
" feature_type: trajectory\n",
113137
" ncei_template_version: NCEI_NetCDF_Trajectory_Template_v2.0\n",
114138
" parcels_kernels: JITParticleAdvectionRK4\n",
115139
" parcels_mesh: flat\n",
116-
" parcels_version: v2.4.2-367-gd9b7b447\n"
140+
" parcels_version: 3.1.1.dev4\n"
117141
]
118142
}
119143
],
@@ -126,17 +150,17 @@
126150
},
127151
{
128152
"cell_type": "code",
129-
"execution_count": 4,
153+
"execution_count": 6,
130154
"metadata": {},
131155
"outputs": [
132156
{
133157
"name": "stdout",
134158
"output_type": "stream",
135159
"text": [
136-
"<xarray.DataArray 'trajectory' (trajectory: 10)>\n",
160+
"<xarray.DataArray 'trajectory' (trajectory: 10)> Size: 80B\n",
137161
"array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
138162
"Coordinates:\n",
139-
" * trajectory (trajectory) int64 0 1 2 3 4 5 6 7 8 9\n"
163+
" * trajectory (trajectory) int64 80B 0 1 2 3 4 5 6 7 8 9\n"
140164
]
141165
}
142166
],
@@ -168,7 +192,7 @@
168192
},
169193
{
170194
"cell_type": "code",
171-
"execution_count": 5,
195+
"execution_count": 7,
172196
"metadata": {},
173197
"outputs": [
174198
{
@@ -217,7 +241,7 @@
217241
},
218242
{
219243
"cell_type": "code",
220-
"execution_count": 6,
244+
"execution_count": 8,
221245
"metadata": {},
222246
"outputs": [],
223247
"source": [
@@ -237,7 +261,7 @@
237261
},
238262
{
239263
"cell_type": "code",
240-
"execution_count": 7,
264+
"execution_count": 9,
241265
"metadata": {},
242266
"outputs": [
243267
{
@@ -276,7 +300,7 @@
276300
},
277301
{
278302
"cell_type": "code",
279-
"execution_count": 8,
303+
"execution_count": 10,
280304
"metadata": {
281305
"scrolled": true
282306
},
@@ -312,18 +336,9 @@
312336
},
313337
{
314338
"cell_type": "code",
315-
"execution_count": 9,
339+
"execution_count": 11,
316340
"metadata": {},
317-
"outputs": [
318-
{
319-
"name": "stderr",
320-
"output_type": "stream",
321-
"text": [
322-
"/Users/erik/miniconda3/envs/parcels/lib/python3.11/site-packages/pandas/core/arrays/timedeltas.py:1093: RuntimeWarning: invalid value encountered in cast\n",
323-
" data = (base * m + (frac * m).astype(np.int64)).view(\"timedelta64[ns]\")\n"
324-
]
325-
}
326-
],
341+
"outputs": [],
327342
"source": [
328343
"# Using xarray\n",
329344
"mean_lon_x = []\n",
@@ -349,7 +364,7 @@
349364
},
350365
{
351366
"cell_type": "code",
352-
"execution_count": 10,
367+
"execution_count": 12,
353368
"metadata": {},
354369
"outputs": [
355370
{
@@ -399,7 +414,7 @@
399414
},
400415
{
401416
"cell_type": "code",
402-
"execution_count": 11,
417+
"execution_count": 13,
403418
"metadata": {},
404419
"outputs": [
405420
{
@@ -454,7 +469,7 @@
454469
},
455470
{
456471
"cell_type": "code",
457-
"execution_count": 12,
472+
"execution_count": 14,
458473
"metadata": {},
459474
"outputs": [],
460475
"source": [
@@ -473,7 +488,7 @@
473488
},
474489
{
475490
"cell_type": "code",
476-
"execution_count": 13,
491+
"execution_count": 15,
477492
"metadata": {},
478493
"outputs": [],
479494
"source": [
@@ -513,7 +528,7 @@
513528
},
514529
{
515530
"cell_type": "code",
516-
"execution_count": 14,
531+
"execution_count": 16,
517532
"metadata": {},
518533
"outputs": [
519534
{
@@ -5945,7 +5960,7 @@
59455960
"<IPython.core.display.HTML object>"
59465961
]
59475962
},
5948-
"execution_count": 14,
5963+
"execution_count": 16,
59495964
"metadata": {},
59505965
"output_type": "execute_result"
59515966
}

‎docs/examples/tutorial_parcels_structure.ipynb‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -350,7 +350,7 @@
350350
"cell_type": "markdown",
351351
"metadata": {},
352352
"source": [
353-
"The final part executes the simulation, given the `ParticleSet`, `FieldSet` and `Kernels`, that have been defined in the previous steps. If you like to store the particle data generated in the simulation, you define the `ParticleFile` to which the output of the kernel execution will be written. Then, on the `ParticleSet` you have defined, you can use the method `ParticleSet.execute()` which requires the following arguments:\n",
353+
"The final part executes the simulation, given the `ParticleSet`, `FieldSet` and `Kernels`, that have been defined in the previous steps. If you like to store the particle data generated in the simulation, you define the `ParticleFile` to which the output of the kernel execution as well as - optionally - any user-specified metadata (see the [Working with Parcels output](https://docs.oceanparcels.org/en/latest/examples/tutorial_output.html) tutorial for more info) will be written. Then, on the `ParticleSet` you have defined, you can use the method `ParticleSet.execute()` which requires the following arguments:\n",
354354
"\n",
355355
"1. The kernels to be executed.\n",
356356
"2. The `runtime` defining how long the execution loop runs. Alternatively, you may define the `endtime` at which the execution loop stops.\n",

0 commit comments

Comments
 (0)