From 7e2e22f9d67cabc619b4141ef756ac6b655b3193 Mon Sep 17 00:00:00 2001 From: koennecke Date: Thu, 23 Jan 2020 09:37:16 +0100 Subject: [PATCH 01/11] Improved documentation about time stamped data --- manual/source/design.rst | 3 +++ manual/source/introduction.rst | 13 ++++++++----- manual/source/rules.rst | 16 ++++++++++++++++ manual/source/strategies.rst | 21 ++++++++++++++++----- 4 files changed, 43 insertions(+), 10 deletions(-) diff --git a/manual/source/design.rst b/manual/source/design.rst index 53985f5564..b2b5b08dc6 100644 --- a/manual/source/design.rst +++ b/manual/source/design.rst @@ -121,6 +121,9 @@ an array of several dimensions. ``data`` (*NX_NUMBER*) Data values from the detector, ``units="NX_ANY"`` +In the case of streaming data acquisition, when time stamped values of data are collected, fields can be replaced with :ref:`NXlog` structures of +the same name. For example, if time stamped data for wavelength is being streamed, wavelength would not be an array but a :ref:`NXlog` structure. + .. index:: ! single: field attribute diff --git a/manual/source/introduction.rst b/manual/source/introduction.rst index 781e84bbe1..793663500e 100644 --- a/manual/source/introduction.rst +++ b/manual/source/introduction.rst @@ -14,18 +14,21 @@ to define a common data exchange format for neutron, X-ray, and muon experiments NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files in addition to a dictionary of well-defined domain-specific field names. The NeXus -data format has two purposes: +data format has three purposes: #. *raw data*: NeXus defines a format that can serve as a container for all relevant data associated with a scientific - instrument or beamline. - This is a very important use case. + instrument or beamline. This is a very important use case. This includes + the case of streaming data acquisition, where time stamped data are logged. #. *processed data*: + NeXus also defines standards for processed data. This is data which has underwent some form of data + reduction or data analysis. NeXus allows to store the results of such processing together with + documentation about how the processed data was generated. +#. *standards*: NeXus defines standards in the form of *application definitions* for the exchange of data - between applications. NeXus provides structures for raw experimental - data as well as for processed data. + between applications. NeXus provides standards for both raw and processed data. A community of scientists and computer programmers working in neutron and synchrotron facilities around the world came to the conclusion that a diff --git a/manual/source/rules.rst b/manual/source/rules.rst index cef2254f30..7cb4cc764a 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -368,6 +368,22 @@ dimensions ``xraster, yraster, xsize, ysize``. .. warning:: Be warned: if you use the 2D rasterisation method with ``xraster, yraster`` you may end up with invalid data if the scan is aborted prematurely. This cannot happen if the first method is used. + +Streaming Data Acquisition And Logging +====================================== + +More and more data is collected in streaming mode. This means that time stamped data is logged for one or more inputs, +possibly together with detector data. Another use case is the logging of parameters, for example temperature, while a long +running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files: + +Just use the standard NeXus raw data file structure for such data. But replace the streamed or logged data element +in its place in the hierarchy through a :ref:`NXlog` or a :ref:`NXevent_data` structure of the same name. + +For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both +NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays. The NXlog structure will +have the same name as the NeXus field name. + + NXcollection ============ diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 80c6a8b6e1..0a2f5daf13 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -149,10 +149,17 @@ which also allows "energy" to be specified if one is so inclined. Strategies: Time-stamped data ############################# *How should I store time-stamped data?* -Time-stamped data can stored in both :ref:`NXlog` and :ref:`NXevent_data`. -:ref:`NXevent_data` is used for storing neutron event data and :ref:`NXlog` would be used -for storing any other time-stamped data, e.g. sample temperature, chopper -top-dead-centre, motor position etc. + +Time-stamped data can be stored in both :ref:`NXlog` and :ref:`NXevent_data` structures. +Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is used only for storing neutron event data +and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, +motor position, detector data etc. + +Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but replace +the fields for streamed data elements through :ref:`NXlog` or :ref:`NXevent_data` structures. For example, consider the +collection of detector images against a change in the magnetic field on the sample. Then, both NXsample/magnetic_field and +NXdetector/data would be :ref:`NXlog` structures containing the time stamped data. + Both :ref:`NXlog` and :ref:`NXevent_data` have additional support for storing time-stamped data in the form of cues; cues can be used to place markers in the data that allow one to @@ -174,7 +181,11 @@ to grab the data between 20 seconds and 40 seconds, and then trim that data to g want. Obviously in this simple example this does not gain us a lot, but it is easy to see that in a large dataset having appropriately placed cues can save significant computational time when looking -up values in a certain time-stamp range. +up values in a certain time-stamp range. NeXus has actually borrowed the cueing table concept +from video file formats where it allows viewing software to quickly access your most favourite scene. +Correspondingly, cueing in NeXus allows you to quickly access your most favourate morsel of time stamped +data. + In the NeXus Features repository, the feature `ECB064453EDB096D `_ shows example code that uses cues to select time-stamped data. From af85397e050c296ae6040b0ec1ed384cf014f798 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:16:50 +0100 Subject: [PATCH 02/11] Update manual/source/strategies.rst --- manual/source/strategies.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 0a2f5daf13..f6020273ba 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -183,7 +183,7 @@ Obviously in this simple example this does not gain us a lot, but it is easy to large dataset having appropriately placed cues can save significant computational time when looking up values in a certain time-stamp range. NeXus has actually borrowed the cueing table concept from video file formats where it allows viewing software to quickly access your most favourite scene. -Correspondingly, cueing in NeXus allows you to quickly access your most favourate morsel of time stamped +Correspondingly, cueing in NeXus allows you to quickly access your favourite morsel of time stamped data. From ae09b8d45490d844366b9d0c5a1df377827e2bf3 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:17:58 +0100 Subject: [PATCH 03/11] Update manual/source/strategies.rst --- manual/source/strategies.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index f6020273ba..15634cea6d 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -182,7 +182,7 @@ want. Obviously in this simple example this does not gain us a lot, but it is easy to see that in a large dataset having appropriately placed cues can save significant computational time when looking up values in a certain time-stamp range. NeXus has actually borrowed the cueing table concept -from video file formats where it allows viewing software to quickly access your most favourite scene. +from video file formats where it allows viewing software to quickly access your favourite scene. Correspondingly, cueing in NeXus allows you to quickly access your favourite morsel of time stamped data. From b5972c1a63037c78e2fad84a887bf14625af7f36 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:21:16 +0100 Subject: [PATCH 04/11] Update manual/source/strategies.rst --- manual/source/strategies.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 15634cea6d..0a29a891e7 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -155,7 +155,7 @@ Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is used and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, motor position, detector data etc. -Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but replace +Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but insert the fields for streamed data elements through :ref:`NXlog` or :ref:`NXevent_data` structures. For example, consider the collection of detector images against a change in the magnetic field on the sample. Then, both NXsample/magnetic_field and NXdetector/data would be :ref:`NXlog` structures containing the time stamped data. From 0ecd81a676436f0b0640e80cefde20aceb91c390 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:21:26 +0100 Subject: [PATCH 05/11] Update manual/source/strategies.rst --- manual/source/strategies.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 0a29a891e7..220853629c 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -150,7 +150,7 @@ Strategies: Time-stamped data ############################# *How should I store time-stamped data?* -Time-stamped data can be stored in both :ref:`NXlog` and :ref:`NXevent_data` structures. +Time-stamped data can be stored in either :ref:`NXlog` and :ref:`NXevent_data` structures. Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is used only for storing neutron event data and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, motor position, detector data etc. From ed2365f3f7d3e653411935bc7a8bac7486f6bb53 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:21:37 +0100 Subject: [PATCH 06/11] Update manual/source/rules.rst --- manual/source/rules.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/rules.rst b/manual/source/rules.rst index 7cb4cc764a..c4befd1fa8 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -377,7 +377,7 @@ possibly together with detector data. Another use case is the logging of paramet running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files: Just use the standard NeXus raw data file structure for such data. But replace the streamed or logged data element -in its place in the hierarchy through a :ref:`NXlog` or a :ref:`NXevent_data` structure of the same name. +with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays. The NXlog structure will From d5cf7376ceb429b66eda7a4e4e58e9b7e98d5ec2 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:21:50 +0100 Subject: [PATCH 07/11] Update manual/source/rules.rst --- manual/source/rules.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/rules.rst b/manual/source/rules.rst index c4befd1fa8..74d2963afe 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -376,7 +376,7 @@ More and more data is collected in streaming mode. This means that time stamped possibly together with detector data. Another use case is the logging of parameters, for example temperature, while a long running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files: -Just use the standard NeXus raw data file structure for such data. But replace the streamed or logged data element +Just use the standard NeXus raw data file structure for such data. But replace the standard data object with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both From 7e4182eb9f27fc2d1c94ee8e5a9ce8e647c15b08 Mon Sep 17 00:00:00 2001 From: Ben Watts Date: Thu, 23 Jan 2020 21:31:08 +0100 Subject: [PATCH 08/11] language improvement --- manual/source/rules.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/rules.rst b/manual/source/rules.rst index 74d2963afe..9f85a14f3c 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -376,7 +376,7 @@ More and more data is collected in streaming mode. This means that time stamped possibly together with detector data. Another use case is the logging of parameters, for example temperature, while a long running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files: -Just use the standard NeXus raw data file structure for such data. But replace the standard data object +Just use the standard NeXus raw data file structure, but replace the corresponding data object with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both From e274131d408b2898d92fe5a8f0971012627f902e Mon Sep 17 00:00:00 2001 From: Freddie Akeroyd Date: Thu, 23 Jan 2020 20:37:44 +0000 Subject: [PATCH 09/11] Update introduction.rst minor edit --- manual/source/introduction.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manual/source/introduction.rst b/manual/source/introduction.rst index 793663500e..0ede69fdf8 100644 --- a/manual/source/introduction.rst +++ b/manual/source/introduction.rst @@ -23,7 +23,7 @@ data format has three purposes: the case of streaming data acquisition, where time stamped data are logged. #. *processed data*: NeXus also defines standards for processed data. This is data which has underwent some form of data - reduction or data analysis. NeXus allows to store the results of such processing together with + reduction or data analysis. NeXus allows storing the results of such processing together with documentation about how the processed data was generated. #. *standards*: NeXus defines standards in From bf4322ae8a62888e94b4356fcc2e66f7260d4b0e Mon Sep 17 00:00:00 2001 From: Freddie Akeroyd Date: Thu, 23 Jan 2020 20:40:23 +0000 Subject: [PATCH 10/11] Update rules.rst minor edit --- manual/source/rules.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/manual/source/rules.rst b/manual/source/rules.rst index 9f85a14f3c..e3456d8fb7 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -380,8 +380,8 @@ Just use the standard NeXus raw data file structure, but replace the correspondi with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both -NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays. The NXlog structure will -have the same name as the NeXus field name. +NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays i.e. the NXlog structure will +have the same name as the NeXus field involved. NXcollection From e7e4193bc4c18ca0c43380580cf7ade2b0ec7ae1 Mon Sep 17 00:00:00 2001 From: Freddie Akeroyd Date: Thu, 23 Jan 2020 20:45:42 +0000 Subject: [PATCH 11/11] Update strategies.rst minor edits --- manual/source/strategies.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 220853629c..30150aa953 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -151,11 +151,12 @@ Strategies: Time-stamped data *How should I store time-stamped data?* Time-stamped data can be stored in either :ref:`NXlog` and :ref:`NXevent_data` structures. -Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is used only for storing neutron event data +Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is normally only used for storing detector +time of flight event data and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, -motor position, detector data etc. +motor position, detector images etc. -Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but insert +Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but insert/replace the fields for streamed data elements through :ref:`NXlog` or :ref:`NXevent_data` structures. For example, consider the collection of detector images against a change in the magnetic field on the sample. Then, both NXsample/magnetic_field and NXdetector/data would be :ref:`NXlog` structures containing the time stamped data.