diff --git a/manual/source/design.rst b/manual/source/design.rst index 53985f5564..b2b5b08dc6 100644 --- a/manual/source/design.rst +++ b/manual/source/design.rst @@ -121,6 +121,9 @@ an array of several dimensions. ``data`` (*NX_NUMBER*) Data values from the detector, ``units="NX_ANY"`` +In the case of streaming data acquisition, when time stamped values of data are collected, fields can be replaced with :ref:`NXlog` structures of +the same name. For example, if time stamped data for wavelength is being streamed, wavelength would not be an array but a :ref:`NXlog` structure. + .. index:: ! single: field attribute diff --git a/manual/source/introduction.rst b/manual/source/introduction.rst index 781e84bbe1..0ede69fdf8 100644 --- a/manual/source/introduction.rst +++ b/manual/source/introduction.rst @@ -14,18 +14,21 @@ to define a common data exchange format for neutron, X-ray, and muon experiments NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files in addition to a dictionary of well-defined domain-specific field names. The NeXus -data format has two purposes: +data format has three purposes: #. *raw data*: NeXus defines a format that can serve as a container for all relevant data associated with a scientific - instrument or beamline. - This is a very important use case. + instrument or beamline. This is a very important use case. This includes + the case of streaming data acquisition, where time stamped data are logged. #. *processed data*: + NeXus also defines standards for processed data. This is data which has underwent some form of data + reduction or data analysis. NeXus allows storing the results of such processing together with + documentation about how the processed data was generated. +#. *standards*: NeXus defines standards in the form of *application definitions* for the exchange of data - between applications. NeXus provides structures for raw experimental - data as well as for processed data. + between applications. NeXus provides standards for both raw and processed data. A community of scientists and computer programmers working in neutron and synchrotron facilities around the world came to the conclusion that a diff --git a/manual/source/rules.rst b/manual/source/rules.rst index cef2254f30..e3456d8fb7 100644 --- a/manual/source/rules.rst +++ b/manual/source/rules.rst @@ -368,6 +368,22 @@ dimensions ``xraster, yraster, xsize, ysize``. .. warning:: Be warned: if you use the 2D rasterisation method with ``xraster, yraster`` you may end up with invalid data if the scan is aborted prematurely. This cannot happen if the first method is used. + +Streaming Data Acquisition And Logging +====================================== + +More and more data is collected in streaming mode. This means that time stamped data is logged for one or more inputs, +possibly together with detector data. Another use case is the logging of parameters, for example temperature, while a long +running data collection is in progress. NeXus covers this case too. There is one simple rule for structuring such files: + +Just use the standard NeXus raw data file structure, but replace the corresponding data object +with an :ref:`NXlog` or :ref:`NXevent_data` structure of the same name. + +For example, consider your instrument is streaming detector images against a magnetic_field on the sample. In this case both +NXsample/magnetic_field and NXdetector/data would become NXlog structures instead of simple arrays i.e. the NXlog structure will +have the same name as the NeXus field involved. + + NXcollection ============ diff --git a/manual/source/strategies.rst b/manual/source/strategies.rst index 80c6a8b6e1..30150aa953 100644 --- a/manual/source/strategies.rst +++ b/manual/source/strategies.rst @@ -149,10 +149,18 @@ which also allows "energy" to be specified if one is so inclined. Strategies: Time-stamped data ############################# *How should I store time-stamped data?* -Time-stamped data can stored in both :ref:`NXlog` and :ref:`NXevent_data`. -:ref:`NXevent_data` is used for storing neutron event data and :ref:`NXlog` would be used -for storing any other time-stamped data, e.g. sample temperature, chopper -top-dead-centre, motor position etc. + +Time-stamped data can be stored in either :ref:`NXlog` and :ref:`NXevent_data` structures. +Of the two, :ref:`NXlog` is the most important one, :ref:`NXevent_data` is normally only used for storing detector +time of flight event data +and :ref:`NXlog` would be used for storing any other time-stamped data, e.g. sample temperature, chopper top-dead-centre, +motor position, detector images etc. + +Regarding the NeXus file structure to use, there is one simple rule: just use the standard NeXus file structure but insert/replace +the fields for streamed data elements through :ref:`NXlog` or :ref:`NXevent_data` structures. For example, consider the +collection of detector images against a change in the magnetic field on the sample. Then, both NXsample/magnetic_field and +NXdetector/data would be :ref:`NXlog` structures containing the time stamped data. + Both :ref:`NXlog` and :ref:`NXevent_data` have additional support for storing time-stamped data in the form of cues; cues can be used to place markers in the data that allow one to @@ -174,7 +182,11 @@ to grab the data between 20 seconds and 40 seconds, and then trim that data to g want. Obviously in this simple example this does not gain us a lot, but it is easy to see that in a large dataset having appropriately placed cues can save significant computational time when looking -up values in a certain time-stamp range. +up values in a certain time-stamp range. NeXus has actually borrowed the cueing table concept +from video file formats where it allows viewing software to quickly access your favourite scene. +Correspondingly, cueing in NeXus allows you to quickly access your favourite morsel of time stamped +data. + In the NeXus Features repository, the feature `ECB064453EDB096D `_ shows example code that uses cues to select time-stamped data.