Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset-specific JSON/TOML configuration #1646

Open
wants to merge 24 commits into
base: dev
Choose a base branch
from
Open

Dataset-specific JSON/TOML configuration #1646

wants to merge 24 commits into from

Conversation

franzpoeschel
Copy link
Contributor

@franzpoeschel franzpoeschel commented Jul 10, 2024 •
edited
Loading

This was initially prototyped in PIConGPU, search for "extended format for JSON options" on this documentation page.

This moves that functionality over into openPMD.
Look at the diff on examples/13_write_dynamic_configuration.cpp for an example use case.
Bringing this to openPMD makes this functionality available in codes other than PIConGPU, and makes it available from TOML as well as from JSON.

TODO:

@franzpoeschel franzpoeschel added the api: new additions to the API label Jul 10, 2024
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 3d5b70a to 37d57ab Compare July 10, 2024 11:23
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 10, 2024
View reviewed changes
src/auxiliary/JSONMatcher.cpp
and is interpreted as a regular expression against which the dataset name
(full path or path within an iteration) must match.)END";
auto throw_up = [&](std::string const &additional_info,
auto &&...additional_path) {

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable additional_path is not used.
src/auxiliary/JSONMatcher.cpp
and is interpreted as a regular expression against which the dataset name
(full path or path within an iteration) must match.)END";
auto throw_up = [&](std::string const &additional_info,
auto &&...additional_path) {

Check notice

Code scanning / CodeQL

Unused static variable Note

Static variable additional_path is never read.
src/auxiliary/JSONMatcher.cpp
and is interpreted as a regular expression against which the dataset name
(full path or path within an iteration) must match.)END";
auto throw_up = [&](std::string const &additional_info,
auto &&...additional_path) {

Check notice

Code scanning / CodeQL

Declaration hides parameter Note

Local variable 'additional_path' hides a
parameter of the same name
Loading
.
src/auxiliary/JSONMatcher.cpp

namespace
{
auto readPattern(

Check warning

Code scanning / CodeQL

Poorly documented large function Warning

Poorly documented function: fewer than 2% comments for a function of 101 lines.
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 4 times, most recently from c95a953 to 68dc005 Compare July 16, 2024 11:12
github-advanced-security[bot]
github-advanced-security bot found potential problems Jul 16, 2024
View reviewed changes
src/auxiliary/JSON.cpp
std::string merge_impl(
std::string const &defaultValue,
std::string const &overwrite,
MPI_Comm_t &&...comm)

Check notice

Code scanning / CodeQL

Unused local variable Note

Variable comm is not used.
src/auxiliary/JSON.cpp
std::string merge_impl(
std::string const &defaultValue,
std::string const &overwrite,
MPI_Comm_t &&...comm)

Check notice

Code scanning / CodeQL

Unused static variable Note

Static variable comm is never read.
@ax3l ax3l modified the milestones: 0.16.0, 0.17.0 Jul 16, 2024
@ax3l ax3l self-requested a review July 16, 2024 17:23
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from c0f942e to fb7a28e Compare July 19, 2024 09:39
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch from fb7a28e to 5f77a13 Compare August 5, 2024 09:30
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch from 5f77a13 to 64b006d Compare October 1, 2024 13:38
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch from 64b006d to 8852258 Compare October 29, 2024 08:19
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch from 8852258 to a162e8d Compare November 15, 2024 15:01
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 7dda24d to 83e436c Compare December 17, 2024 11:19
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from bdc319f to 0650789 Compare February 12, 2025 13:17
@franzpoeschel franzpoeschel mentioned this pull request Feb 13, 2025
Merged
7 tasks
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch 2 times, most recently from 6b2b5ee to 71536e6 Compare February 18, 2025 18:09
@ax3l ax3l self-assigned this Feb 18, 2025
@franzpoeschel
Seems to work
310c91a
@franzpoeschel
Adapt Coretest to new output of myPath()
2feb7f5
@franzpoeschel
Better error messages and some documentation inside example
016eb52
@franzpoeschel
Adapt constructors for installation without ADIOS2/HDF5
7fbeaad
@franzpoeschel
CI fixes
a7c6c14
@franzpoeschel
Basic implementation
36b74e7
@franzpoeschel
Update documentation and tests
8f7e9e3
@franzpoeschel
Add a JSON translation of the config for NVHPC compilers
bab84df 
might also be good for documentation purposes as JSON is more widely
known
@franzpoeschel
Use dataset-specific config in tests
21c64b1
@franzpoeschel
Fix: do_prune parameter for merge()
5a6d060
@franzpoeschel
Rename merge() -> merge_internal()
7d2d6af 
Having the same name as the public function provoked errors due to
conversion from nlohmann::json types.
@franzpoeschel
Don't compute the matchers for all backends
d6a09b2
@franzpoeschel
Add default block to test configs
7534159
@franzpoeschel
Documentation
9019783
@franzpoeschel
Add TOML example
b2ebe1d
@franzpoeschel
Add Python binding for openPMD_path
41de428
@franzpoeschel
Fix doxygen
9beff67
@franzpoeschel
Read dataset-specific configuration also in ADIOS2::openDataset
1cfea7b
@franzpoeschel
Cleanup
9607c83
@franzpoeschel
Fix initialization from Dummy IO Handler
48ec4e3
@franzpoeschel
Fix Doxygen
5e87f47
@franzpoeschel franzpoeschel force-pushed the dataset-specific-config branch from 71536e6 to 5e87f47 Compare February 20, 2025 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ax3l ax3l Awaiting requested review from ax3l

Assignees

@ax3l ax3l

Labels
api: new additions to the API
Projects
None yet
Milestone
0.17.0
Development

Successfully merging this pull request may close these issues.
2 participants
@franzpoeschel @ax3l