Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JP2OpenJPEG: add support for generating files with PLT marker segments and multithreaded encoding #2436

Merged
merged 6 commits into from
Oct 9, 2020
Merged

JP2OpenJPEG: add support for generating files with PLT marker segments and multithreaded encoding #2436

merged 6 commits into from
Oct 9, 2020

Conversation

rouault
Copy link
Member

@rouault rouault commented Apr 23, 2020

Available with OpenJPEG > 2.3.1

Funded by ESA for S2-MPC project

@rouault
Copy link
Member Author

rouault commented Apr 23, 2020

CC @savmickael This is ready for your review

Locally, with openjpeg installed from the branch of uclouvain/openjpeg#1246, and after patching the version number in opj_config.h to be 2.3.2:

$ pytest ../autotest/gdrivers/jp2openjpeg.py::test_jp2openjpeg_generate_PLT
[...]
collecting ... 
 gdrivers/jp2openjpeg.py ✓                                                                                                                                                                           100% ██████████

Results (0.18s):
       1 passed

@rouault rouault force-pushed the openjpeg_plt branch 2 times, most recently from 5df25ae to c562100 Compare May 4, 2020 18:13
@rouault rouault changed the title JP2OpenJPEG: add support for generating files with PLT marker segments JP2OpenJPEG: add support for generating files with PLT marker segments and multithreaded encoder May 4, 2020
@rouault rouault changed the title JP2OpenJPEG: add support for generating files with PLT marker segments and multithreaded encoder JP2OpenJPEG: add support for generating files with PLT marker segments and multithreaded encoding May 4, 2020
@stale
Copy link

stale bot commented May 25, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label May 25, 2020
@rouault
Copy link
Member Author

rouault commented May 25, 2020

ping @savmickael

@stale stale bot removed the stale label May 25, 2020
@rouault rouault force-pushed the openjpeg_plt branch 2 times, most recently from 1cb7f95 to 55fc1dd Compare May 25, 2020 22:26
@stale
Copy link

stale bot commented Jun 15, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label Jun 15, 2020
@rouault
Copy link
Member Author

rouault commented Jun 15, 2020

keep alive

@stale stale bot removed the stale label Jun 15, 2020
@stale
Copy link

stale bot commented Jul 6, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label Jul 6, 2020
@rouault
Copy link
Member Author

rouault commented Jul 7, 2020

keep alive

@stale stale bot removed the stale label Jul 7, 2020
@stale
Copy link

stale bot commented Jul 29, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label Jul 29, 2020
@rouault
Copy link
Member Author

rouault commented Jul 29, 2020

keep alive

@stale stale bot removed the stale label Jul 29, 2020
@stale
Copy link

stale bot commented Aug 22, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label Aug 22, 2020
@rouault
Copy link
Member Author

rouault commented Aug 23, 2020

keep alive

@stale stale bot removed the stale label Aug 23, 2020
@stale
Copy link

stale bot commented Sep 13, 2020

The GDAL project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 21 days and is being automatically marked as "stale". If you think this pull request should be merged, please check - that all unit tests are passing - that all comments by reviewers have been addressed - that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes
  • add a description of workflows which this pull request fixes
  • add screenshots if applicable
  • that you have written unit tests where possible In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request. If there is no further activity on this pull request, it will be closed in a week.

@stale stale bot added the stale label Sep 13, 2020
@rouault
Copy link
Member Author

rouault commented Sep 13, 2020

keep alive

@stale stale bot removed the stale label Sep 13, 2020
rouault added 6 commits October 1, 2020 19:19
@rouault
JP2OpenJPEG: add support for generating files with PLT marker segments
5f3ecba 
Available with OpenJPEG > 2.3.1

Funded by ESA for S2-MPC project
@rouault
jp2openjpeg.py: cleanup
433947d
@rouault
Doc: document PLT creation option of JP2OpenJPEG driver
8a4afef
@rouault
JP2OpenJPEG: code cleanup: use IS_OPENJPEG_OR_LATER() macro
adf1cb3
@rouault
OpenJPEG: add support to enable multi-threaded encoded
415865b 
with openjpeg > 2.3.1

$ time gdal_translate T36JTT_20160914T074612_B02.tif out.jp2 -of JP2OPENJPEG --config GDAL_NUM_THREADS 4

real	0m15.717s
user	0m35.864s
sys	0m0.288s

$ time gdal_translate T36JTT_20160914T074612_B02.tif out.jp2 -of JP2OPENJPEG --config GDAL_NUM_THREADS 1

real	0m33.224s
user	0m33.332s
sys	0m0.336s
@rouault
OpenJPEG writer: acquire input data in background thread
442b177 
On a input TIFF, 10980x10980 with 256x256 tiles, DEFLATE compressed:

// 4 compressing threads, 1 background input data acquisition thread
$ time gdal_translate T36JTT_20160914T074612_B02.tif  out.jp2 -of JP2OPENJPEG --config GDAL_NUM_THREADS 4
real	0m11.193s
user	0m41.448s
sys	0m0.252s

// 4 compressing threads, no background input data acquisition
$ time gdal_translate T36JTT_20160914T074612_B02.tif  out.jp2 -of JP2OPENJPEG --config GDAL_NUM_THREADS 4 --config JP2OPENJPEG_USE_THREADED_IO NO
real	0m18.338s
user	0m39.620s
sys	0m0.224s

// No multithreading at all
$ time gdal_translate T36JTT_20160914T074612_B02.tif  out.jp2 -of JP2OPENJPEG --config GDAL_NUM_THREADS 1 --config JP2OPENJPEG_USE_THREADED_IO NO
real	0m35.510s
user	0m35.704s
sys	0m0.228s
@rouault rouault added this to the 3.2.0 milestone Oct 9, 2020
@rouault rouault merged commit 5b59b82 into OSGeo:master Oct 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
3.2.0
Development

Successfully merging this pull request may close these issues.
1 participant
@rouault