[R] CRAN packaging checklist for version 19.0.1

[amoeba]

Packaging checklist for CRAN release

For a high-level overview of the release process see the
Apache Arrow Release Management Guide.

Before the release candidate is cut

• Create a GitHub issue entitled [R] CRAN packaging checklist for version X.X.X and copy this checklist to the issue.
• Review deprecated functions to advance their deprecation status, including removing preprocessor directives that no longer apply (search for ARROW_VERSION_MAJOR in r/src).
• Evaluate the status of any failing nightly tests and nightly packaging builds. These checks replicate most of the checks that CRAN runs, so we need them all to be passing or to understand that the failures may (though won't necessarily) result in a rejection from CRAN.
• Check current CRAN check results
• Ensure the contents of the README are accurate and up to date.
• Run urlchecker::url_check() on the R directory at the release candidate.
  commit. Ignore any errors with badges as they will be removed in the CRAN release branch.
• Polish NEWS but do not update version numbers (this is done automatically later). You can find commits by, for example, git log --oneline <sha of last release>..HEAD | grep "\[R\]"
• For major releases, prepare tweet thread highlighting new features.

Wait for the release candidate to be cut:

After release candidate has been cut

• Create a CRAN-release branch from the release candidate commit, name the new branch e.g. maint-X.X.X-r and push to upstream

Prepare and check the .tar.gz that will be released to CRAN.

• git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d
• Run make build. This copies Arrow C++ into tools/cpp, prunes some
  unnecessary components, and runs R CMD build to generate the source tarball.
  Because this will install the package, you will need to ensure that the version
  of Arrow C++ available to the configure script is the same as the version
  that is vendored into the R package (e.g., you may need to unset ARROW_HOME).
• devtools::check_built("arrow_X.X.X.tar.gz") locally

Release vote

• Release vote passed!

Generate R package to submit to CRAN

• If the release candidate commit updated, rebase the CRAN release branch
  on that commit.
• Pick any commits that were made to main since the release commit that
  were needed to fix CRAN-related submission issues identified in the above
  steps.
• Remove badges from README.md
• Run urlchecker::url_check() on the R directory
• Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
  a comment @github-actions crossbow submit --group r to run all R crossbow
  jobs against the CRAN-specific release branch.
• Run Rscript tools/update-checksums.R <libarrow version> to download the checksums for the pre-compiled binaries from the ASF artifactory into the tools directory.
• Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Check binary Arrow C++ distributions specific to the R package

• Upload the .tar.gz to win-builder (r-devel only)
  and confirm (with Jon, who will automatically receive an email about the results) that the check is clean.
• Upload the .tar.gz to MacBuilder
  and confirm that the check is clean
• Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
  hosted binaries are used
• devtools::check_built("arrow_X.X.X.tar.gz") locally one more time (for luck)

CRAN submission

• Upload arrow_X.X.X.tar.gz to the
  CRAN submit page
• Confirm the submission email

Wait for CRAN...

• Accepted!
• Tag the tip of the CRAN-specific release branch with r-universe-release
• Add a new line to the matrix in the backwards compatability job
• (patch releases only) Update the package version in ci/scripts/PKGBUILD, dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb, r/DESCRIPTION, and r/NEWS.md
• (CRAN-only releases) Rebuild news page with pkgdown::build_news() and submit a PR to the asf-site branch of the docs site with the contents of arrow/r/docs/news/index.html replacing the current contents of arrow-site/docs/r/news/index.html
• (CRAN-only releases) Bump the version number in r/pkgdown/assets/versions.json, and update this on the the asf-site branch of the docs site too.
• Update the packaging checklist template to reflect any new realities of the
  packaging process.
• Wait for CRAN-hosted binaries on the
  CRAN package page to reflect the
  new version
• Tweet!
  • Use Bryce's script for contributor calculation.

Post-release tasks

• File an issue to update this checklist so it's up to date
• File a MINOR PR to add the keyword internal fix from the testing PR
• File minor PR to make update-checksums.R cross-platform (macOS)

[amoeba]

• Uploaded to winbuilder,
• Uploaded to macbuilder
  • Release: https://mac.r-project.org/macbuilder/results/1740180500-4ddb551dfae58eb7/
  • Devel: https://mac.r-project.org/macbuilder/results/1740180540-e3334cbe09b369aa/

[jonkeane]

Winbuilder is clean

* using log directory 'd:/RCompile/CRANguest/R-devel/arrow.Rcheck'
* using R Under development (unstable) (2025-02-20 r87772 ucrt)
* using platform: x86_64-w64-mingw32
* R was compiled by
    gcc.exe (GCC) 13.3.0
    GNU Fortran (GCC) 13.3.0
* running under: Windows Server 2022 x64 (build 20348)
* using session charset: UTF-8
* checking for file 'arrow/DESCRIPTION' ... OK
* this is package 'arrow' version '19.0.1'
* package encoding: UTF-8
* checking CRAN incoming feasibility ... [12s] OK
* checking package namespace information ... OK
* checking package dependencies ... OK
* checking if this is a source package ... OK
* checking if there is a namespace ... OK
* checking for hidden files and directories ... OK
* checking for portable file names ... OK
* checking whether package 'arrow' can be installed ... OK
* used C++ compiler: 'g++.exe (GCC) 13.3.0'
* checking C++ specification ... OK
* checking installed package size ... OK
* checking package directory ... OK
* checking for future file timestamps ... OK
* checking DESCRIPTION meta-information ... OK
* checking top-level files ... OK
* checking for left-over files ... OK
* checking index information ... OK
* checking package subdirectories ... OK
* checking code files for non-ASCII characters ... OK
* checking R files for syntax errors ... OK
* checking whether the package can be loaded ... OK
* checking whether the package can be loaded with stated dependencies ... OK
* checking whether the package can be unloaded cleanly ... OK
* checking whether the namespace can be loaded with stated dependencies ... OK
* checking whether the namespace can be unloaded cleanly ... OK
* checking loading without being on the library search path ... OK
* checking whether startup messages can be suppressed ... OK
* checking use of S3 registration ... OK
* checking dependencies in R code ... OK
* checking S3 generic/method consistency ... OK
* checking replacement functions ... OK
* checking foreign function calls ... OK
* checking R code for possible problems ... [21s] OK
* checking Rd files ... OK
* checking Rd metadata ... OK
* checking Rd line widths ... OK
* checking Rd cross-references ... OK
* checking for missing documentation entries ... OK
* checking for code/documentation mismatches ... OK
* checking Rd \usage sections ... OK
* checking Rd contents ... OK
* checking for unstated dependencies in examples ... OK
* checking line endings in shell scripts ... OK
* checking line endings in C/C++/Fortran sources/headers ... OK
* checking line endings in Makefiles ... OK
* checking compilation flags in Makevars ... OK
* checking for GNU extensions in Makefiles ... OK
* checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
* checking use of PKG_*FLAGS in Makefiles ... OK
* checking use of SHLIB_OPENMP_*FLAGS in Makefiles ... OK
* checking pragmas in C/C++ headers and code ... OK
* checking compilation flags used ... OK
* checking compiled code ... OK
* checking examples ... OK
* checking for unstated dependencies in 'tests' ... OK
* checking tests ... [79s] OK
  Running 'testthat.R' [79s]
* checking PDF version of manual ... [18s] OK
* checking HTML version of manual ... [21s] OK
* DONE
Status: OK

[amoeba]

Thanks @jonkeane. Macbuilder jobs are failing and for those I think we need to cherry-pick 288a431. I'm doing that now and will send back up to Macbuilder after I check it locally.

[amoeba]

New Macbuilder jobs with the cherry-pick:

• Release https://mac.r-project.org/macbuilder/results/1740248623-869a763df3e3dfa9/
• Devel: https://mac.r-project.org/macbuilder/results/1740248634-dbd16f08a04cf8c7/

Edit: Release and devel are clean.

[amoeba]

Hi @jonkeane , https://files.brycemecum.com/arrow_19.0.1.tar.gz is ready to submit to CRAN whenever you have a moment. And the checklist in this issue is up to date with respect to what's been done.

[jonkeane]

Have you looked at the current CRAN issues? We might have gotten unlucky and these just started now :( I see two that are worrying.

1. On r-devel-linux-x86_64-fedora-clang we are getting some compiler warnings. The first two are in cpp11, so I think we can ignore (there's already an upstream issue New CRAN warning: identifier preceded by whitespace in a literal operator declaration r-lib/cpp11#447). But the rest are in our code and we should probably address them before submitting.

2. on r-release-windows-x86_64 that's a bit more worrying. It looks like it's not able to find the binaries. Maybe this is temporary, but when we submit we do have to assert that we've addressed the issues listed there, so ideally this isn't still an issue when we submit. A point in this-being-transient's column: binaries for 18.1.0.1 have been found in the past cause they are being distributed on the main page.

*** pkg-config found.
*** Unable to retrieve libarrow for version 18.1.0 (windows)
*** Proceeding without libarrow (build not authorized)
Arrow C++ library was not found
------------------------- NOTE ---------------------------
There was an issue preparing the Arrow C++ libraries.
See https://arrow.apache.org/docs/r/articles/install.html
----------------------------------------------------------
ERROR: configuration failed for package 'arrow'

[jonkeane]

I've created #45605 and am poking at that (at first trying to get a replication of the failure — the change I think is just delete the whitespace before the identifier)

[amoeba]

The whitespace one is new, glad you caught it. The second (windows) one was there before and I maybe incorrectly assumed it would magically fix itself. Do we have a good way of reproducing their r-release-windows-x86_64 env?

[jonkeane]

Do we have a good way of reproducing their r-release-windows-x86_64 env?

Not exactly reproduce. And if they have cut our internet access off there for some reason (like they did with macos for a while) we're stuck. One thing we could try is increasing the timeout floor that we have it's possible (though I don't see direct evidence that) it's timing out and that's the issue

[amoeba]

I'm happy to give that a try and lean on your experience with this but could we just submit and hope (with some extra notes in cran-comments about what we're seeing)?

[jonkeane]

I'm happy to give that a try and lean on your experience with this but could we just submit and hope (with some extra notes in cran-comments about what we're seeing)?

I'm going to wait for the clang20 image to build tonight and then try the whitespace fix. IMO we should include the timeout increase as well and if we add those two things before submitting we can say we've tried to address them. Incoming timeout increase PR in a sec #45607

[amoeba]

Sounds good to me. I can cherry-pick those two PRs once we've merged them and prep a new submission archive for you. Thanks for taking the initiative on the fixes!

[amoeba]

Just to track the status of cherry-picks here:

• MINOR: [R] Bump the timeout for library download #45607: cherry-picked onto maint-19.0.1-r as 38973615a48e4cbd1251718bab55e6e4af9f4f0c.
• GH-45605: [R][C++] Fix identifier ... preceded by whitespace warnings #45606: cherry-picked onto maint-19.0.1-r as c47a0227d681514895e225255e9415d62ca90ea6

[amoeba]

@jonkeane do we want to run through winbuilder and macbuilder again with the new tarball before submitting to CRAN?

[jonkeane]

@jonkeane do we want to run through winbuilder and macbuilder again with the new tarball before submitting to CRAN?

Yes, please! 🙏

[amoeba]

Okay, doing that now.

[amoeba]

Latest round of macbuilder runs:

• Release: https://mac.r-project.org/macbuilder/results/1740422944-6ee8261105f9a5c4/
• Devel: https://mac.r-project.org/macbuilder/results/1740422970-d2d33743c3f04b0e/

[amoeba]

Mac builder looks good. @jonkeane here's the latest archive for CRAN: https://files.brycemecum.com/arrow_19.0.1.tar.gz.

[jonkeane]

Thanks! I just submitted it

[jonkeane]

Womp womp, we got bounced for revdeps, just got this output from an email. I haven't had a chance to look into what the issue is / how we could/should go about it.

Changes to worse in reverse depends:

Package: pxmake
Check: DESCRIPTION meta-information
New result: NOTE
    Missing dependency on R >= 4.1.0 because package code uses the pipe
    |> or function shorthand \(...) syntax added in R 4.1.0.
    File(s) using such syntax:
      ‘px_add_totals.px.Rd’ ‘px_aggregallowed.px.Rd’ ‘px_autopen.px.Rd’
      ‘px_axis_version.px.Rd’ ‘px_baseperiod.px.Rd’ ‘px_cellnote.px.Rd’
      ‘px_cellnotex.px.Rd’ ‘px_cfprices.px.Rd’ ‘px_charset.px.Rd’
      ‘px_codepage.px.Rd’ ‘px_confidential.px.Rd’ ‘px_contact.px.Rd’
      ‘px_contents.px.Rd’ ‘px_contvariable.px.Rd’ ‘px_copyright.px.Rd’
      ‘px_creation_date.px.Rd’ ‘px_decimals.px.Rd’ ‘px_description.px.Rd’
      ‘px_descriptiondefault.px.Rd’ ‘px_domain.px.Rd’
      ‘px_elimination.px.Rd’ ‘px_infofile.px.Rd’ ‘px_language.px.Rd’
      ‘px_languages.px.Rd’ ‘px_last_updated.px.Rd’ ‘px_link.px.Rd’
      ‘px_map.px.Rd’ ‘px_matrix.px.Rd’ ‘px_micro.Rd’ ‘px_next_update.px.Rd’
      ‘px_note.px.Rd’ ‘px_notex.px.Rd’ ‘px_order.px.Rd’
      ‘px_precision.px.Rd’ ‘px_showdecimals.px.Rd’ ‘px_source.px.Rd’
      ‘px_stockfa.px.Rd’ ‘px_subject_area.px.Rd’ ‘px_subject_code.px.Rd’
      ‘px_tableid.px.Rd’ ‘px_timeval.px.Rd’ ‘px_title.px.Rd’
      ‘px_units.px.Rd’ ‘px_update_frequency.px.Rd’ ‘px_validate.Rd’
      ‘px_valuenote.px.Rd’ ‘px_valuenotex.px.Rd’ ‘px_values.px.Rd’
      ‘px_variable_label.px.Rd’ ‘px_variable_type.px.Rd’

[amoeba]

I can take a look.

[amoeba]

I'm not yet sure what's going on.

pxmake v0.15.1 passed revdep check when we ran that in https://github.com/ursacomputing/crossbow/actions/runs/13445757103/job/37570963064. They had a release last week (https://github.com/StatisticsGreenland/pxmake/releases/tag/v0.15.1) and one earlier one since our last CRAN submission but no changes look related to this. I manually checked pxmake from the v0.15.1 tag against the version of arrow installed from the CRAN tarball and it checks cleanly. Is there another/better way to run try to reproduce what CRAN sent? I'm worried that our recheck job can't catch issues like this for us.

The NOTE you shared above looks clear enough but it looks like there's nothing we should be required to do about it.

[amoeba]

@jonkeane I'm back to looking at this again and I see that Kurt Hornik added this check on Jan 25, see https://stat.ethz.ch/pipermail/r-devel/2025-January/083793.html and noted on Jan 22, 2025:

Currently, we seem to have about 1149 packages on CRAN with package code
using the pipe or function shorthand, with only 415 depending on R >=
4.1.0, and 734 not. Argh.

So we're likely not alone in this.

[jonkeane]

nods yeah. Maybe we should send an issue + PR to to pxmake since they (transitively) depend on 4.1.0 now? For them it's "just" jumping here: https://github.com/StatisticsGreenland/pxmake/blob/fb92b379aebf95ab998ec8edf556df95e6c57170/DESCRIPTION#L54

[jonkeane]

I can reply back as it being a false positive, but it would be nice to have us upstreaming a helping change / issue there (in addition to the mailing list)

[amoeba]

Sounds good. I filed an issue and PR over on pxmake, see StatisticsGreenland/pxmake#376.