wikiteam3 v4 release #176

yzqzss · 2023-08-27T01:24:50Z

Fixes

fix: --xmlapiexport infinite loop on page with missing revision
fix: config is not passed to log_error()
fix: parentid should be a positive integer
fix: truncated API response for "allrevisions" causes infinite loop (truncated API response for "allrevisions" causes infinite loop #166)
fix: yield bug in red_titles()
fix: Unable to traverse Special:Filelist
fix: can't find API URL from HTML (new MW versions)
...

feats

feat: ensure params["limit"] is int and >= 2
feat: save timestamp metadata to images.txt
feat: cli: --disable-image-verify (Some images get dropped from the dump #170)
feat: cli: add --image-timestamp-interval to download images uploaded in the given time interval (Add option to backup recent images only (with a custom cut-off date) #151)
feat: cli: add --user-agent to custom UA
feat: --ia-wbm-booster Incremental image dump powered by web.archive.org (experimental)
feat: write .mark file after dumped
feat: image: dynamic load config.delay
feat: If there are archives on IA in the last year, exit
feat: a trick to get original file for fandom.com
feat: more robust convert URLs to prefix filename
...

refactor

countless
...

drop legacy code

gui.py
wikia.py
launcher.py
...

Special:Export

Do not strip sha1 tags from XML

Breaking changes

image: change the magic NULL string from False to null
drop .desc xmlfile support

Drop launcher

Shifts compression responsibilities from the launcher to the uploader.

dependencies

remove: all useless dependencies
add: python-slugify
add (uploader): zstd

refactor uploader

Use zstd to compress .xml for better compression speed, compression ratio, decompression speed, streaming decompression. （with -{19..22} --ultra --long=31)
Instead of packaging the whole wikidump into wikidump&history.7z, we upload each file separately for better parsing and management in the future.
By default, the compression tasks of each process will be queued to avoid overload/OOM. (socket lock) (--parallel to disable it)
Upload logo. (optional)
Stop converting language codes to human readable formats.
IA ITEM medadata: add 'upload-state' and 'scanner'.
...

https://pypi.org/project/wikiteam3/

strip and unquote url Convert domain to IDNA Remove port `:80` and `:443` if `http://` and `https://` respectively remove last slash if exists truncate to last slash remove "/any.php" suffix (`r"(/[^/]+\.php)"`) remove ~ tilde sulgify the url path if `ascii_slugify` is True replace port(`:`) with underscore(`_`) lower case

pre-bump 4.0.0 WIP: deprecate launcher WIP: refactor uploader WIP: rename function names from camelCase to snake_case WIP: default args WIP: feat: `--noverify-image-size` WIP: enable `--xmlrevisions --curonly` support TODO: feat: save `Special:Log` wiki page break change: image: change the magic NULL string from `False` to `null` chore: remove deprecated alias for xml.etree.ElementTree TODO: feat: cli: add an option to disable random UserAgent TODO: feat: Incremental image dump powered by archive.org (experimental) WIP: feat: add option to dump recent images only (with a custom cut-off date)

Reasons: * if the limit is float and between 0 to 1, the MW backend will force-int it to 0 * To loop over all the revisions, we need to retrieve at least 2 at a time * If the historical revisions cannot be retrieved even with limit=2, it should fail at this time so that the user can try to use --curonly .

close: mediawiki-client-tools#166

The last titlelist may not be yielded

deprecate: `.desc` xmlfile support feat: cli: `--disable-image-verify`

ref: mediawiki-client-tools#156 (comment)

…ed in the given time interval close: mediawiki-client-tools#151

pre-bump: version 4.0.0 feat: cli: add `--user-agent` to custom UA

feat: `--ia-wbm-booster` feat: write .mark file after dumped refactor: SessionMonkeyPatch feat: image: dynamic load config.delay

feat: cli: `--upload`: Upload wikidump to Internet Archive after successfully dumped

bump 4.0.14

(new MediaWiki versions)

yzqzss · 2023-08-27T01:32:19Z

Point of Conflict.

eeab93a

randomnetcat · 2023-08-27T01:37:25Z

Several things:

This is a truly massive PR. Could you consider splitting it up into smaller ones? I wouldn't even know where to start reviewing this.
Just glancing at the changes, you use the "saveweb" project in URLs several times, which is not this project.
From reading the description, it sounds like you're breaking format compatibility with upstream wikiteam and older versions of this project. That's not a good idea, since people are depending on that (at least I am in some of my code).

elsiehupp · 2023-08-27T01:51:02Z

This is the sort of thing where I will defer to everyone else.

In general I would say that anything breaking backwards compatibility should be dependent on implementation of build-versioning that would allow users to reliably target older versions.

Right now this repository does not even have version-tagged GitHub builds, let alone versioned PyPI builds, so at this point breaking backwards compatibility is a no-go.

As for drastically refactoring the code… that's fine, and probably for the better, as long as there is build-versioning in place to protect existing users.

elsiehupp · 2023-08-27T01:59:07Z

Regarding format compatibility: I think introducing a new default format is fine as long as the existing upstream format continues to be supported alongside it for a substantial "bridge" period, with (a) an ability to convert existing dumps to the new format, and (b) there are strong "deprecation" nudges encouraging users to migrate.

Refactoring (and abstracting much of the backend) could of course facilitate this, hence why I'm supportive of refactoring more generally.

Basically what I'm saying is that introducing a new data format should be dependent on first establishing a stable public API for the backend, which currently does not exist.

robkam · 2023-08-27T14:15:05Z

This is too complex for my level of comprehension - except that this PR for wikiteam3 is not in that repository.

elsiehupp · 2023-08-29T11:23:21Z

Here's my thought…

@yzqzss why don't you open a new Pull Request from an earlier commit on this branch? (I think you have to create a branch from that commit in order to do so.)

This would be much easier to approach if it wasn't a gigantic total total rewrite all at once, and breaking it into chunks this way would help.

If you're not interested in doing thing, to be fair, we could try and do so ourselves, but you're more familiar with your own code than we are.

yzqzss added 30 commits August 3, 2023 21:35

CI: remove pre-commit and travis

65719b7

dependencies: remove useless dependencies

48cd660

deprecate: wikia.py and wikkii-spider.py

1804882

remove: wikiteam3.gui

c9f7681

fix: standardize_url()

7c0f08e

WIP: refactor uploader

6a49a93

do not strip sha1s from XML

d0b7db6

fix: Accidentally changed session.post to session.get in refactoring :(

074dd42

fix: config is not passed to log_error()

b283425

fix: parentid should be a positive integer

9b1c2a5

<element name="parentid" type="positiveInteger" minOccurs="0" maxOccurs="1"/>

sort config.json

51eab0a

fix: --xmlapiexport infinite loop on page with missing revision

65ce4a5

fix: truncated API response for "allrevisions" causes infinite loop

76465d3

close: mediawiki-client-tools#166

fix: yield bug in red_titles()

57421f3

The last titlelist may not be yielded

update .gitattributes

790b9a7

feat: save timestamp metadata to images.txt

7987c33

deprecate: `.desc` xmlfile support feat: cli: `--disable-image-verify`

fix: Unable to traverse Special:Filelist

1043567

ref: mediawiki-client-tools#156 (comment)

feat: cli: add --image-timestamp-interval to download images upload…

2b567a5

…ed in the given time interval close: mediawiki-client-tools#151

drop: launcher.py

75d04ac

deprecate: dumpgenerator script (use wikiteam3dumpgenerator instead)

62f0c56

pre-bump: version 4.0.0 feat: cli: add `--user-agent` to custom UA

rename: parseLastPageChunk() -> parse_last_page_chunk()

3b9a582

dependencies: re-lock

f581f32

name back to wikiteam3 !

5646b79

refactor: Delay

310339e

Update: README.md

73022e5

refactor: ...

ff3b066

feat: `--ia-wbm-booster` feat: write .mark file after dumped refactor: SessionMonkeyPatch feat: image: dynamic load config.delay

refactor: uploader

9b047cd

yzqzss added 22 commits August 19, 2023 10:43

fix: Delay.__init__() got an unexpected keyword argument 'session'

c2b18bc

bump 4.0.10

ba68b7a

Update: README.md

0231c2c

ia_wbm_booster: fallfast on HTTP 429

bb0f0f2

feat: cli: `--upload`: Upload wikidump to Internet Archive after successfully dumped

uploader: print item url

a484901

bump 4.0.12

983cfb2

include errors.log into IA item

8c549cf

bump 4.0.13

f8c51e7

feat: uploader: --parallel

d3f1414

bump 4.0.14

feat: If there are archives on IA in the last year, exit

4bda5d9

fix: can't find API URL from HTML

ddc73cf

(new MediaWiki versions)

feat: uploader: --zstd-level

8343aa3

bump 4.0.15

5a9a777

fix: --upload only work with --resume

83666a6

bump 4.0.16

55e79ea

session: set Accept header

36fc712

bump 4.0.17

b434e37

increase the default delay from 0.5 to 1.5

6475ffc

update User-Agent

fdc11f6

temporary comment out an unicode check

d1b5a8c

feat: a trick to get original file for fandom.com

47e2cbb

bump 4.0.18

82ad61e

yzqzss marked this pull request as ready for review August 27, 2023 01:31

yzqzss closed this Aug 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wikiteam3 v4 release #176

wikiteam3 v4 release #176

yzqzss commented Aug 27, 2023 •

edited

Loading

yzqzss commented Aug 27, 2023 •

edited

Loading

randomnetcat commented Aug 27, 2023

elsiehupp commented Aug 27, 2023

elsiehupp commented Aug 27, 2023

robkam commented Aug 27, 2023

elsiehupp commented Aug 29, 2023

wikiteam3 v4 release #176

wikiteam3 v4 release #176

Conversation

yzqzss commented Aug 27, 2023 • edited Loading

Fixes

feats

refactor

drop legacy code

Special:Export

Breaking changes

Drop launcher

dependencies

refactor uploader

yzqzss commented Aug 27, 2023 • edited Loading

randomnetcat commented Aug 27, 2023

elsiehupp commented Aug 27, 2023

elsiehupp commented Aug 27, 2023

robkam commented Aug 27, 2023

elsiehupp commented Aug 29, 2023

yzqzss commented Aug 27, 2023 •

edited

Loading

yzqzss commented Aug 27, 2023 •

edited

Loading