Page MenuHomeNicheWork Phabricator

MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABSExperimental
ActivePublic

Watchers

  • This project does not have any watchers.

Details

Description

MABS Proof of Concept June 2018

DEVELOPER: NicheWork, LLC WIKI: An instance of a MediaWiki installation

  1. For the purpose of this project, there are two wikis; one must be a clone of the other. They cannot be two independently created wikis. When the second wiki is cloned from the first, this is the initial synchronization event.
  1. SOURCE: The wiki that edits are being copied from. TARGET: The wiki that edits from the source are being copied to.

At any time between edits, one wiki (the TARGET) can be synchronized with the other (the SOURCE). Each wiki's role as source or target, depending on the direction of synchronization at the time, is interchangeable. In this way, synchronization will be bidirectional. During a synchronization event, the MERGE PROCESS will attempt to incorporate the changes made on the source wiki since the previous synchronization event into the target wiki. The merge process will continue until the content from the source is merged into the target unless there is a MERGE CONFLICT.

  1. A merge conflict occurs when the merge process cannot reconcile changes from the source wiki with edits on the target wiki. When a merge conflict occurs, the merge process is halted and the MERGE OPERATOR is alerted to the merge conflict. The wiki will be placed into read-only mode for all users except the merge operator. It is the responsibility of the merge operator to reconcile the merge conflict and notify the merge process that the merge process can continue.
  1. Code should be written in such a way that the purpose of any section is easy to understand by another developer with a similar skill set. Inline comments should be provided where the code would not be clear.

Recent Activity

Nov 11 2019

hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

Rebuilding on earth:

$ sed -i 's#/..wgDBTableOptions./#ENGINE=InnoDB, DEFAULT CHARSET=binary#' init_1.27.sql 
$ mysqladmin drop -f wiki_demo; mysqladmin create wiki_demo; (echo "set names 'utf8';"; echo "set character set utf8;";echo 'set unique_checks = 0;'; echo 'set autocommit = 0;'; cat init_1.27.sql ;echo "alter table page drop index name_title;"; pv 01-enwikivoyage.sql.bz2 | bzcat ; echo 'set unique_checks = 1;' ) | mysql wiki_demo
Nov 11 2019, 2:18 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Nov 9 2019

hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.
$ git remote rename origin moon
$ git remote add earth mediawiki::https://asyncwiki-earth.wmflabs.org/demo
Nov 9 2019, 3:26 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Nov 5 2019

hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

Created a bot user user Special:BotPasswords and pushed an edit back to the moon. Voila!

Nov 5 2019, 3:01 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

Wiki is kind of working, so I'm gonna stop fiddling with it. Trying the original git clone:

Nov 5 2019, 2:42 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Sep 30 2019

hexmode changed the visibility for T27: Automated MABS setup - in case of failure. Fail case Oct 23, 2018.
Sep 30 2019, 5:56 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T151: MABS: Refactor out static class.
Sep 30 2019, 5:56 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T171: MABS: Use History browser for merge tool.
Sep 30 2019, 5:54 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T44: MABS: Discussion of current and future plans.
Sep 30 2019, 5:54 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T242: MABS: How to import.
Sep 30 2019, 5:53 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T152: MABS: Special page is in bad state, fix now.
Sep 30 2019, 5:52 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T287: MABS: Create docker for quick testing.
Sep 30 2019, 5:50 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Sep 16 2019

hexmode moved T287: MABS: Create docker for quick testing from Current to Backlog on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:02 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T301: MABS: Fix double-fetch of pages from Backlog to Current on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:02 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T152: MABS: Special page is in bad state, fix now from Backlog to Completed on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:01 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T325: MABS: Disable UploadWizard, SMW and any other extensions interfereing with demo from Backlog to Completed on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:01 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T287: MABS: Create docker for quick testing from Completed to Current on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:01 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T338: MABS: Don't always fetch all revisions from Backlog to Completed on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:01 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T287: MABS: Create docker for quick testing from Backlog to Completed on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:00 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T307: MABS: merge should rebase wikiX onto wikiY from Backlog to Current on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:00 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T319: MABS: Set up MABS demonstration on Wikimedia cloud services from Backlog to Current on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:00 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode moved T326: MABS: Sync ONE article between asywiki demo wikis from Backlog to Current on the MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS board.
Sep 16 2019, 10:00 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode lowered the priority of T324: MABS: Check out the 2Col conflict extension from Normal to Low.
Sep 16 2019, 9:59 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode lowered the priority of T305: MABS: push to recent clone should probably never say most recent remote ref was 0 from Normal to Low.
Sep 16 2019, 9:57 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode lowered the priority of T304: MABS: Test fetch strategies from Normal to Low.
Sep 16 2019, 9:56 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode lowered the priority of T302: MABS: Remote rename "loses" current fetch and results in a forced update from Normal to Low.
Sep 16 2019, 9:56 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode raised the priority of T326: MABS: Sync ONE article between asywiki demo wikis from Normal to High.
Sep 16 2019, 9:55 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T326: MABS: Sync ONE article between asywiki demo wikis.
Sep 16 2019, 9:45 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T325: MABS: Disable UploadWizard, SMW and any other extensions interfereing with demo.
Sep 16 2019, 9:45 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T324: MABS: Check out the 2Col conflict extension.
Sep 16 2019, 9:44 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T318: MABS: Import Wikivoyage into testing.
Sep 16 2019, 9:42 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T319: MABS: Set up MABS demonstration on Wikimedia cloud services.
Sep 16 2019, 9:42 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T307: MABS: merge should rebase wikiX onto wikiY.
Sep 16 2019, 9:41 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T306: MABS: Empty lines removed.
Sep 16 2019, 9:40 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T305: MABS: push to recent clone should probably never say most recent remote ref was 0.
Sep 16 2019, 9:38 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T304: MABS: Test fetch strategies.
Sep 16 2019, 9:36 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T302: MABS: Remote rename "loses" current fetch and results in a forced update.
Sep 16 2019, 9:35 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T300: MABS: Create working demo.
Sep 16 2019, 9:30 AM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Sep 15 2019

hexmode shifted T301: MABS: Fix double-fetch of pages from the Restricted Space space to the S3 Public NicheWork space.
Sep 15 2019, 11:02 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T301: MABS: Fix double-fetch of pages.
Sep 15 2019, 11:00 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode closed T338: MABS: Don't always fetch all revisions as Resolved.

with the work done today, this is done. At least when only a few pages are specified, only those are fetched.

Sep 15 2019, 11:00 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode closed T338: MABS: Don't always fetch all revisions, a subtask of T300: MABS: Create working demo, as Resolved.
Sep 15 2019, 11:00 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

Note that the above was resolved by re-importing the SQL and then running rebuildrecentchanges.php

Sep 15 2019, 10:57 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode changed the visibility for T338: MABS: Don't always fetch all revisions.
Sep 15 2019, 3:43 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T338: MABS: Don't always fetch all revisions.

This is demonstrated by the following:

git clone -c remote.origin.pages='Atlanta Toledo ' mediawiki::http://asyncwiki-moon.wmflabs.org/demo/ async

which should only fetch the histories for two pages.

Sep 15 2019, 3:42 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode updated the task description for T338: MABS: Don't always fetch all revisions.
Sep 15 2019, 3:39 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Sep 14 2019

hexmode triaged T338: MABS: Don't always fetch all revisions as Normal priority.
Sep 14 2019, 6:42 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS

Sep 12 2019

hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

keep ending up here:

Refreshing links tables.
Starting from page_id 1 of 173884.
[b0f10473827284f0a2844f14] [no req]   MediaWiki\Revision\RevisionAccessException from line 1635 of /opt/htdocs/mediawiki/includes/Revision/RevisionStore.php: Main slot of revision 934 not found in database!
Backtrace:
#0 /opt/htdocs/mediawiki/includes/Revision/RevisionStore.php(1671): MediaWiki\Revision\RevisionStore->loadSlotRecords(string, integer)
#1 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#2 /opt/htdocs/mediawiki/includes/Revision/RevisionSlots.php(165): call_user_func(Closure)
#3 /opt/htdocs/mediawiki/includes/Revision/RevisionRenderer.php(169): MediaWiki\Revision\RevisionSlots->getSlots()
#4 /opt/htdocs/mediawiki/includes/Revision/RevisionRenderer.php(128): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#5 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#6 /opt/htdocs/mediawiki/includes/Revision/RenderedRevision.php(175): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#7 /opt/htdocs/mediawiki/includes/Storage/DerivedPageDataUpdater.php(1266): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#8 /opt/htdocs/mediawiki/includes/Storage/DerivedPageDataUpdater.php(1288): MediaWiki\Storage\DerivedPageDataUpdater->getCanonicalParserOutput()
#9 /opt/htdocs/mediawiki/includes/Storage/DerivedPageDataUpdater.php(1560): MediaWiki\Storage\DerivedPageDataUpdater->getSecondaryDataUpdates(boolean)
#10 /opt/htdocs/mediawiki/includes/page/WikiPage.php(2106): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#11 /opt/htdocs/mediawiki/maintenance/refreshLinks.php(275): WikiPage->doSecondaryDataUpdates(array)
#12 /opt/htdocs/mediawiki/maintenance/refreshLinks.php(198): RefreshLinks::fixLinksFromArticle(integer, boolean)
#13 /opt/htdocs/mediawiki/maintenance/refreshLinks.php(83): RefreshLinks->doRefreshLinks(integer, boolean, string, boolean, boolean)
#14 /opt/htdocs/mediawiki/maintenance/rebuildall.php(60): RefreshLinks->execute()
#15 /opt/htdocs/mediawiki/maintenance/doMaintenance.php(94): RebuildAll->execute()
#16 /opt/htdocs/mediawiki/maintenance/rebuildall.php(67): require_once(string)
#17 {main}
Sep 12 2019, 7:57 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

php maintenance/populateContentModel.php --ns all --table revision

Sep 12 2019, 7:23 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

fixed by adding a row with role_id=2 and role_name='aux'

Sep 12 2019, 6:40 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS
hexmode added a comment to T326: MABS: Sync ONE article between asywiki demo wikis.

Apparently it fixed itself without my knowledge. Now getting what looks like this bug.

Sep 12 2019, 6:09 PM · MULTILATERAL, ASYNCHRONOUS, BIDIRECTIONAL SYNCHRONISATION (OF WIKIS) MABS