diff --git a/source/faq.txt b/source/faq.txt index 3becd22d0..dd6c15db7 100644 --- a/source/faq.txt +++ b/source/faq.txt @@ -102,6 +102,8 @@ available again, restart the ``mongosync`` process with the same parameters. ``mongosync`` resumes the operation from where it stopped when ``mongosync`` became unavailable. +.. include:: /includes/fact-restart-resume-delay.rst + Can the source or destination be a replica set with arbiters? ------------------------------------------------------------- diff --git a/source/includes/fact-restart-resume-delay.rst b/source/includes/fact-restart-resume-delay.rst new file mode 100644 index 000000000..ddf0a0ff1 --- /dev/null +++ b/source/includes/fact-restart-resume-delay.rst @@ -0,0 +1,6 @@ +.. note:: + + Starting in ``mongosync`` 1.7.3, ``mongosync`` can take at least two minutes + to respond when you resume or restart a sync operation. During this time, + any calls to the :ref:`c2c-api-progress` endpoint might fail. If a + ``progress`` call fails, it is safe to retry. diff --git a/source/index.txt b/source/index.txt index 90fc25181..2bc69c961 100644 --- a/source/index.txt +++ b/source/index.txt @@ -21,6 +21,8 @@ and data residency compliance. In addition to continuous data synchronization, ``mongosync`` can also facilitate a one time data migration between clusters. +For an overview of how ``mongosync`` works, see :ref:`about-mongosync`. + To get started with ``mongosync``, refer to the :ref:`Quick Start Guide `. For more detailed information, refer to the :ref:`c2c-install` or :ref:`c2c-connecting` page that best fits your diff --git a/source/quickstart.txt b/source/quickstart.txt index 2483c123f..7d3662fc4 100644 --- a/source/quickstart.txt +++ b/source/quickstart.txt @@ -247,6 +247,15 @@ documentation provides details on using the following endpoints: * - :ref:`c2c-api-reverse` - Reverses the direction of a committed sync operation. +Finalize Cutover Process +------------------------ + +You can finalize a migration and transfer your application +workload from the source to the destination cluster using the +``mongosync`` cutover process. + +For more information, see :ref:`c2c-cutover-process`. + One-Time Sync ------------- diff --git a/source/reference.txt b/source/reference.txt index c2f432e0f..c00a2d7df 100644 --- a/source/reference.txt +++ b/source/reference.txt @@ -2,8 +2,11 @@ Reference ========= +:ref:`about-mongosync` + The primary process used in ``mongosync``. + :ref:`c2c-mongosync` - The primary process used in {+c2c-product-name+}. + The options and behavior {+c2c-product-name+}. :ref:`c2c-config` Configure :program:`mongosync` instances at startup using a @@ -51,6 +54,7 @@ Reference .. toctree:: :titlesonly: + /reference/about-mongosync /reference/mongosync /reference/configuration /reference/api diff --git a/source/reference/about-mongosync.txt b/source/reference/about-mongosync.txt new file mode 100644 index 000000000..840ded992 --- /dev/null +++ b/source/reference/about-mongosync.txt @@ -0,0 +1,132 @@ +.. _about-mongosync: + +==================== +About ``mongosync`` +==================== + +.. default-domain:: mongodb + +.. contents:: On this page + :local: + :backlinks: none + :depth: 1 + :class: singlecol + +The ``mongosync`` binary is the primary process used in {+c2c-product-name+}. +``mongosync`` migrates data from a source cluster to a destination cluster and +keeps the clusters in continuous sync. You can use ``mongosync`` to create +dedicated analytics, development, or testing clusters that mirror your +production environment. + +``mongosync`` keeps track of its current actions through +:ref:`states `. ``mongosync`` enters different states depending on +the requests it receives. The current ``mongosync`` state determines which API +operations you can run. + +How ``mongosync`` Works +----------------------- + +Connect Source and Destination Clusters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You must create an initial connection to the source and destination +clusters before ``mongosync`` can start to sync data. For more information, see +:ref:`c2c-connecting`. + +When ``mongosync`` connects to the source and destination clusters, the +synchronization process enters the :ref:`IDLE ` state. + +During Sync +~~~~~~~~~~~ + +To start the synchronization process, call the :ref:`start ` +endpoint. + +The ``start`` endpoint begins the ``RUNNING`` state, which is when ``mongosync`` +syncs data from the source cluster to the destination cluster through the +:ref:`collection copy ` process. If you make subsequent +writes to the source cluster, ``mongosync`` applies the operations to the +destination cluster through the :ref:`change event application ` +process. + +For every collection on the source, ``mongosync`` creates a corresponding +collection on the destination. + +``mongosync`` temporarily alters some collection characteristics during +the ``RUNNING`` state and changes these back to their source +cluster characteristics during the ``COMMIT`` state. For more information, see +:ref:`Commit Alters `. + +.. _c2c-collection-copy: + +Collection Copy +''''''''''''''' + +The collection copy process is the initial full data transfer that copies each +collection from the source cluster to the destination cluster. ``mongosync`` +performs collection copy by splitting each source collection into partitions +and copying all the partitions to their corresponding destination collections. + +.. _c2c-cea: + +Change Event Application +'''''''''''''''''''''''' + +The change event application (CEA) process continuously updates +the destination cluster when you make changes to the source cluster. If there +are multiple ``mongosync`` instances, all instances finish +collection copy before starting the CEA process. + +During CEA, ``mongosync`` receives :ref:`change stream ` events +from the source cluster, converts these events into writes, and applies these +writes on the destination cluster. + +Pausing Sync +~~~~~~~~~~~~ + +To pause the sync process and enter the :ref:`PAUSED ` state, +call the :ref:`pause ` endpoint. + +To resume a paused synchronization session and return ``mongosync`` to the +``RUNNING`` state, call the :ref:`resume ` endpoint. + +Finalizing Sync +~~~~~~~~~~~~~~~ + +To finalize the sync between the source and destination clusters, +call the :ref:`commit ` endpoint. + +The ``commit`` endpoint starts the :ref:`COMMITTING ` +state. ``mongosync`` stops continuous sync between the source and desitnation +clusters. It also restores collection characteristics that ``mongosync`` +temporarily alters during synchronization. + +Once you call the ``commit`` endpoint, ``mongosync`` halts any writes on the +source cluster and retrieves the timestamp of the most recent operation in the +source cluster. + +If there are no errors, ``mongosync`` sends a success message and exits CEA by +processing events up until the commit timestamp. Then, ``mongosync`` corrects +indexes and resolves document conflicts. If write blocking is enabled, +``mongosync`` turns write operations back on. + +When the above process is completed, ``mongosync`` enters the +:ref:`COMMITTED ` state. The ``COMMITTED`` state indicates +that the :ref:`cutover process ` is finished, and you can +direct writes to the destination cluster. + +Reversing Sync +~~~~~~~~~~~~~~ + +To reverse the direction of a committed sync operation, call the +:ref:`reverse ` endpoint. + +The reverse endpoint starts the :ref:`REVERSING ` state. +``mongosync`` copies data from the destination cluster to the source cluster. +Then, ``mongosync`` swaps the source and destination clusters and resumes +applying change events. + +If the reverse sync is successful, ``mongosync`` enters the ``RUNNING`` state. +The synchronization continues in the reverse direction from the original +sync job. + diff --git a/source/reference/api/commit.txt b/source/reference/api/commit.txt index 5fbf63db6..35ba1090e 100644 --- a/source/reference/api/commit.txt +++ b/source/reference/api/commit.txt @@ -16,7 +16,11 @@ Description ----------- Finalizes the sync between the source cluster and the destination -cluster and stops continuous sync between clusters. ``commit`` also +cluster and stops continuous sync between clusters. + +.. _c2c-commit-alters: + +``commit`` also restores collection characteristics that ``mongosync`` temporarily alters during synchronization. diff --git a/source/reference/api/progress.txt b/source/reference/api/progress.txt index 8e53cdc7d..00406f5db 100644 --- a/source/reference/api/progress.txt +++ b/source/reference/api/progress.txt @@ -64,6 +64,8 @@ Behavior - The endpoint does not auto-refresh. To get updated status, call the ``progress`` endpoint again. + + .. include:: /includes/fact-restart-resume-delay.rst Endpoint Protection ~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/api/resume.txt b/source/reference/api/resume.txt index 334c0b0fd..32e63177a 100644 --- a/source/reference/api/resume.txt +++ b/source/reference/api/resume.txt @@ -71,12 +71,13 @@ Response Behavior -------- +- When you send a ``resume`` request, ``mongosync`` might take at least two + minutes before resuming the migration. To see if the migration resumed, you + can call the :ref:`c2c-api-progress` endpoint. + - If the ``resume`` request is successful, ``mongosync`` enters the ``RUNNING`` state. -- When you send a ``resume`` request, there may be a delay of up to a - few minutes before ``mongosync`` resumes the migration. - Endpoint Protection ~~~~~~~~~~~~~~~~~~~ diff --git a/source/reference/cutover-process.txt b/source/reference/cutover-process.txt index e4a39575f..c84ec547a 100644 --- a/source/reference/cutover-process.txt +++ b/source/reference/cutover-process.txt @@ -36,12 +36,29 @@ Steps Ensure that the ``mongosync`` process status indicates the following values: + - ``canCommit`` is ``true``. + - ``lagTimeSeconds`` is small (near ``0``). If ``lagTimeSeconds`` isn't close to ``0`` when the cutover starts, cutover might take a long time. - - ``canCommit`` is ``true``. + The following example returns the status of the synchronization process. + + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. literalinclude:: /includes/api/responses/progress.json + :language: json + :copyable: false + :emphasize-lines: 5, 8 + .. step:: Stop any write operations to the synced collections on the source. @@ -50,8 +67,16 @@ Steps the entire source cluster during the commit (step 4) for you. - If you didn't start ``mongosync`` with ``enableUserWriteBlocking``, ensure that writes are disabled. - For example, run the ``setUserWriteBlockMode`` command on the - source cluster. + For example, run the :dbcommand:`setUserWriteBlockMode` command on the + source cluster: + + .. code-block:: javascript + + db.adminCommand( { + setUserWriteBlockMode: 1, + global: true + } ) + - If ``mongosync`` uses filtered sync, it's not necessary to disable writes to the entire source cluster. But you must ensure that write operations are stopped for the collections included @@ -63,12 +88,24 @@ Steps migration, you must issue a commit request for each ``mongosync`` instance. + Request + ~~~~~~~~ + + .. literalinclude:: /includes/api/requests/commit.sh + :language: shell + + Response + ~~~~~~~~ + + .. literalinclude:: /includes/api/responses/success.json + :language: json + :copyable: false + .. note:: - Ensure that the ``mongosync`` process response for the - ``commit`` request indicates that the ``mongosync`` state is - ``COMMITTING`` or ``COMMITTED``, which is the correct - ``mongosync`` state when you send a ``commit`` request. + After you submit a ``commit`` request, call the ``progress`` endpoint + to ensure that the ``mongosync`` state is ``COMMITTING`` or + ``COMMITTED``. .. step:: Wait until you can perform writes on the destination cluster. @@ -76,6 +113,42 @@ Steps ``true``. If ``canWrite`` is ``false``, wait until ``progress`` shows ``canWrite`` is ``true``. + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. code-block:: json + :emphasize-lines: 6 + :copyable: false + + { + "progress": + { + "state":"COMMITTED", + "canCommit":true, + "canWrite":true, + "info":"change event application", + "lagTimeSeconds":0, + "collectionCopy": + { + "estimatedTotalBytes":694, + "estimatedCopiedBytes":694 + }, + "directionMapping": + { + "Source":"cluster0: localhost:27017", + "Destination":"cluster1: localhost:27018" + } + }, + "success": true + } + + .. step:: Verify data transfer. Verify the successful sync of data from the source to the @@ -102,6 +175,43 @@ Steps ``mongosync`` state is ``COMMITTED``, the cutover process is complete. + Request + ~~~~~~~ + + .. literalinclude:: /includes/api/requests/progress.sh + :language: shell + + Response + ~~~~~~~~ + + .. code-block:: json + :emphasize-lines: 4 + :copyable: false + + { + "progress": + { + "state":"COMMITTED", + "canCommit":true, + "canWrite":false, + "info":"change event application", + "lagTimeSeconds":0, + "collectionCopy": + { + "estimatedTotalBytes":694, + "estimatedCopiedBytes":694 + }, + "directionMapping": + { + "Source":"cluster0: localhost:27017", + "Destination":"cluster1: localhost:27018" + } + }, + "success": true + } + + + Learn More ---------- diff --git a/source/reference/mongosync-states.txt b/source/reference/mongosync-states.txt index 0b69d69f9..bff611711 100644 --- a/source/reference/mongosync-states.txt +++ b/source/reference/mongosync-states.txt @@ -17,6 +17,14 @@ receives. ``mongosync`` can only be in a single state at a given time. The current ``mongosync`` states determines which API operations you can run. +.. note:: + + Most state transitions happen as the result of an API call. The exceptions + are: + + - The transition from ``COMMITTING`` to ``COMMITTED`` + - The transition from ``REVERSING`` to ``RUNNING`` + This page describes ``mongosync`` states. View the Current State @@ -26,6 +34,8 @@ To view the current state of ``mongosync``, use the :ref:`/progress `. endpoint. The :ref:`/progress ` endpoint returns the state in the ``state`` field. +.. include:: /includes/fact-restart-resume-delay.rst + .. _c2c-states-descriptions: State Descriptions diff --git a/source/reference/mongosync.txt b/source/reference/mongosync.txt index cd050b5f9..7a2cb6421 100644 --- a/source/reference/mongosync.txt +++ b/source/reference/mongosync.txt @@ -206,6 +206,8 @@ Even if documents move between source shards during replication, {+c2c-product-name+} maintains the :term:`eventual consistency` guarantee on the destination cluster. +.. _c2c-mongosync-ddl: + Data Definition Language (DDL) Operations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/source/release-notes.txt b/source/release-notes.txt index 5047706f5..4ffade3d7 100644 --- a/source/release-notes.txt +++ b/source/release-notes.txt @@ -38,6 +38,7 @@ Previous Rapid Releases .. toctree:: :titlesonly: + /release-notes/1.8 /release-notes/1.7 /release-notes/1.6 /release-notes/1.5 diff --git a/source/release-notes/1.7.txt b/source/release-notes/1.7.txt index 34113e303..83c88ca65 100644 --- a/source/release-notes/1.7.txt +++ b/source/release-notes/1.7.txt @@ -70,6 +70,10 @@ Issues Fixed: events when mongosync is interrupted and resumed. This is an unlikely edge case. +Other Changes: + +- Adds a two minute delay when restarting and resuming sync operations. + Limitations: - You must stop the balancer on both the source and destination sharded