GitHub - mongodb-developer/liberate-data: Learn how data trapped in an RDBMS can be liberated using MongoDB's Developer Data Platform (DDP) by migrating to Atlas, powering with search, and expose to API and mobile clients. This demo was created for AWS re:Invent 2023 and presented at the MongoDB booth theater at the Venetian expo hall.

Notice: Repository Deprecation

This repository is deprecated and no longer actively maintained. It contains outdated code examples or practices that do not align with current MongoDB best practices. While the repository remains accessible for reference purposes, we strongly discourage its use in production environments. Users should be aware that this repository will not receive any further updates, bug fixes, or security patches. This code may expose you to security vulnerabilities, compatibility issues with current MongoDB versions, and potential performance problems. Any implementation based on this repository is at the user's own risk. For up-to-date resources, please refer to the MongoDB Developer Center.

Liberate your data: From RDBMS to Vector Search, GraphQL & Mobile ... in minutes!

Reduce the time it takes to modernize your applications by freeing the data trapped in your relational database and migrating to the next-gen fully transactional DB of MongoDB Atlas.

Power it with advanced vector and textual search, enable consumption via a fully-managed GraphQL API, and expose data to mobile and edge consumers via the Realm mobile DB and SDKs.

Watch a walkthrough of this demo here

Prerequisites

A target DB environment - MongoDB Atlas Cluster
- Sign up for Atlas
A source DB environment - PostGreSQL
- Install Docker Desktop
The new MongoDB Relational Migrator - see Info for details.
- Install a Migrator Release
An OpenAI Account with an API Key generated
- Consult the OpenAI API reference
A tool to generate API calls - Postman
- Install Postman
Upfront setup of the Production Atlas App is required as this contains triggers to automatically embed data as it is being migrated into the Atlas cluster.
A mobile application coding environment - Xcode with Swift
- Install Xcode using App Store - see here for details on the how to download the SDK Swift
Download and install the realm-cli by running:
- npm install -g mongodb-realm-cli

Steps to Show the Migration and Mobile Deployment

Clone the Github Repo

Clone the repo and change to the Repo directory

git clone https://github.com/mongodb-developer/liberate-data.git && cd liberate-data

Create a PostgreSQL instance in Docker

Build the image:

docker build -t liberate-data-postgres .

Launch a docker container for the Postgres instance by:

docker run -d --name my-postgres -p "5432:5432" -e POSTGRES_PASSWORD=password --rm liberate-data-postgres -c wal_level=logical

Validate the Northwind schema by running this command:

docker exec -i my-postgres psql -U postgres <<EOF
WITH tbl AS
(SELECT table_schema, TABLE_NAME
FROM information_schema.tables
WHERE TABLE_NAME not like 'pg_%'
AND table_schema in ('northwind'))
SELECT table_schema, TABLE_NAME,
(xpath('/row/c/text()', query_to_xml(format('select count(*) as c from %I.%I', table_schema, TABLE_NAME), FALSE, TRUE, '')))[1]::text::int AS rows_n
FROM tbl
ORDER BY rows_n DESC;
\q
EOF

The output should look like this...

------------------------------------------------
 table_schema |       table_name       | rows_n 
--------------+------------------------+--------
 northwind    | order_details          |   2155
 northwind    | orders                 |    830
 northwind    | customers              |     91
 northwind    | products               |     77
 northwind    | territories            |     53
 northwind    | us_states              |     51
 northwind    | employee_territories   |     49
 northwind    | suppliers              |     29
 northwind    | employees              |      9
 northwind    | categories             |      8
 northwind    | shippers               |      6
 northwind    | region                 |      4
 northwind    | customer_customer_demo |      0
 northwind    | customer_demographics  |      0

Build a MongoDB Atlas Cluster

We build a small Atlas cluster to serve as the target DB and then as the back-end for the mobile app.

Login to MongoDB Atlas: Login
Create an Atlas Project named Liberate Data
Create a single region Atlas M10 cluster named production at version 6.0+. do not use another name
Create a db user named demo with readWriteAnyDatabase permissions.
Add your IP (or 0.0.0.0.0) to the Access List
Get your cluster connection string.

MongoDB Relational Migrator

In this segment, the PostgreSQL DB will be migrated to the MongoDB cluster.

Start the Mongodb Relational Migrator app - or click to reconnect Migrator
Click Import project file and select the project file: ./relational-migrator/liberate-data.relmig
Inspect the Relational and MDB diagrams. Notice how the orders collection uses the Extended Reference schema design pattern to store most frequently accessed data together.
The destination orders collection should look like this:
Perform the database migration
- Start a Data Migration Job by clicking on the other top tab along side Mapping
- Click on Create sync job
- Connect to your Source DB (postgres)
  - Postgres Credentials: Username = postgres / Password = password
- Connect to your Destination DB (Mongodb)
  - enter the Mongodb Connection string (use northwind db)
  - Like: mongodb+srv://demo:demo@production.3sov9.mongodb.net/northwind?retryWrites=true&w=majority
- Click Start
When this job is complete, validate the outcome in the MongoDB cluster
- In the Atlas UI, ensure all the collections were migrated.
- Inspect the orders collection. The most frequently accessed data from orderDetails, product, customer & employee should be nested.

Enable MongoDB Atlas Search

These steps build Search indexes on the orders and categories collection enable one to run queries via Atlas Search.

Configure MongoDB Data Services

Now we careate a public/private key pair that will be used to authenticate the realm-cli.

Create an API Key Pair. Save these values for deployment in the production-app in the next section.
Description: Liberate Data
Project Permissions: Project Owner
Save Key Pair to Terminal variables

export PUB_KEY=<REPLACEME>
export PRIV_KEY=<REPLACEME>

Setup MongoDB Atlas App Services

In this section, you will deploy an Atlas Application production-app from your local machine to Atlas. production-app contains all the preconfigured Linked Data Sources, Rules, Schema, Device Sync, GraphQL, Functions and Custom Resolvers you will need to complete this demo.

In a terminal shell, authenticate to the realm-cli by running this:
```
realm-cli login --api-key $PUB_KEY --private-api-key $PRIV_KEY
```
- If prompted with This action will terminate blah blah blah, just proceed with y
- When you see Successfully logged in, chances are you're successfully logged in.
Deploy the production-app from the root of this repo project.
- NOTE: If your cluster is not named production this command will fail. Either create a new cluster named production, or update the config.clusterName in the config.json
- Then run this:
```
realm-cli push --local app-services/production-app
```

Accept all the default prompts. The following message indicates success:

Creating draft
Pushing changes
Deploying draft
Deployment complete
Successfully pushed app up: production

Configure the App Service

Now finalize the application setup by creating an application API key which will be used in the next steps.

In the browser for your MongoDB Cluster, click on the App Services Tab.
Click on the Production App item.
Grab the App Id value - should be something like: production-app-xxxxx
Now, go to Authentication from the left panel
Click on the EDIT button for the API Keys row and Click create API Key - capturing this value too.
Now click on GraphQL from the left panel and capture GraphQL Endpoint value.
Add your OpenAI API Key in the openAI_secret in the Values section

Validate the App Service Deployment

Walk through the following steps to show that the app service is configured properly.

Linked Data Sources: Inspect that the production cluster is linked as the data source.
Rules: The orders collection should have the readAndWriteAll role. All other collections should have the readAll role.
Schema: Ensure the schema for all collections is defined. The schema for the orders collection should define required fields as below in addition to their bson types:

{
  "title": "order",
  "required": [
    "_id",
    "customerId",
    "employeeId",
    "freight",
    "orderDate",
    "shipAddress",
    "shipCity",
    "shipCountry",
    "shipName",
    "shipPostalCode",
    "shipRegion",
    "shipVia",
    "shippedDate"
  ],
  ...
}

Authentication: Two authentication providers should be enabled: email/password and API Keys. The API key named demo was created by you.
Device Sync: Flexible device sync should be enabled, set to the linked atlas cluster and the northwind database.
GraphQL: All entity types should be defined along with two custom resolvers named searchOrders and vectorSearchOrders which are linked to the Atlas Functions named funcSearchOrders and funcVectorSearchOrders respectively.

Use Postman for API testing

This step will allow one to run queries via GraphQL. This will show that the API is working as expected.

Startup the Postman app and import the collection file: ``postman/liberate-data - GraphQL.postman_collection.json```
In My Workspace, Click Collections in left panel.
Click on the "Liberate data!" heading and then in the middle panel, click on Variables.
Then enter the three variable api_key,atlas_app_id, and graphql_api - using the values previous gathered in the CURRENT VALUE column.
- Note: Obtain the GraphQL API (graphql_api) endpoint value from the GraphQL section in Atlas App Services.
  - See this diagram from an example:
Click Save
Execute the 1st POST operation Auth: Get Bearer & Access Token to authenticate and obtain tokens by hitting the blue Send button.
If successful, copy/save the value of the output access_token - without the quotes - to be used in the other queries.
Now, execute any of the other operations after inserting the access_token value in the Authorization tab. Feel free to change query values.
The Vector Search: Semantic search on Orders operation uses a custom resolver which in turn executes an Atlas Vector Search pipeline. This pipeline is implemented in the funcVectorSearchOrders function and performs a real-time embedding of the search string against OpenAI and performs a vector search.
The Search: Orders by search string operation uses a custom resolver which in turn executes an Atlas Search pipeline. This pipeline is implemented in the funcSearchOrders function and performs a fuzzy text search on the orders collection, plus a union ($unionWith) and join ($lookup) to the categories collection, thus performing a text search on orders and categories.

Swift mobile app with Realm SDK

This segment shows a mobile app built with MongoDB Realm for an Apple iPhone communicating with a MongoDB Atlas database.

Start up Xcode
Open the the Swift app called App.xcodeproj under the app-swift folder.
In the App section, open the Realm object and replace the appId and appUrl. Compile and run.
- The appId was captured earlier.
- The appUrl is found on the browser by by clicking on tApp Services production-app and copying the URL up to the /dashboard ending.
In the mobile app, register with a new user via an email and password.
Browse orders. For the purpose of this demo, all users have access to all orders.

Atlas Device Sync

In this final segment, one can show Atlas Device Sync. This shows changes in data are propagated from Mobile App to Atlas Cluster (or the reverse) immediately.

Modify an order using the mobile app.
Open the same Order document in Atlas or Compass and notice the changes. Now modify the same order and the changes will be reflected on the mobile app. Atlas Device Sync works.
Finally, run the Mutation: Change a Sales Order GraphQL operation in postman. Change the Order ID and any fields in the order body. The changes should reflect in the mobile app.