Node script to copy projects from staging or prod by rmunn · Pull Request #1816 · sillsdev/web-languageforge

@rmunn

Often fails to copy assets because `kubectl cp` or `kubectl exec tar`
get cut off partway through, but that should go away once Kubernetes
version 1.30 is released.

@rmunn added the engineering

Tasks which do not directly relate to a user-facing feature or fix

label

May 21, 2024
This should ensure that the project assets eventually get copied over to
the local Docker setup even under conditions where `kubectl exec` is
flaky and fails every couple of minutes.
Now that this is working, I can get rid of the `ls -lR` step (which is
effectively redundant anyway as `docker cp` is chatty about what files
it's copying), and enable the final cleanup of the temporary directory.

@rmunn

@rmunn

The docker cp command was preserving the UID/GID of the copied files
even though I didn't pass it the `-a` parameter (whose purpose is to
preserve the UID/GID of the copied files). To work around this issue, we
set the file ownership to 33/33 before copying the files into Docker.
The previous solution was too Linux-y; this one doesn't rely on `sudo`
or `id` working in a Git Bash environment on Windows.

@rmunn

Not done yet, so don't try to run this yet.
Now takes project ID or URL as parameter

@rmunn

megahirt

Was used during development as a way to test remote connection, no
longer needed.
Defaults to qa/staging for obvious reasons
Our staging server has a port defined on the db service, but our
production server does not. Switching to port forward to `deploy/db`,
which will automatically select the Mongo pod (which *does* have a port
open to forward to).
This will save a bit of time when kubectl cp is being reliable

Also force languageforge namespace just in case

@rmunn rmunn changed the title Bash script to copy projects from staging or prod Node script to copy projects from staging or prod

May 23, 2024

@rmunn

Before including pictures and audio in the tarball, make sure they're
really there, and skip them if they are a broken symlink.

hahn-kev

myieye

myieye

@rmunn

It's possible that Windows is doing something strange here that's
causing the `echo` tobe handled by the Windows shell instead of as part
of the kubectl input passed to `sh`. Switching to plain echo and then
stripping newlines from the result should produce the same result
without any cross-platform hiccups.

@rmunn

Windows has issues with single-quotes for quoting command-line params,
but thankfully Linux handles double-quotes correctly in all the places I
used single-quotes, so we'll just switch to double-quotes everywhere.
Mongo doesn't like it when you call `.insertMany` and pass it an empty
list. You'd think they would handle that case gracefully, but they don't
and Mongo throws an error "Invalid BulkOperation, Batch cannot be empty".
So we will skip calling `.insertMany` if there are no records to insert.

@rmunn

@rmunn

@rmunn

@rmunn

myieye

Also use pod name instead of deploy/app since not every user account has
access to deploy objects, at least on production

@rmunn

@myieye

@rmunn rmunn deleted the feat/backup-projects-to-local-mongodb branch

June 14, 2024 07:04

@rmunn rmunn mentioned this pull request

Jul 10, 2024