Node script to copy projects from staging or prod by rmunn · Pull Request #1816 · sillsdev/web-languageforge
Often fails to copy assets because `kubectl cp` or `kubectl exec tar` get cut off partway through, but that should go away once Kubernetes version 1.30 is released.
added
the
engineering
label
May 21, 2024This should ensure that the project assets eventually get copied over to the local Docker setup even under conditions where `kubectl exec` is flaky and fails every couple of minutes.
Now that this is working, I can get rid of the `ls -lR` step (which is effectively redundant anyway as `docker cp` is chatty about what files it's copying), and enable the final cleanup of the temporary directory.
The docker cp command was preserving the UID/GID of the copied files even though I didn't pass it the `-a` parameter (whose purpose is to preserve the UID/GID of the copied files). To work around this issue, we set the file ownership to 33/33 before copying the files into Docker.
Our staging server has a port defined on the db service, but our production server does not. Switching to port forward to `deploy/db`, which will automatically select the Mongo pod (which *does* have a port open to forward to).
rmunn
changed the title
Bash script to copy projects from staging or prod
Node script to copy projects from staging or prod
Before including pictures and audio in the tarball, make sure they're really there, and skip them if they are a broken symlink.
It's possible that Windows is doing something strange here that's causing the `echo` tobe handled by the Windows shell instead of as part of the kubectl input passed to `sh`. Switching to plain echo and then stripping newlines from the result should produce the same result without any cross-platform hiccups.
Windows has issues with single-quotes for quoting command-line params, but thankfully Linux handles double-quotes correctly in all the places I used single-quotes, so we'll just switch to double-quotes everywhere.
Mongo doesn't like it when you call `.insertMany` and pass it an empty list. You'd think they would handle that case gracefully, but they don't and Mongo throws an error "Invalid BulkOperation, Batch cannot be empty". So we will skip calling `.insertMany` if there are no records to insert.
Also use pod name instead of deploy/app since not every user account has access to deploy objects, at least on production
rmunn
deleted the
feat/backup-projects-to-local-mongodb
branch
rmunn
mentioned this pull request
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters