A scalable, distributed system for extracting and transforming LibreTexts content into various export formats.
Development
Shapeshift services are split into two containers, shapeshift-api and shapeshift-processor, which use MySQL for job data storage.
Ephermeral MySQL
If you want to start an ephermeral MySQL container for development, you can do so with:
# Start docker compose -f docker-compose-mysql.dev.yml up -d # Stop docker compose -f docker-compose-mysql.dev.yml down
Connecting to Locally Installed Resources
If you have an existing local MySQL or LocalStack installation running on your machine, you
can connect to them by enabling host networking mode. Add this config to each service in docker-compose.dev.yaml:
This will allow your Shapeshift containers to connect to the MySQL/LocalStack running on your device using localhost as the host name.
The Main Stack
After MySQL is set up, start/stop the stack with:
# Start docker compose -f docker-compose.dev.yaml up -d # Stop docker compose -f docker-compose.dev.yaml down
Run Development Build
You can use the run-dev-build.sh script to build and start the API and Processor containers with your local changes. The first build may take a few moments; subsequent builds will be faster.
Use Control-C to stop the containers.
AWS Emulation
LocalStack can be used to emulate AWS services (like SQS and Secrets Manager) locally as a
Docker container, allowing you to test the entire job workflow end-to-end. Create a free account and
install the CLI to get started. Set the AWS_REGION,
LOCALSTACK_HOST, and LOCALSTACK_PORT environment variables appropriately. Test data is stored on your machine unless
the Cloud Pods feature is used.
Running a Test Request
curl --request POST \ --url http://localhost:80/api/v1/job \ --header 'content-type: application/json' \ --data '{"url":"https://dev.libretexts.org/Sandboxes/eaturner_at_ucdavis.edu/Test_Book","highPriority":false}'