diff --git a/docs/dbdump_mongo.html b/docs/dbdump_mongo.html new file mode 100644 index 0000000..69013fb --- /dev/null +++ b/docs/dbdump_mongo.html @@ -0,0 +1,500 @@ + + + + + + MongoDB Dump Utility + + + +
+
+

MongoDB Dump Utility

+

Database Server Backup Management

+
+ +
+

Table of Contents

+ +
+ +
+
+

+ This directory contains a standalone tool for backing up MongoDB databases from remote or local servers. It manages multiple environment configurations for easy switching between prod, uat, and other environments. +

+
+ +
+

Quick Start

+ +

First Run

+ 
./db-dump.sh dump
+ +

The script will detect no environments and guide you through creating one. Answer the prompts with:

+
    +
  • Environment name (required): e.g., prod, uat, staging
    • +
  • MongoDB username (required): The admin user
    • +
  • MongoDB password (required): The admin password
    • +
  • MongoDB host (required): e.g., cluster0.mongodb.net
    • +
  • MongoDB port (required): Usually 27017 or 10255
    • +
  • Application name (required): Used in connection string and dump folder name
    • +
+ +

Subsequent Runs

+ 
./db-dump.sh dump
+ +

The script will show you a list of configured environments and let you choose which one to backup, or create a new environment.

+
+ +
+

Commands

+ 
# Perform a backup of a selected environment
+./db-dump.sh dump
+
+# List all configured environments
+./db-dump.sh list
+
+# Add a new environment
+./db-dump.sh add
+
+# Delete an environment
+./db-dump.sh delete
+
+# Show help
+./db-dump.sh help
+
+ +
+

How It Works

+
    +
  1. Environment Selection: On each run, you select from existing environments or create new ones
    2. +
  2. Credential Storage: Each environment's credentials are stored in .env.<name> files
    3. +
  3. MongoDB Connection: Connects using your credentials with MongoDB Atlas parameters (ssl, replicaSet, etc.)
    4. +
  4. Dump Creation: Uses mongodump to create a backup
    5. +
  5. Timestamped Output: Dumps are organized as ./dumps/<appName>/<yyyymmdd_hhmm>/
    6. +
+
+ +
+

Configuration Files

+ +

Each environment is stored in a separate file:

+ 
.env.prod       # Production environment
+.env.uat        # UAT environment
+.env.staging    # Staging environment (if created)
+ +

Each file contains:

+ 
MONGO_USERNAME=<username>
+MONGO_PASSWORD=<password>
+MONGO_HOST=<host>
+MONGO_PORT=<port>
+APP_NAME=<appName>
+
+ +
+

Directory Structure

+
dbdump/ +├── db-dump.sh # Main script +├── .env.prod # Production config (created by user) +├── .env.uat # UAT config (created by user) +├── .env.example # Example configuration +├── .gitignore # Git ignore rules +├── README.md # This file +└── dumps/ # Created automatically + ├── prod/ + │ ├── 20260414_1632/ + │ │ ├── admin/ + │ │ └── myapp/ + │ └── 20260414_1645/ + └── uat/ + └── 20260414_1632/
+
+ +
+

Dump Output

+ +

Dumps are stored with the following structure:

+ 
dumps/<appName>/<yyyymmdd_hhmi>/
+  admin/           # System databases
+  config/
+  myapp/           # Application databases
+  system.profile/
+ +

Each backup preserves the complete database structure and is ready to be restored using mongorestore.

+
+ +
+

Accessing Dumps

+ +

To view or restore a dump:

+ 
# List recent dumps
+ls -lh dumps/prod/
+
+# Restore a dump to local MongoDB
+mongorestore --uri "mongodb://localhost:27017" dumps/prod/20260414_1632/
+
+# Copy dump to another location
+cp -r dumps/prod/20260414_1632/ /backup/prod-backup-20260414/
+
+ +
+

Security

+
    +
  • Configuration files have restricted permissions (600)
    • +
  • Credentials are stored locally in .env.* files
    • +
  • These files are git-ignored for security
    • +
  • Never commit .env.* files to version control
    • +
+
+ +
+

Requirements

+ +
+ +
+

Installation of MongoDB Database Tools

+ +

macOS

+ 
brew tap mongodb/brew
+brew install mongodb-database-tools
+ +

Linux (Ubuntu/Debian)

+ 
wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.8.0.tgz
+tar -xzf mongodb-database-tools-ubuntu2004-x86_64-100.8.0.tgz
+sudo cp mongodb-database-tools-ubuntu2004-x86_64-100.8.0/bin/* /usr/local/bin/
+ +

Windows

+

Download the MSI installer from https://www.mongodb.com/try/download/database-tools and run it.

+
+ +
+

Troubleshooting

+ +

mongodump not found

+ 
# Check if MongoDB Database Tools is installed
+mongodump --version
+
+# If not installed, see "Installation" section above
+ +

Connection refused

+
    +
  • Verify host and port are correct
    • +
  • Check network connectivity to MongoDB server
    • +
  • Ensure firewall allows access
    • +
+ +

Authentication failed

+
    +
  • Verify username and password
    • +
  • Check the user has admin permissions
    • +
  • Confirm the authSource=admin parameter
    • +
+ +

Port already displaying in error

+

The connection string is checked for errors. If you see connection errors, the script displays the exact MongoDB error message.

+
+ +
+

Advanced Usage

+ +

Manual mongodump command (for reference)

+ +

The script constructs and runs this command automatically:

+ 
mongodump --uri "mongodb://<username>:<password>@<host>:<port>/?ssl=true&replicaSet=globaldb&retrywrites=false&maxIdleTimeMS=120000&appName=@<appName>@" -o ./dumps/<appName>/<yyyymmdd_hhmm>
+ +

Scheduling automated backups

+

You can schedule regular backups using cron (Linux/macOS):

+ 
# Add to crontab
+crontab -e
+
+# Daily dump at 3 AM for production environment
+0 3 * * * cd /path/to/dbdump && ./db-dump.sh dump << EOF
+1
+EOF
+
+ +
+

Support

+ +

For MongoDB connection issues, refer to:

+ +
+ +
+

About

+

+ Author: Julien Gautier (jgautier.webdev@gmail.com) +

+

+ Organization: AdAstra +

+

+ Project: adastra_scripts +

+

+ This utility was created to enable efficient backup and versioning of MongoDB databases from remote or local servers for AdAstra projects. +

+
+
+ + +
+ + \ No newline at end of file diff --git a/docs/dbrestore_mongo.html b/docs/dbrestore_mongo.html new file mode 100644 index 0000000..4933d0d --- /dev/null +++ b/docs/dbrestore_mongo.html @@ -0,0 +1,546 @@ + + + + + + MongoDB Restore Utility + + + +
+
+

MongoDB Restore Utility

+

Standalone Backup Restoration Tool

+
+ +
+

Table of Contents

+ +
+ +
+
+

+ This directory contains a standalone tool for restoring MongoDB database backups created with db-dump.sh. It provides interactive navigation through backup directories and restores them to a configured MongoDB instance. +

+
+ +
+

Quick Start

+ +

1. Configure Connection Details

+ 
cp .env.example .env
+ +

Edit .env with your MongoDB connection details:

+ 
MONGO_USERNAME=admin
+MONGO_PASSWORD=password
+MONGO_HOST=localhost
+MONGO_PORT=27017
+MONGO_DATABASE=adastradb
+MONGO_AUTHSOURCE=admin
+ +

2. Restore a Backup

+ 
./db-restore.sh
+ +

The script will:

+
    +
  1. Verify MongoDB tools are installed
    2. +
  2. Load your configuration
    3. +
  3. Ask for the path to your backups directory
    4. +
  4. Guide you through the directory structure
    5. +
  5. Show a summary of what will be restored
    6. +
  6. Ask for confirmation
    7. +
  7. Restore the backup
    8. +
+
+ +
+

How It Works

+ +

Configuration

+ +

The script requires a .env file with these variables:

+
    +
  • MONGO_USERNAME - MongoDB admin username
    • +
  • MONGO_PASSWORD - MongoDB admin password
    • +
  • MONGO_HOST - MongoDB server hostname/IP
    • +
  • MONGO_PORT - MongoDB server port
    • +
  • MONGO_DATABASE - Target database name
    • +
  • MONGO_AUTHSOURCE - Authentication source (usually admin)
    • +
+
+ +
+

Interactive Navigation

+ +

The script intelligently navigates through your backup directory structure:

+
dumps/ +├── prod/ - Choose this +│ ├── 20260414_1632/ - Then this +│ │ └── (database folders) - Then this +│ └── 20260414_1645/ +└── uat/ + └── 20260414_1700/
+ +

At each level, the script:

+
    +
  1. Shows available subdirectories
    2. +
  2. Asks you to choose one
    3. +
  3. Continues until it finds .bson files (backup data)
    4. +
+
+ +
+

Backup Detection

+ +

The script automatically stops navigation when it finds a directory containing .bson files, which are the actual backup data files.

+ +

Example backup structure:

+ 
~/Works/dbdump/dumps/
+└── bo-uat-cosmodb/
+    └── 20260414_1732/
+        └── admin/                 - BSON files here
+            ├── system.profile.bson
+            └── system.version.bson
+
+ +
+

Example Use Case

+ +

Given this backup structure:

+ 
~/Works/dbdump/dumps/
+├── bo-prod-cosmodb/
+│   └── 20260414_1600/
+│       └── AdAstra/
+│           ├── collection1.bson
+│           └── collection1.metadata.json
+└── bo-uat-cosmodb/
+    └── 20260414_1732/
+        └── AdAstra/
+            ├── orders.bson
+            └── orders.metadata.json
+ +

Restore process:

+ 
./db-restore.sh
+
+# Script asks: Dumps path: ~/Works/dbdump/dumps
+# Shows: 1) bo-prod-cosmodb  2) bo-uat-cosmodb
+# You choose: 2
+# Shows: 1) 20260414_1732
+# You choose: 1
+# Shows: 1) AdAstra
+# You choose: 1
+# Script finds BSON files and shows summary
+
+ +
+

Restore Summary

+ +

Before executing the restore, the script displays:

+ 
MongoDB Connection:
+  Host:       localhost
+  Port:       27017
+  Username:   admin
+  Database:   adastradb
+  AuthSource: admin
+
+Backup Source:
+  Location: /home/user/Works/dbdump/dumps/bo-uat-cosmodb/20260414_1732
+  Backup files: 42
+  Total size: 250M
+ +

You must confirm with "yes" to proceed.

+
+ +
+

Connection String

+ +

The script constructs the MongoDB connection string as:

+ 
mongodb://<username>:<password>@<host>:<port>/<database>?authSource=<authSource>
+ +

Example:

+ 
mongodb://admin:password@localhost:27017/adastradb?authSource=admin
+
+ +
+

Directory Structure

+
dbrestore/ +├── db-restore.sh # Main script +├── .env # Configuration (created by you, git-ignored) +├── .env.example # Example configuration +├── .gitignore # Git ignore rules +└── README.md # This file
+
+ +
+

Security

+
    +
  • Configuration file (.env) has restricted permissions
    • +
  • Contains sensitive credentials (username, password)
    • +
  • Never commit actual .env file to version control
    • +
  • Use .env.example as template
    • +
+
+ +
+

Requirements

+ +
+ +
+

Installation of MongoDB Database Tools

+ +

macOS

+ 
brew tap mongodb/brew
+brew install mongodb-database-tools
+ +

Linux (Ubuntu/Debian)

+ 
wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.8.0.tgz
+tar -xzf mongodb-database-tools-ubuntu2004-x86_64-100.8.0.tgz
+sudo cp mongodb-database-tools-ubuntu2004-x86_64-100.8.0/bin/* /usr/local/bin/
+ +

Windows

+

Download the MSI installer from https://www.mongodb.com/try/download/database-tools and run it.

+
+ +
+

Troubleshooting

+ +

mongorestore not found

+ 
# Check if MongoDB Database Tools is installed
+mongorestore --version
+
+# If not installed, see "Installation" section above
+ +

.env file not found

+ 
# Copy the example configuration
+cp .env.example .env
+
+# Edit with your values
+nano .env  # or your favorite editor
+ +

Connection refused

+
    +
  • Verify host and port are correct
    • +
  • Check network connectivity to MongoDB server
    • +
  • Ensure firewall allows access
    • +
  • Verify MongoDB service is running (if local)
    • +
+ +

Authentication failed

+
    +
  • Verify username and password in .env
    • +
  • Check user has sufficient permissions
    • +
  • Confirm authSource is correct (usually admin)
    • +
+ +

No BSON files found

+
    +
  • Make sure you're pointing to the root of dumps directory
    • +
  • Navigate through subdirectories until you reach the folder with actual backup files
    • +
  • BSON files are typically deep in the directory structure
    • +
+
+ +
+

Manual Restore Command

+ +

For reference, here's the command the script executes:

+ 
mongorestore --uri "mongodb://user:pass@host:port/db?authSource=admin" /path/to/dumps
+
+ + + +
+

Workflow Example

+ +

Complete workflow to backup and restore:

+ 
# 1. Backup from production MongoDB (in dbdump folder)
+cd ~/Works/dbdump
+./db-dump.sh dump
+# Select prod environment and wait for backup
+
+ +
+

About

+

+ Author: Julien Gautier (jgautier.webdev@gmail.com) +

+

+ Organization: AdAstra +

+

+ Project: adastra_scripts +

+

+ These utilities were created to simplify MongoDB management during local development of AdAstra projects. +

+
+
+ + +
+ + \ No newline at end of file diff --git a/docs/dbutils_mongo.html b/docs/dbutils_mongo.html new file mode 100644 index 0000000..c744e78 --- /dev/null +++ b/docs/dbutils_mongo.html @@ -0,0 +1,513 @@ + + + + + + MongoDB Database Utility (Standalone) + + + +
+
+

MongoDB Database Utility

+

Standalone Local Development Setup

+
+ +
+

Table of Contents

+ +
+ +
+
+

+ This directory contains a standalone MongoDB setup for local development. It is completely independent from the main backend and frontend projects. +

+
+ +
+

Quick Start

+ +

First Run (Setup)

+ 
./db-start.sh up
+ +

The script will prompt you for:

+
    +
  • MongoDB username (default: admin)
    • +
  • MongoDB password (default: password)
    • +
  • Database name (default: adastradb)
    • +
  • MongoDB port (default: 27017)
    • +
  • Mongo Express port (default: 8081)
    • +
+ +

Your configuration will be stored in .env file (automatically git-ignored).

+
+ +
+

Common Commands

+ 
# Start MongoDB (prompts for credentials on first run)
+./db-start.sh up
+
+# View logs
+./db-start.sh logs
+
+# Stop MongoDB
+./db-start.sh down
+
+# Restart MongoDB
+./db-start.sh restart
+
+# Check container status
+./db-start.sh status
+
+# Reconfigure credentials
+./db-start.sh reconfigure
+
+# Remove all containers and volumes (DATA LOSS!)
+./db-start.sh clean
+
+# Show help
+./db-start.sh help
+
+ +
+

Accessing MongoDB

+ +

From Your Application

+
    +
  • Connection String: mongodb://<username>:<password>@localhost:27017/<database>?authSource=admin
    • +
  • Example: mongodb://admin:password@localhost:27017/adastradb?authSource=admin
    • +
+ +

Using Mongo Express (GUI)

+ +
+ +
+

Data Persistence

+

All data is stored in Docker volumes and persists across container restarts. To permanently delete data, use:

+ 
./db-start.sh clean
+
+ +
+

Collections

+

The database is automatically initialized with the following collection:

+
    +
  • dataImportHistory: Stores records of all data imports for audit and recovery tracking +
      +
    • Indexed on importDate (descending) and status
      • +
    • Ready for future data import functionality
      • +
    +
    • +
+
+ +
+

File Structure

+
dbutils/ +├── db-start.sh # Main script +├── docker-compose-mongodb.yml # Docker Compose configuration +├── mongo-init.js # Database initialization script +├── .env # Configuration (generated on first run, git-ignored) +├── .env.example # Example configuration +├── .gitignore # Git ignore rules +└── README.md # This file
+
+ +
+

Configuration

+

The script uses a .env file to store configuration. You can:

+
    +
  1. Let the script generate it: Run ./db-start.sh up and answer the prompts
    2. +
  2. Pre-create it: Copy .env.example to .env and modify values
    3. +
  3. Edit it manually: Edit .env directly and run ./db-start.sh up
    4. +
  4. Reconfigure: Run ./db-start.sh reconfigure to re-run the setup wizard
    5. +
+
+ +
+

Requirements

+
    +
  • Docker
    • +
  • Docker Compose (included with Docker Desktop)
    • +
  • Bash shell
    • +
+
+ +
+

Standalone Usage

+

This utility can be used independently from AdAstra projects:

+ 
# From anywhere
+cd /path/to/dbutils
+./db-start.sh up
+
+# Or add to PATH and use globally (optional)
+export PATH=$PATH:/path/to/dbutils
+db-start.sh up
+
+ +
+

Troubleshooting

+ +

Port already in use

+

If MongoDB or Mongo Express port is already in use:

+
    +
  1. Run ./db-start.sh reconfigure
    2. +
  2. Choose a different port (e.g., 27018 for MongoDB, 8082 for Mongo Express)
    3. +
+ +

Docker not running

+ 
# Check Docker status
+docker info
+ +

Permission denied

+ 
# Make script executable
+chmod +x db-start.sh
+
+ +
+

Notes

+
    +
  • Credentials are stored in .env (git-ignored for security)
    • +
  • Mongo Express is included for easy database management
    • +
  • Data persists in Docker volumes
    • +
  • No initialization scripts are run (unlike the backend docker-compose)
    • +
  • This is standalone and independent from the main project configuration
    • +
+
+ +
+

About

+

+ Author: Julien Gautier (jgautier.webdev@gmail.com) +

+

+ Organization: AdAstra +

+

+ Project: adastra_scripts +

+

+ These utilities were created to simplify MongoDB management during local development of AdAstra projects. +

+
+
+ + +
+ + \ No newline at end of file