This document contains information related to developing the Druzhba application, specifically how to run unit and integration tests and the release procedure for publishing a new version of Druzha. It will be of minimal use to the average end user.
Unsolicited pull requests may be submitted against the
master branch. Please
run Isort and Black (See:
.pre-commit.sh for commands or better yet copy the
.git/hooks/pre-commit.sh to run them automatically) prior to opening
your pull request and ensure that unit tests pass. Maintainers will run the
integration test suite if appropriate.
This is a list of most requested Druzhba features:
Complete SQL Server support (currently partial support)
Support for multiple output database types
Unrecognized table names in supplied via the
--tablesargument should be surfaced as a warning or error
Add support to allow basic transformations from YAML configs (i.e. renaming or concatenating columns) without requiring a custom SQL table.
The table extract functionality is tied unnecessarily to the name of the table in the source database, with a fake source table name supplied for manual queries. This prevents one source table from being copied to multiple destination tables with, for example, different subsets of columns.
On the first run of a new source table, Druzhba should create the destination table even if the query against the source table returned no rows. The current behavior should be considered a bug.
All major new functionality is expected to have adequate test coverage.
To run unit tests locally:
python setup.py test
To use Docker for testing, create a .env.test file based on env.test.sample with any environment variable overrides you wish to assign (can be empty).
To run unit tests in Docker:
docker-compose run test
To run integration tests of full pipelines in Docker, you’ll need to add Redshift credentials to your environment or your .env.test file. This makes use of a test schema in an existing Redshift database, and for safety will fail if the schema name already exists.
source .env.test.sample source .env.test # For whatever overrides you need docker-compose up -d postgres mysql docker-compose build test docker-compose run test bash test/integration/run_test.sh mysql postgres