Local Development Environment

Here we describe how to set up a local development environment to test the different parts of the sensitivity calculator. This is in general only required to gain fine grained control over the different development stages.

The implementation details are described in Implementation.

Outline and dependencies

The backend libraries have the following Python dependencies:

• numpy

• scipy

• astropy

• astropy_healpix

The frontend server is implemented using Flask and has as an extra dependency marshmallow.

The frontend uses bootstrap 4 and custom libraries written in TypeScript. The assets are managed by a Node script that compiles the TypeScript libraries.

The documentation uses Sphinx and is written in rst format. The Python dependencies to build the documentation are:

• sphinx

• sphinx_rtd_theme

• recommonmark

Python and Typescript development

The Python version can be managed with pyenv which allows the installation and use of a specific Python version for the project. This could be later used to test the code, particularly the backend, in different Python versions. The additional extension pyenv-virtualenv can be used to manage virtual environments in combination with pyenv.

After installing pyenv and pyenv-virtualenv the following commands can be executed from the project root directory to install a Python 3.7 version that will be used locally for the project. For example:

pyenv install 3.7.8
pyenv local 3.7.8

A .python-version file will be created in the root folder with the Python version that will be used.

The virtual environment and dependencies are managed by Poetry which allows the pinning of dependencies and to resolve possible conflicts. It can also prepare the local virtual environment and manage the building and distribution of the project. The configuration is written in the pyproject.toml file (see PEP-518).

We can set poetry to use a local virtual environment with:

poetry config virtualenvs.in-project true
poetry config virtualenvs.create true

The dependencies and virtual environment can be installed with:

poetry install

Please, note that Poetry is not using the Python registry defined by the PIP_INDEX_URL environment variable at the moment. This parameter is set in the pyproject.toml section called tool.poetry.source.

After that, all the commands can be run prepending poetry run before.

The frontend assets are managed with Node.js which is used, at the moment, to install and run the TypeScript tools, the testing environment cypress, and the Typescript tests. The Node.js versions can be managed with nvm with the latest stable LTS version being v12.18.4 (Erbium). After installing nvm, the required version of Node.js can be installed with:

nvm install v14.16.1
nvm use v14.16.1

To make npm work with the SKA Central Artefact Reository npm registry we can set the environment variable to the appropriate value:

export npm_config_registry=https://artefact.skao.int/repository/npm-all/

The TypeScript dependencies can be installed using:

npm install --no-cache

The TypeScript assets can be compiled using a Node.js script (defined in the file package.json; see for example this link) which can be run with:

npm run build

This command will run the necessary scripts to transpile the TypeScript code to es6 Javascript.

To run the Flask server

poetry run flask run

Documentation

Before compiling the documentation make sure that the libraries required are installed with:

poetry install -E docs

Then, the documentation can be compiled from the root directory running the following command:

poetry run make -C ./docs html

The documentation will be located in docs/build/html and from this directory it can be inspected locally with a command like:

python -m http.server 8020

and opening a browser tab in the address http://localhost:8020/

Tests with Cypress

Cypress is a testing framework that can be easily installed using Node.js. Cypress API is a chaining API. See: https://docs.cypress.io/api

Install Cypress with:

npm install --save-dev cypress

Before running cypress, the sensitivity calculator must be running using, for example, make up.

To open the cypress GUI:

npx cypress open

To run in a headless mode like in CI:

npx cypress run

All the tests are saved in the cypress directory.

Base Docker images

The project is installed in base Docker images that are fetched from the Central Artefact Repository (CAR). These images contain the basic dependencies for the project to run or to be tested. They are also used by the CI pipelines which are also used to create them when necessary (in the build_base_images step).

The creation of these images is handled by the code located in the directory called docker. To trigger the creation and pushing of a new release in the CI pipeline, the version of the release must be updated in the file .release. At the moment there are two types of images that can be created: the production image with just the basic dependencies, and the e2e image that contains some additional dependencies to help the running of the End 2 End tests with Cypress.

The creation of a new base image can be tested locally using the command:

make release

from the docker directory. However, the uploading to the CAR should fail as the credentials will only be available in the CI environment.