Local Development Environment
Here we describe how to set up a local development environment to test the different parts of the sensitivity calculator. This is in general only required to gain fine grained control over the different development stages.
The implementation details are described in Implementation.
Outline and dependencies
The backend libraries have the following Python dependencies:
numpy
scipy
astropy
astropy_healpix
The frontend server is implemented using Flask
and has as an extra dependency marshmallow
.
The frontend uses bootstrap 4
and custom libraries written in TypeScript.
The assets are managed by a Node script that compiles the TypeScript libraries.
The documentation uses Sphinx and is written in rst format. The Python dependencies to build the documentation are:
sphinx
sphinx_rtd_theme
recommonmark
Python and Typescript development
The Python version can be managed with pyenv which allows the installation and use of a specific Python version for the project. This could be later used to test the code, particularly the backend, in different Python versions. The additional extension pyenv-virtualenv can be used to manage virtual environments in combination with pyenv.
After installing pyenv and pyenv-virtualenv the following commands can be executed from the project root directory to install a Python 3.7 version that will be used locally for the project. For example:
pyenv install 3.7.8
pyenv local 3.7.8
A .python-version file will be created in the root folder with the Python version that will be used.
The virtual environment and dependencies are managed by Poetry which allows the pinning of dependencies and to resolve possible conflicts. It can also prepare the local virtual environment and manage the building and distribution of the project. The configuration is written in the pyproject.toml file (see PEP-518).
We can set poetry to use a local virtual environment with:
poetry config virtualenvs.in-project true
poetry config virtualenvs.create true
The dependencies and virtual environment can be installed with:
poetry install
Please, note that Poetry is not using the Python registry defined by the PIP_INDEX_URL
environment variable at the
moment. This parameter is set in the pyproject.toml
section called tool.poetry.source
.
After that, all the commands can be run prepending poetry run before.
The frontend assets are managed with Node.js which is used, at the moment, to
install and run the TypeScript tools, the testing environment cypress
, and the Typescript tests. The Node.js
versions can be managed with nvm with the latest stable LTS version being v12.18.4
(Erbium). After installing nvm, the required version of Node.js can be installed with:
nvm install v14.16.1
nvm use v14.16.1
To make npm work with the SKA Central Artefact Reository npm registry we can set the environment variable to the appropriate value:
export npm_config_registry=https://artefact.skao.int/repository/npm-all/
The TypeScript dependencies can be installed using:
npm install --no-cache
The TypeScript assets can be compiled using a Node.js script (defined in the file package.json
; see for example
this link) which can be
run with:
npm run build
This command will run the necessary scripts to transpile the TypeScript code to es6 Javascript.
To run the Flask server
poetry run flask run
Documentation
Before compiling the documentation make sure that the libraries required are installed with:
poetry install -E docs
Then, the documentation can be compiled from the root directory running the following command:
poetry run make -C ./docs html
The documentation will be located in docs/build/html
and from this directory it can be inspected locally with a
command like:
python -m http.server 8020
and opening a browser tab in the address http://localhost:8020/
Tests with Cypress
Cypress is a testing framework that can be easily installed using Node.js. Cypress API is a chaining API. See: https://docs.cypress.io/api
Install Cypress with:
npm install --save-dev cypress
Before running cypress, the sensitivity calculator must be running using, for example, make up
.
To open the cypress GUI:
npx cypress open
To run in a headless mode like in CI:
npx cypress run
All the tests are saved in the cypress
directory.
Base Docker images
The project is installed in base Docker images that are fetched from the
Central Artefact Repository (CAR).
These images contain the basic dependencies for the project to run or to be tested. They are also used by the CI
pipelines which are also used to create them when necessary (in the build_base_images
step).
The creation of these images is handled by the code located in the directory called docker
. To trigger the creation
and pushing of a new release in the CI pipeline, the version of the release must be updated in the file .release
.
At the moment there are two types of images that can be created: the production
image with just the basic
dependencies, and the e2e
image that contains some additional dependencies to help the running of the End 2 End
tests with Cypress.
The creation of a new base image can be tested locally using the command:
make release
from the docker
directory. However, the uploading to the CAR should fail as the credentials will only be available
in the CI environment.