Development environment setup
This document will help you setup your development environment so that you can build, test and run CLOMonitor locally from source.
For more information about the CLOMonitor’s architecture, please see this document.
The instructions provided in this document rely on a set of aliases available at the end. These aliases are used by some of the maintainers and are provided only as examples. Please feel free to adapt them to suit your needs. You may want to add them to your shell’s configuration file so that they are loaded automatically.
To start, please clone the CLOMonitor repository. If you plan to use the aliases mentioned above, you should set the
CLOMONITOR_SOURCE variable to the path where you cloned the repository.
The datastore used by CLOMonitor is PostgreSQL. You can install it locally using your favorite OS package manager.
Once PostgreSQL is installed and its binaries are available in your
PATH, we can initialize the database cluster and launch the database server:
Once the database server is up an running, we can create the
clomonitor database and we’ll be ready to go:
We need to create a configuration file so that Tern knows how to connect to our database. We’ll create a file called
~/.config/clomonitor with the following content (please adjust if needed):
[database] host = localhost port = 5432 database = clomonitor user = postgres
Now that the
clomonitor database server is up and ready, we just need to apply all available migrations using the following command:
If you plan to do some work on the database layer, some extra setup is needed to be able to run the database tests. Schema and database functions are tested using the unit testing framework pgTap, so you need to install the pgTap PostgreSQL extension on your machine. To run the tests you will also need to install a perl tool called pg_prove from CPAN (
Similarly to what we did during our initial database setup, we’ll create a configuration file for Tern for the tests database in the same folder (
tern-tests.conf with the following content (please adjust if needed):
[database] host = localhost port = 5432 database = clomonitor_tests user = postgres
Once you have all the tooling required installed and the tests database set up, you can run all database tests as often as you need this way:
clomonitor_db_recreate_tests && clomonitor_db_tests
You can load some sample data by using the
psql PostgreSQL client this way:
Once you’ve connected to the database, you can run the following command to load one sample foundation:
insert into foundation values ('cncf', 'CNCF', 'https://raw.githubusercontent.com/cncf/clomonitor/main/data/cncf.yaml');
registrar component will process the foundations’ data files and register the projects available.
At this point our database is ready to launch our local instance of CLOMonitor and start doing some work on it.
To build the backend components, please run the command below:
Even if you don’t plan to do any work on the frontend, you will need to build it once if you want to interact with the CLOMonitor backend from the browser. To do this, you will have to install yarn. Once you have it installed, you can build the frontend application this way:
cd web && yarn install clomonitor_frontend_build
Once you have a working Rust development environment set up and the web application built, it’s time to launch the
apiserver. Before running it, we’ll need to create a configuration file in
apiserver.yaml with the following content (please adjust
staticPath as needed):
db: host: localhost port: "5432" dbname: clomonitor user: postgres password: "" apiserver: addr: 127.0.0.1:8000 staticPath: /<YOUR_CLOMONITOR_LOCAL_PATH>/web/build
Now you can run the
apiserver process launches an http server that serves the web application and the API that powers it. Once it is up and running, you can point your browser to http://localhost:8000 and you should see the CLOMonitor web application. Initially there won’t be any projects listed on it, but we’ll take care of that in the next section.
registrar is a backend component responsible for registering the projects available in each foundation’s data file into the database. Foundations’ data files are expected to be served by an HTTP server, and their url should be provided to CLOMonitor when registering the foundation in the database. On production deployments, it is usually run periodically using a
cronjob on Kubernetes. Locally, while developing, you can just run it as often as you need as any other CLI tool.
This component can be configured using a
yaml file. We’ll create one in
registrar.yaml with the following content (please adjust as needed):
db: host: localhost port: "5432" dbname: clomonitor user: postgres password: "" registrar: concurrency: 1
Once the configuration file is ready, it’s time to launch the
registrar for the first time. If you added the suggested sample foundation when setting up the database, you should see some projects registered.
tracker is a backend component in charge of linting the repositories registered in the database and updating the scores and ratings as needed. On production deployments, it is usually run periodically using a
cronjob on Kubernetes. Locally, while developing, you can just run it as often as you need as any other CLI tool. The tracker requires the
git command to be installed and available in your PATH.
If you opened the url suggested before, you probably noticed there were no projects listed yet. This happened because no repositories had been processed yet. When we set up the database, we loaded some sample repositories. To process them, we need to run the
Similarly to the
apiserver server, the
tracker can be configured using a
yaml file. We’ll create one in
tracker.yaml with the following content (please adjust as needed):
db: host: localhost port: "5432" dbname: clomonitor user: postgres password: "" creds: githubTokens: - <YOUR_GITHUB_TOKEN> tracker: concurrency: 10
Some checks use the Github GraphQL API, which requires authentication, so you’ll need to add your own Github token to the
tracker configuration file.
Once the configuration file is ready, it’s time to launch the
tracker for the first time:
Depending on the speed of your Internet connection and machine, this may take one or two minutes. The first time it runs all repositories will be linted. Subsequent runs will only lint repositories that have changed, so it’ll be much faster. Once the tracker has completed, you should see projects in the web application.
In the section above we saw how the
tracker is able to lint all repositories registered in the database. But sometimes it may be desirable to lint a single repository manually in an isolated way, maybe to quickly test some checks or to integrate with some other processes, like continuous integration or deployment tools. The
linter CLI tool is designed to help in those scenarios.
CLOMonitor delegates some of the security checks to OpenSSF Scorecard, so you’ll need to install it before running
clomonitor-linter locally. Both CLOMonitor and OpenSSF Scorecard use the Github GraphQL API for some checks, which requires authentication. A Github token (with
public_repo scope) must be provided via the
GITHUB_TOKEN environment variable to authenticate those requests.
If you are using the aliases provided below, you can run it this way:
export GITHUB_TOKEN=<your token> clomonitor_linter --help
You can run the backend tests by using
In the backend section we mentioned how to install the frontend dependencies and build it. That should be enough if you are only going to work on the backend. However, if you are planning to do some work on the frontend, it’s better to launch an additional server which will rebuild the web application as needed whenever a file is modified.
The frontend development server can be launched using the following command:
That alias will launch an http server that will listen on the port
3000. Once it’s running, you can point your browser to http://localhost:3000 and you should see the CLOMonitor web application. The page will be automatically reloaded everytime you make a change in the frontend code. Build errors and build warnings will be visible in the console.
You can use the command below to run all frontend tests:
In addition to running the tests, you may also be interested in running the linter. To do that, you can run:
The following aliases are used by some of the maintainers and are provided only as examples. Please feel free to adapt them to suit your needs.
export CLOMONITOR_SOURCE=~/projects/clomonitor export CLOMONITOR_DATA=~/tmp/data_clomonitor alias clomonitor_db_init="mkdir -p $CLOMONITOR_DATA && initdb -U postgres $CLOMONITOR_DATA" alias clomonitor_db_create="psql -U postgres -c 'create database clomonitor'" alias clomonitor_db_create_tests="psql -U postgres -c 'create database clomonitor_tests' && psql -U postgres clomonitor_tests -c 'create extension if not exists pgtap'" alias clomonitor_db_drop="psql -U postgres -c 'drop database clomonitor with (force)'" alias clomonitor_db_drop_tests="psql -U postgres -c 'drop database if exists clomonitor_tests'" alias clomonitor_db_recreate="clomonitor_db_drop && clomonitor_db_create && clomonitor_db_migrate" alias clomonitor_db_recreate_tests="clomonitor_db_drop_tests && clomonitor_db_create_tests && clomonitor_db_migrate_tests" alias clomonitor_db_server="postgres -D $CLOMONITOR_DATA" alias clomonitor_db_client="psql -h localhost -U postgres clomonitor" alias clomonitor_db_migrate="pushd $CLOMONITOR_SOURCE/database/migrations; TERN_CONF=~/.config/clomonitor/tern.conf ./migrate.sh; popd" alias clomonitor_db_migrate_tests="pushd $CLOMONITOR_SOURCE/database/migrations; TERN_CONF=~/.config/clomonitor/tern-tests.conf ./migrate.sh; popd" alias clomonitor_db_tests="pushd $CLOMONITOR_SOURCE/database/tests; pg_prove --host localhost --dbname clomonitor_tests --username postgres --verbose **/*.sql; popd" alias clomonitor_apiserver="$CLOMONITOR_SOURCE/target/debug/clomonitor-apiserver -c ~/.config/clomonitor/apiserver.yaml" alias clomonitor_registrar="$CLOMONITOR_SOURCE/target/debug/clomonitor-registrar -c ~/.config/clomonitor/registrar.yaml" alias clomonitor_tracker="$CLOMONITOR_SOURCE/target/debug/clomonitor-tracker -c ~/.config/clomonitor/tracker.yaml" alias clomonitor_linter="$CLOMONITOR_SOURCE/target/debug/clomonitor-linter" alias clomonitor_frontend_build="pushd $CLOMONITOR_SOURCE/web; yarn build; popd" alias clomonitor_frontend_dev="pushd $CLOMONITOR_SOURCE/web; yarn start; popd" alias clomonitor_frontend_tests="pushd $CLOMONITOR_SOURCE/web; yarn test; popd" alias clomonitor_frontend_lint_fix="pushd $CLOMONITOR_SOURCE/web; yarn lint:fix; popd"