Access

The repository of the MatD3 database is hosted by GitHub:

It’s a public project, so anybody is able to clone the source code and create issues and merge requests. In order to contribute to the project directly, contact one of the MatD3 team members who will then grant you the developer status. For pull/push privileges you must upload your RSA public key to GitHub. If you already have one, it is likely located at ~/.ssh/id_rsa.pub. If not, the RSA public/private key pair can be generated by issuing

ssh-keygen

It is sufficient to press enter at each query until the keys have been generated. However, protecting the key with a password would be a good idea if you intend to keep it on a public server. Then, at the GitHub website, go to your settings, find “SSH keys”, and enter the public key. You can upload several keys in order to have pull/push access from more than one computer.

From a command line prompt (i.e., using a terminal application, which is usually provided along with any Unix/Linux distribution or with MacOS), you can now clone the Git repository with

git clone https://github.com/HybriD3-database/MatD3.git

The result of this command should be the creation of a directory MatD3 that contains the software needed to operate the MatD3 database.

Setup

A detailed set of setup notes is included in the file Readme.md that is provided directly with the source code. You can access these instructions here:

https://github.com/HybriD3-database/MatD3/blob/master/Readme.md

Before proceeding, you will typically need a few other software packages to be installed. For instance, you will typically want to install a SQL database, program langage headers and a driver that provides a HTTP interface, needed in order to display the output of MatD3 in a browser.

For example, on Ubuntu, you may need:

sudo apt install libmysqlclient-dev
sudo apt install python3-dev
sudo apt install firefox-geckodriver

On MacOS, open source software is provided by the Homebrew project, so you may need to install Homebrew and, once that is done, install

brew install mariadb
brew install geckodriver

With the prerequisites installed, change to the directory MatD3 by indicating

cd MatD3

In order to ensure compatibility with the correct Python packages, it is important to do any development in a Python virtual environment. You can create a virtual environment with

python3 -m venv venv

and activate the environment with

source venv/bin/activate

In some instances of python, the relevant command might be found in a directory called venv/Scripts/activate instead.

You’ll notice that the shell’s prompt has changed to remind you that you are in a virtual environment. Any packages installed with Python’s pip command are now part of the current project only.

Typically, you will also need to update the pip command to a current version:

pip install --upgrade pip

The correct versions of the packages that are required for developing the MatD3 database are listed in requirements.txt, which is a file located in the MatD3 directory. From a command line prompt with the MatD3 directory as the working directory, install the requirements by issuing

pip install -r requirements.txt

When you make changes to the requirements, such as upgrading a Python package, use

pip freeze > requirements.txt

to record the new list of requirements. Since this will overwrite the current requirements file, it is important to apply the old requirements first before making any changes.

Before starting the server, it is necessary to configure it in order to reflect your specific environment. The configuration variables are read from a file called .env in the root directory. You need to create this file or make a copy of env.example in the root directory and edit it.

SECRET_KEY
Used for cryptographic signing. The default value is only for quickly setting up the server and should not be used in practice.
MATD3_NAME
Name of this instance of the server
MATD3_URL
URL of this instance of the server
ALLOWED_HOSTS
List of host/domain names that this instance can serve
EMAIL_HOST
The host to use for sending email.
EMAIL_HOST_USER
Username to use for the SMTP server defined in EMAIL_HOST
EMAIL_HOST_PASSWORD
Password to use for the SMTP server defined in EMAIL_HOST
SELENIUM_DRIVER
Which driver to use for running tests with Selenium. Options are “Firefox” and “Chrome” (case insensitive). If not present, Firefox is used.
USE_SQLITE
Whether to use the SQLite database. If false or not present, mySQL is used instead.
DEBUG
Whether to run MatD3 in debug mode. This is useful for quickly setting up and testing the website but should be removed when serving on a production server.

Some troubleshooting notes

The packages listed in requirements.txt are many. The pip command should handle incompatibilities, but it does not handle all of them and package incompatibilities can still creep in. If that happens, the output of pip can, unfortunately, be rather cryptic.

One common consequence of such incompatbilities is that the web server does not start up properly and all you see from your browser is a “502 error” without any useful details.

In this case, the next way forward is to manually log in to the server in question and attempt to understand errors that the different components of Django produce.

Specifically, Django can run based on a combination of servers called nginx and gunicorn. The following blog post explains their interaction and, importantly, where the log files are and which commands can be used for troubleshooting:

At this point, the exercise becomes one of Linux administration and starting and stopping services on the server that runs the database. For example, the sysctl command (which controls services running on a particular server) becomes involved.

For instance, this command here, executed at the command line, will try to start a MatD3 service:

sudo systemctl status (name).service -l

where “(name)” needs to be replaced by the specific name chosen during the server setup of your particular MatD3 instance.

The following command, executed in the directory “/var/www/MatD3”, restarts the gunicorn service manually:

sudo /var/www/MatD3/venv/bin/gunicorn -t 3600 --workers 2 --bind unix:/run/(name).sock mainproject.wsgi

(Again, “(name)” needs to be replaced by the specific name chosen during the server setup.)