Access¶
The repository of the MatD3 database is hosted by GitHub:
It’s a public project, so anybody is able to clone the source code and create issues and merge requests. In order to contribute to the project directly, contact one of the MatD3 team members who will then grant you the developer status. For pull/push privileges you must upload your RSA public key to GitHub. If you already have one, it is likely located at ~/.ssh/id_rsa.pub
. If not, the RSA public/private key pair can be generated by issuing
ssh-keygen
It is sufficient to press enter at each query until the keys have been generated. However, protecting the key with a password would be a good idea if you intend to keep it on a public server. Then, at the GitHub website, go to your settings, find “SSH keys”, and enter the public key. You can upload several keys in order to have pull/push access from more than one computer.
From a command line prompt (i.e., using a terminal application, which is usually provided along with any Unix/Linux distribution or with MacOS), you can now clone the Git repository with
git clone https://github.com/HybriD3-database/MatD3.git
The result of this command should be the creation of a directory MatD3
that contains the software needed to operate the MatD3 database.
Setup¶
A detailed set of setup notes is included in the file Readme.md
that is provided directly with the source code. You can access these instructions here:
https://github.com/HybriD3-database/MatD3/blob/master/Readme.md
Before proceeding, you will typically need a few other software packages to be installed. For instance, you will typically want to install a SQL database, program langage headers and a driver that provides a HTTP interface, needed in order to display the output of MatD3 in a browser.
For example, on Ubuntu, you may need:
sudo apt install libmysqlclient-dev
sudo apt install python3-dev
sudo apt install firefox-geckodriver
On MacOS, open source software is provided by the Homebrew project, so you may need to install Homebrew and, once that is done, install
brew install mariadb
brew install geckodriver
With the prerequisites installed, change to the directory MatD3
by indicating
cd MatD3
In order to ensure compatibility with the correct Python packages, it is important to do any development in a Python virtual environment. You can create a virtual environment with
python3 -m venv venv
and activate the environment with
source venv/bin/activate
In some instances of python, the relevant command might be found in a directory called venv/Scripts/activate
instead.
You’ll notice that the shell’s prompt has changed to remind you that you are in a virtual environment. Any packages installed with Python’s pip
command are now part of the current project only.
Typically, you will also need to update the pip command to a current version:
pip install --upgrade pip
The correct versions of the packages that are required for developing the MatD3 database are listed in requirements.txt
, which is a file located in the MatD3
directory. From a command line prompt with the MatD3
directory as the working directory, install the requirements by issuing
pip install -r requirements.txt
When you make changes to the requirements, such as upgrading a Python package, use
pip freeze > requirements.txt
to record the new list of requirements. Since this will overwrite the current requirements file, it is important to apply the old requirements first before making any changes.
Before starting the server, it is necessary to configure it in order to reflect your specific environment. The configuration variables are read from a file called .env
in the root directory. You need to create this file or make a copy of env.example
in the root directory and edit it.
- SECRET_KEY
- Used for cryptographic signing. The default value is only for quickly setting up the server and should not be used in practice.
- MATD3_NAME
- Name of this instance of the server
- MATD3_URL
- URL of this instance of the server
- ALLOWED_HOSTS
- List of host/domain names that this instance can serve
- EMAIL_HOST
- The host to use for sending email.
- EMAIL_HOST_USER
- Username to use for the SMTP server defined in EMAIL_HOST
- EMAIL_HOST_PASSWORD
- Password to use for the SMTP server defined in EMAIL_HOST
- SELENIUM_DRIVER
- Which driver to use for running tests with Selenium. Options are “Firefox” and “Chrome” (case insensitive). If not present, Firefox is used.
- USE_SQLITE
- Whether to use the SQLite database. If false or not present, mySQL is used instead.
- DEBUG
- Whether to run MatD3 in debug mode. This is useful for quickly setting up and testing the website but should be removed when serving on a production server.
Some troubleshooting notes¶
The packages listed in requirements.txt
are many. The pip
command should handle incompatibilities, but it does not handle all of them and package incompatibilities can still creep in. If that happens, the output of pip
can, unfortunately, be rather cryptic.
One common consequence of such incompatbilities is that the web server does not start up properly and all you see from your browser is a “502 error” without any useful details.
In this case, the next way forward is to manually log in to the server in question and attempt to understand errors that the different components of Django produce.
Specifically, Django can run based on a combination of servers called nginx
and gunicorn
. The following blog post explains their interaction and, importantly, where the log files are and which commands can be used for troubleshooting:
At this point, the exercise becomes one of Linux administration and starting and stopping services on the server that runs the database. For example, the sysctl
command (which controls services running on a particular server) becomes involved.
For instance, this command here, executed at the command line, will try to start a MatD3 service:
sudo systemctl status (name).service -l
where “(name)” needs to be replaced by the specific name chosen during the server setup of your particular MatD3 instance.
The following command, executed in the directory “/var/www/MatD3”, restarts the gunicorn service manually:
sudo /var/www/MatD3/venv/bin/gunicorn -t 3600 --workers 2 --bind unix:/run/(name).sock mainproject.wsgi
(Again, “(name)” needs to be replaced by the specific name chosen during the server setup.)