rnsrk/wisski-cloud-distillery
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Switch to using Docker

Browse source 
This commit refactors all code in this project to make use of docker.
This has not yet been documented properly.
This commit is contained in:
Tom Wiesing 2020-06-26 12:54:47 +02:00
parent 9ece280e72
commit 76ef5d8e68
No known key found for this signature in database
GPG key ID: DC1F29F2BC78AB15
43 changed files with 943 additions and 545 deletions
Download patch file Download diff file Expand all files Collapse all files

2
README.md
View file

@ -2,6 +2,8 @@
This repository contains a factory server implementation that creates and maintains a list of Drupal Instances.
** This documentation is not yet updated to the new approach **
** This is a work in progress and nothing in this repository is ready for production use **
## Overview

63
Vagrantfile vendored
View file

@ -1,67 +1,24 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
# The most common configuration options are documented and commented below.
# For a complete reference, please see the online documentation at
# https://docs.vagrantup.com.
# Every Vagrant development environment requires a box. You can search for
# boxes at https://vagrantcloud.com/search.
# use an iamge of debian, in this case buster 64
config.vm.box = "debian/buster64"
# Disable automatic box update checking. If you disable this, then
# boxes will only be checked for updates when the user runs
# `vagrant box outdated`. This is not recommended.
# config.vm.box_check_update = false
# forward ports 80 and 443 to the host system
# this will allow accessing the webserver from the real system.
config.vm.network "forwarded_port", guest: 80, host: 80
config.vm.network "forwarded_port", guest: 443, host: 443
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
# NOTE: This will enable public access to the opened port
# config.vm.network "forwarded_port", guest: 80, host: 8080
# share the factory folder in /factory/
config.vm.synced_folder "distillery/", "/distillery/"
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine and only allow access
# via 127.0.0.1 to disable public access
config.vm.network "forwarded_port", guest: 80, host: 8080
# Create a private network, which allows host-only access to the machine
# using a specific IP.
# config.vm.network "private_network", ip: "192.168.33.10"
# Create a public network, which generally matched to bridged network.
# Bridged networks make the machine appear as another physical device on
# your network.
# config.vm.network "public_network"
# Share an additional folder to the guest VM. The first argument is
# the path on the host to the actual folder. The second argument is
# the path on the guest to mount the folder. And the optional third
# argument is a set of non-required options.
config.vm.synced_folder "factory/", "/factory/"
# Provider-specific configuration so you can fine-tune various
# backing providers for Vagrant. These expose provider-specific options.
# Example for VirtualBox:
#
# for performance, we setup 4GB of memort and 2 CPUs
config.vm.provider "virtualbox" do |vb|
vb.memory = 4096
vb.cpus = 2
end
#
# View the documentation for the provider you are using for more
# information on available options.
# Enable provisioning with a shell script. Additional provisioners such as
# Ansible, Chef, Docker, Puppet and Salt are also available. Please see the
# documentation for more information about their specific syntax and use.
# config.vm.provision "shell", inline: <<-SHELL
# apt-get update
# apt-get install -y apache2
# SHELL
# tell the user where things are.
config.vm.post_up_message = "Ready to distil and make WissKIs. Scripts can be found in /distillery/. "
end

21
factory/.env.example → distillery/.env.sample
View file

@ -1,24 +1,33 @@
# Sometimes when http(s) is enabled, it is advisable to listen on a port different from 80.
# The public port can be configured here.
PUBLIC_PORT=80
# All WissKI and Drupal Installations are contained within a single directory.
# All WissKi and Drupal Installations are contained within a single directory.
# The name of each subfolder corresponds to the appropriate domain name.
# This variable determines the subfolder to place installations into.
DRUPAL_ROOT=/var/www/factory
# Furthermore, several docker-compose files are created to manage global services and the system itself.
# These will be placed into a folder, set its' path here.
COMPOSER_ROOT=/var/www/deploy
# The system can support setting up certificate(s) automatically.
# It can be enabled by setting an email for certbot certificates.
# This email address can be configured here.
CERTBOT_EMAIL=
# Each Drupal instance requires a corresponding system user, database users and databases.
# These are also set by the appropriate domain name.
# To differentiate them from other users of the system, these names can be prefixed.
# The prefix to use can be configured here.
# When changing these please consider that no system user may exist that has the same name as a mysql user.
# This is a MariaDB restriction.
SYSTEM_USER_PREFIX=factory-
MYSQL_USER_PREFIX=mysql-factory-
MYSQL_DATABASE_PREFIX=mysql-factory-
GRAPHDB_USER_PREFIX=graphdb-factory-
GRAPHDB_REPO_PREFIX=graphdb-factory-
# In addition to the filesystem the WissKI distillery requires a single SQL table.
# It uses this database to store a list of installed things
DISTILLERY_BOOKKEEPING_DATABASE=distillery
DISTILLERY_BOOKKEEPING_TABLE=distillery
# Each created Drupal Instance corresponds to a single domain name.
# These domain names should either be a complete domain name or a sub-domain of a default domain.
# This setting configures the default domain-name to create subdomains of.

15
factory/lib/00_init.sh → distillery/lib/00_init.sh
View file

@ -44,3 +44,18 @@ function log_warn() {
function log_error() {
echo -e "\033[0;31m$1\033[0m"
}
if [ -n "$DISABLE_LOG" ]; then
function log_info() {
true
}
function log_ok() {
true
}
function log_warn() {
true
}
function log_error() {
true
}
fi

55
factory/lib/10_config.sh → distillery/lib/10_config.sh
View file

@ -67,6 +67,15 @@ function is_valid_number() {
fi;
}
# 'is_valid_email' checks if a value is a valid email address
function is_valid_email() {
if [[ "$1" =~ ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$ ]]; then
return 0;
else
return 1;
fi
}
# The 'DRUPAL_ROOT' variable must be an absolute path.
if ! is_valid_abspath "$DRUPAL_ROOT"; then
log_error "Variable 'DRUPAL_ROOT' is missing or not a valid path. ";
@ -75,10 +84,11 @@ if ! is_valid_abspath "$DRUPAL_ROOT"; then
exit 1;
fi
# The 'SYSTEM_USER_PREFIX' variable must be a valid slug.
if ! is_valid_slug "$SYSTEM_USER_PREFIX"; then
log_error "Variable 'SYSTEM_USER_PREFIX' is missing or not a valid slug. ";
# The 'COMPOSER_ROOT' variable must be an absolute path.
if ! is_valid_abspath "$COMPOSER_ROOT"; then
log_error "Variable 'COMPOSER_ROOT' is missing or not a valid path. ";
log_info "Please verify that it is set correctly in '.env'. ";
log_info "Please ensure that it does not end in '/'. ";
exit 1;
fi
@ -96,6 +106,21 @@ if ! is_valid_slug "$MYSQL_DATABASE_PREFIX"; then
exit 1;
fi
# The 'DISTILLERY_BOOKKEEPING_DATABASE' variable must be a valid slug.
if ! is_valid_slug "$DISTILLERY_BOOKKEEPING_DATABASE"; then
log_error "Variable 'DISTILLERY_BOOKKEEPING_DATABASE' is missing or not a valid slug. ";
log_info "Please verify that it is set correctly in '.env'. ";
exit 1;
fi
# The 'DISTILLERY_BOOKKEEPING_TABLE' variable must be a valid slug.
if ! is_valid_slug "$DISTILLERY_BOOKKEEPING_TABLE"; then
log_error "Variable 'DISTILLERY_BOOKKEEPING_TABLE' is missing or not a valid slug. ";
log_info "Please verify that it is set correctly in '.env'. ";
exit 1;
fi
# The 'GRAPHDB_USER_PREFIX' variable must be a valid slug.
if ! is_valid_slug "$GRAPHDB_USER_PREFIX"; then
log_error "Variable 'DATABASE_PREFIX' is missing or not a valid slug. ";
@ -110,14 +135,22 @@ if ! is_valid_slug "$GRAPHDB_REPO_PREFIX"; then
exit 1;
fi
# The 'DOMAIN' variable must be a valid domain.
# The 'DEFAULT_DOMAIN' variable must be a valid domain.
if ! is_valid_domain "$DEFAULT_DOMAIN"; then
log_error "Variable 'DEFAULT_DOMAIN' is missing or not a valid domain. ";
log_info "Please verify that it is set correctly in '.env'. ";
exit 1;
fi
# The 'CERTBOT_EMAIL' variable should either be empty or a valid email
if [ -n "$CERTBOT_EMAIL" ]; then
if ! is_valid_email "$CERTBOT_EMAIL"; then
log_error "Variable 'CERTBOT_EMAIL' is not a valid email address. ";
log_info "Please verify that it is set correctly in '.env' or remove it completly. ";
exit 1;
fi;
fi
# The 'PASSWORD_LENGTH' variable must be a valid number.
if ! is_valid_number "$PASSWORD_LENGTH"; then
log_error "Variable 'PASSWORD_LENGTH' is missing or not a valid number. ";
@ -125,11 +158,11 @@ if ! is_valid_number "$PASSWORD_LENGTH"; then
exit 1;
fi
# The 'PUBLIC_PORT' must be a valid number.
if ! is_valid_number "$PUBLIC_PORT"; then
log_error "Variable 'PUBLIC_PORT' is missing or not a valid number. ";
log_info "Please verify that it is set correctly in '.env'. ";
exit 1;
fi
# paths to composer things
COMPOSER_WEB_DIR="$COMPOSER_ROOT/core/web"
COMPOSER_TRIPLESTORE_DIR="$COMPOSER_ROOT/core/triplestore"
COMPOSER_SQL_DIR="$COMPOSER_ROOT/core/sql"
COMPOSER_INSTANCES_DIR="$COMPOSER_ROOT/instances"
log_ok "Read and validated configuration file. "

86
distillery/lib/20_sql.sh Normal file
View file

@ -0,0 +1,86 @@
#!/bin/bash
set -e
# This is a library file.
# It should be 'source'd only, if it is not we bail out here.
if [[ "$0" = "$BASH_SOURCE" ]]; then
echo "This file should not be executed directly, it should be 'source'd only. "
exit 1;
fi
###
### General SQL functions
###
# wait_for_sql waits for the sql database to be ready
function wait_for_sql() {
log_info "Waiting for database to start ..."
_wait_for_sql_internal
log_ok "Database responded to query "
}
function _wait_for_sql_internal() {
timeout=30
times=1
until dockerized_mysql -e 'show databases;' > /dev/null 2>&1; do
((times=times+1))
if [ "$times" -gt $timeout ]; then
log_error "Database timed out after ${timeout} seconds(s). "
fi;
echo -n "."
sleep 1
done
}
# 'dockerized_mysql' runs an sql command in the sql docker container
function dockerized_mysql() {
pushd "$COMPOSER_SQL_DIR" > /dev/null
docker exec -i `docker-compose ps -q sql` mysql "$@"
retval=$?
popd > /dev/null
return $retval
}
# 'dockerized_mysql' runs an sql command in the sql docker container interactively
function dockerized_mysql_interactive() {
pushd "$COMPOSER_SQL_DIR" > /dev/null
docker exec -ti `docker-compose ps -q sql` mysql "$@"
retval=$?
popd > /dev/null
return $retval
}
###
### Bookkeeping sql
###
# 'sql_bookkeep_exists' checks if a given site already exists
function sql_bookkeep_exists() {
COUNT=$(dockerized_mysql -D "$DISTILLERY_BOOKKEEPING_DATABASE" -e "SELECT COUNT(*) FROM \`$DISTILLERY_BOOKKEEPING_TABLE\` WHERE slug=\"$1\"; " | tail -n +2)
if [ "$COUNT" = "1" ]; then
return 0;
else
return 1;
fi;
}
# 'sql_bookkeep_insert' inserts a new pair of values into the sql database
function sql_bookkeep_insert() {
dockerized_mysql -D "$DISTILLERY_BOOKKEEPING_DATABASE" -e "INSERT INTO \`$DISTILLERY_BOOKKEEPING_TABLE\`($1) VALUES ($2) ;"
}
# 'sql_bookkeep_delete' removes a slug into the bookeeping table
function sql_bookeep_delete() {
dockerized_mysql -D "$DISTILLERY_BOOKKEEPING_DATABASE" -e "DELETE FROM \`$DISTILLERY_BOOKKEEPING_TABLE\` WHERE slug=\"$1\";"
}
# 'sql_bookkeep_load' reads from the bookkeeping table
function sql_bookkeep_load() {
dockerized_mysql -D "$DISTILLERY_BOOKKEEPING_DATABASE" -e "SELECT $2 FROM \`$DISTILLERY_BOOKKEEPING_TABLE\` WHERE slug=\"$1\";"
}
# 'sql_bookkeep_list' lists all slugs from the bookkeeping table
function sql_bookkeep_list() {
dockerized_mysql -D "$DISTILLERY_BOOKKEEPING_DATABASE" -e "SELECT slug FROM \`$DISTILLERY_BOOKKEEPING_TABLE\`; " | tail -n +2
}

27
factory/lib/20_slug.sh → distillery/lib/30_slug.sh
View file

@ -21,8 +21,7 @@ function require_slug_argument() {
log_info " => Deriving configuration for '$SLUG'. "
echo "Domain Name: $INSTANCE_DOMAIN"
echo "Base Directory: $BASE_DIR"
echo "System User: $SYSTEM_USER"
echo "Base Directory: $INSTANCE_BASE_DIR"
echo "MySQL User: $MYSQL_USER"
echo "MySQL Database: $MYSQL_DATABASE"
echo "GraphDB User: $GRAPHDB_USER"
@ -52,29 +51,17 @@ USERNAME_BASE="${USERNAME_BASE//_/--u}"
USERNAME_BASE="${USERNAME_BASE//./-}"
# Generate the user and database names for the various systems
SYSTEM_USER="${SYSTEM_USER_PREFIX}${USERNAME_BASE}"
MYSQL_USER="${MYSQL_USER_PREFIX}${USERNAME_BASE}"
MYSQL_DATABASE="${MYSQL_DATABASE_PREFIX}${USERNAME_BASE}"
GRAPHDB_USER="${GRAPHDB_USER_PREFIX}${USERNAME_BASE}"
GRAPHDB_REPO="${GRAPHDB_REPO_PREFIX}${USERNAME_BASE}"
# Compute the base directory for the files that will live on disk.
BASE_DIR="$DRUPAL_ROOT/$INSTANCE_DOMAIN"
ENV_FILE="$BASE_DIR/wisski-env"
COMPOSER_DIR="$BASE_DIR/project"
WEB_DIR="$COMPOSER_DIR/web"
ONTOLOGY_DIR="$WEB_DIR/sites/default/files/ontology"
INSTANCE_BASE_DIR="$COMPOSER_INSTANCES_DIR/$INSTANCE_DOMAIN"
INSTANCE_DATA_DIR="$INSTANCE_BASE_DIR/data/"
# Setup aliases for drush and composer.
alias composer="sudo -u $SYSTEM_USER /usr/local/bin/composer"
alias drush="sudo -u $SYSTEM_USER $COMPOSER_DIR/vendor/bin/drush"
if [ -n "$CERTBOT_EMAIL" ]; then
LETSENCRYPT_HOST="$INSTANCE_DOMAIN"
LETSENCRYPT_EMAIL="$CERTBOT_EMAIL"
fi;
# Because of a bug in Drupal we constantly have to reset the permissions of the site directory.
# See https://www.drupal.org/project/drupal/issues/3091285.
function drupal_sites_permission_workaround() {
chmod -R u+w "$WEB_DIR/sites/"
}
# Apache configuration paths
APACHE_CONFIG_SITE_AVAILABLE="/etc/apache2/sites-available/${INSTANCE_DOMAIN}.conf"
APACHE_CONFIG_SITE_ENABLED="/etc/apache2/sites-enabled/${INSTANCE_DOMAIN}.conf"

17
factory/lib/30_templates.sh → distillery/lib/40_templates.sh
View file

@ -8,7 +8,8 @@ if [[ "$0" = "$BASH_SOURCE" ]]; then
exit 1;
fi
TEMPLATE_DIR="$SCRIPT_DIR/resources/templates/"
RESOURCE_DIR="$SCRIPT_DIR/resources"
TEMPLATE_DIR="$RESOURCE_DIR/templates/"
# load_template will load a template $1 from the template directory
# and replace ${$2} with $3, ${$4} with $5 etc.
@ -30,5 +31,19 @@ function load_template() {
echo "$TEMPLATE"
}
# install_resource_dir will copy over a resource directory
# to a destination path recursively.
function install_resource_dir() {
from="$RESOURCE_DIR/$1"
to=$2
# copythe template files
for filename in "$from"/*; do
dest="$to/`basename "${filename}"`"
echo "Writing \"$dest\""
cp -r "$filename" "$dest"
done
}
# path where common apache files will be installed.
WISSKI_COMMON_PATH="/etc/apache2/conf/wisski"

0
factory/lib/40_utils.sh → distillery/lib/50_utils.sh
View file

7
factory/lib/lib.sh → distillery/lib/lib.sh
View file

@ -16,6 +16,7 @@ LIB_DIR="$SCRIPT_DIR/lib"
# These contain functionality used in the various scripts.
source "$LIB_DIR/00_init.sh";
source "$LIB_DIR/10_config.sh";
source "$LIB_DIR/20_slug.sh";
source "$LIB_DIR/30_templates.sh";
source "$LIB_DIR/40_utils.sh";
source "$LIB_DIR/20_sql.sh";
source "$LIB_DIR/30_slug.sh";
source "$LIB_DIR/40_templates.sh";
source "$LIB_DIR/50_utils.sh";

14
distillery/mysql.sh Normal file
View file

@ -0,0 +1,14 @@
#!/bin/bash
set -e
# disable the log
DISABLE_LOG=1
# read the lib/shared.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
# wait for sql to come up
wait_for_sql > /dev/null
dockerized_mysql_interactive "$@"

107
distillery/provision.sh Normal file
View file

@ -0,0 +1,107 @@
# To install a new system:
# This script will provision a new Drupal instance and make it available to apache.
# Usage: sudo ./provision.sh $SLUG
# In case the installation fails, it will bail out and leave you with an incomplete installation.
# To delete an incomplete installation, use the ./purge.sh script, or try fixing the error manually.
set -e
# read the lib/shared.sh and read the slug argument.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
# wait for sql to be awake
wait_for_sql
# check if the site exists
if sql_bookkeep_exists "$SLUG"; then
log_error "=> Site '$SLUG' already exists in bookeeping table. "
echo "Refusing to work"
exit 1;
fi
# Randomly generate the database name and user we will configure.
# Use the 'randompw' alias for this.
log_info " => Generating new MySQL password"
MYSQL_PASSWORD="$(randompw)"
# Initialize the SQL database with those credentials.
log_info " => Intializing new SQL database '${MYSQL_DATABASE}' and user '$MYSQL_USER'. "
dockerized_mysql -e "CREATE DATABASE \`${MYSQL_DATABASE}\`;"
dockerized_mysql -e "CREATE USER \`${MYSQL_USER}\`@'%' IDENTIFIED BY '${MYSQL_PASSWORD}';"
dockerized_mysql -e "GRANT ALL PRIVILEGES ON \`${MYSQL_DATABASE}\`.* TO \`${MYSQL_USER}\`@\`%\`;"
dockerized_mysql -e "FLUSH PRIVILEGES;"
# Create a new repository for GraphDB.
# Use the template for this.
log_info " => Generating new GraphDB repository '$GRAPHDB_REPO'"
load_template "repository/graphdb-repo.ttl" "GRAPHDB_REPO" "${GRAPHDB_REPO}" "INSTANCE_DOMAIN" "${INSTANCE_DOMAIN}" | \
curl -X POST \
http://127.0.0.1:7200/rest/repositories \
--header 'Content-Type: multipart/form-data' \
-F "config=@-"
# Generate a random password for the GraphDB user
log_info " => Generating a new GraphDB password"
GRAPHDB_PASSWORD="$(randompw)"
# Create the user and grant them access to the creatd database.
log_info " => Creating GraphDB user '$GRAPHDB_USER'"
load_template "repository/graphdb-user.json" "GRAPHDB_USER" "${GRAPHDB_USER}" "GRAPHDB_REPO" "${GRAPHDB_REPO}" | \
curl -X POST \
"http://127.0.0.1:7200/rest/security/user/${GRAPHDB_USER}" \
--header 'Content-Type: application/json' \
--header 'Accept: text/plain' \
--header "X-GraphDB-Password: $GRAPHDB_PASSWORD" \
-d @-
log_info " => Creating local directory '$INSTANCE_BASE_DIR'"
mkdir -p "$INSTANCE_BASE_DIR"
mkdir -p "$INSTANCE_DATA_DIR"
mkdir -p "$INSTANCE_DATA_DIR/.composer"
mkdir -p "$INSTANCE_DATA_DIR/data"
# Generate some more random credentials, this time for drupal.
# We again make use of the randompw alias.
log_info " => Generating new drupal credentials"
DRUPAL_USER="admin"
DRUPAL_PASS="$(randompw)"
# TODO: copy over docker-compose into the right directory
log_info " => Creating instance directory"
install_resource_dir "compose/runtime" "$INSTANCE_BASE_DIR"
# Log all the details into the bookeeping database
log_info "=> Storing configuration in bookkeeping table"
sql_bookkeep_insert \
"slug,filesystem_base,sql_database,sql_user,sql_password,graphdb_repository,graphdb_user,graphdb_password" \
"\"${SLUG}\",\"${INSTANCE_BASE_DIR}\",\"${MYSQL_DATABASE}\",\"${MYSQL_USER}\",\"${MYSQL_PASSWORD}\",\"${GRAPHDB_REPO}\",\"${GRAPHDB_USER}\",\"${GRAPHDB_PASSWORD}\""
log_info " => Writing configuration file"
load_template "runtime-config/environment" \
"REAL_PATH" "${INSTANCE_DATA_DIR}" \
"VIRTUAL_HOST" "${INSTANCE_DOMAIN}" \
"LETSENCRYPT_HOST" "${LETSENCRYPT_HOST}" \
"LETSENCRYPT_EMAIL" "${LETSENCRYPT_EMAIL}" \
> "$INSTANCE_BASE_DIR/.env"
log_info " => Running and building image"
cd "$INSTANCE_BASE_DIR"
docker-compose build --pull
docker-compose pull
log_info " => Running provision script"
docker-compose run --rm runtime /bin/bash -c "sudo PATH=\$PATH -u www-data /bin/bash /provision_container.sh \
\"${INSTANCE_DOMAIN}\" \
\"${MYSQL_DATABASE}\" \"${MYSQL_USER}\" \"${MYSQL_PASSWORD}\" \
\"${GRAPHDB_REPO}\" \"${GRAPHDB_USER}\" \"${GRAPHDB_PASSWORD}\" \
\"${DRUPAL_USER}\" \"${DRUPAL_PASS}\" \
\"${USE_DRUPAL_9}\""
log_info " => Starting container"
docker-compose up -d

67
distillery/purge.sh Normal file
View file

@ -0,0 +1,67 @@
# To install a new system:
# This script will provision a new Drupal instance and make it available to apache.
# Usage: sudo ./provision.sh $SLUG
# In case the installation fails, it will bail out and leave you with an incomplete installation.
# To delete an incomplete installation, use the ./purge.sh script, or try fixing the error manually.
set -e
# read the lib/shared.sh and read the slug argument.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
# wait for sql to be awake
wait_for_sql
while true; do
log_info " => I'm about to delete the '$SLUG' site from this system. "
read -p "This can not be undone. Please type 'y' to continue: " yn
case $yn in
[Yy]* ) break;;
* ) echo "Abort. "; exit 1;;
esac
done
# check if the site exists
if ! sql_bookkeep_exists "$SLUG"; then
log_error "=> Site '$SLUG' does not exist in bookeeping table. "
echo "I'll try to cleanup with the current defaults. "
echo "This may or may not work. "
else
# Read all the configuration from the database
log_info " => Reading components from database"
read -r INSTANCE_BASE_DIR MYSQL_DATABASE MYSQL_USER GRAPHDB_REPO GRAPHDB_USER <<< "$(sql_bookkeep_load "${SLUG}" "filesystem_base,sql_database,sql_user,graphdb_repository,graphdb_user" | tail -n +2)"
fi
# stop the running system container
if [ -d "$INSTANCE_BASE_DIR" ] ; then
log_info "=> Stopping running system"
cd "$INSTANCE_BASE_DIR"
docker-compose down -v || true
fi;
cd
# delete the mysql database.
log_info " => Deleting MySQL database '$MYSQL_DATABASE' and user '$MYSQL_USER'. "
dockerized_mysql -e "DROP DATABASE IF EXISTS \`${MYSQL_DATABASE}\`;"
dockerized_mysql -e "DROP USER IF EXISTS \`${MYSQL_USER}\`@\`%\`;"
dockerized_mysql -e "FLUSH PRIVILEGES;"
# Clear the GraphDB repository.
log_info " => Deleting GraphDB user '$GRAPHDB_USER'"
curl -X DELETE http://127.0.0.1:7200/rest/security/user/$GRAPHDB_USER/
log_info " => Deleting GraphDB repository '$GRAPHDB_REPO'"
curl -X DELETE http://127.0.0.1:7200/rest/repositories/$GRAPHDB_REPO/
# Delete the directory
log_info " => Deleting '$INSTANCE_BASE_DIR'"
rm -rf "$INSTANCE_BASE_DIR"
log_info " => Clearing bookkeeping record"
sql_bookeep_delete "$SLUG" || true
log_info " => '$SLUG' has been purged. "

1
distillery/resources/.dockerignore Normal file
View file

@ -0,0 +1 @@
data/

6
distillery/resources/compose/runtime/.dockerignore Normal file
View file

@ -0,0 +1,6 @@
# Ignore everything
*
# allow the following files:
!conf/*
!scripts/*

27
distillery/resources/compose/runtime/.env.sample Normal file
View file

@ -0,0 +1,27 @@
#######################
# Meta Settings
#######################
# Real path for volumes to be stored
REAL_PATH=/var/www/example.slug
#######################
### Web Server settings
#######################
# the hostname for the website
VIRTUAL_HOST=example.com
# optional letsencrypt support
# when blank, ignore
LETSENCRYPT_HOST=
LETSENCRYPT_EMAIL=
### SQL settings
MYSQL_HOST=mysql
MYSQL_USER=user
MYSQL_PASS=pass
### GraphDB settings
GRAPHDB_HOST=graphdb
GRAPHDB_USER=user
GRAPHDB_PASS=pass

75
distillery/resources/compose/runtime/Dockerfile Normal file
View file

@ -0,0 +1,75 @@
FROM php:7-apache-buster
WORKDIR /var/www
# install and enable the various required php extension
RUN apt-get update && apt-get install -y \
libcurl4-openssl-dev curl \
libpng-dev \
libicu-dev \
libxml2-dev \
libssh2-1-dev \
sudo \
zip unzip \
default-mysql-client \
&& \
docker-php-source extract && \
docker-php-ext-install \
curl \
gd \
intl \
soap \
mysqli \
opcache \
pdo_mysql \
xml \
xmlrpc \
&& \
pecl install ssh2-1.2 && \
docker-php-ext-enable \
curl \
gd \
intl \
mysqli \
opcache \
pdo_mysql \
soap \
ssh2 \
mysqli \
xml \
xmlrpc \
&& \
docker-php-source delete
# enable the apache rewrite mod
RUN a2enmod rewrite
# install composer and add it to path
RUN curl -sS https://getcomposer.org/installer | php && \
mv composer.phar /usr/local/bin/composer
ENV PATH "/usr/local/bin:/var/www/data/project/vendor/bin:$PATH"
# remove default configuration
RUN rm /etc/apache2/sites-available/*.conf && \
rm /etc/apache2/sites-enabled/*.conf
# Add wisski configuration
ADD conf/ports.conf /etc/apache2/ports.conf
ADD conf/wisski.conf /etc/apache2/sites-available/wisski.conf
RUN a2ensite wisski
# volumes for composer
VOLUME /var/www/.composer
VOLUME /var/www/data
# increase the php memory limit to 2g
RUN echo 'memory_limit=2G' > /usr/local/etc/php/conf.d/memory-limit.ini
# Add and configure the entrypoint
ADD scripts/entrypoint.sh /entrypoint.sh
ENTRYPOINT [ "/entrypoint.sh" ]
CMD ["apache2-foreground"]
# Add the provision script
ADD scripts/provision_container.sh /provision_container.sh
# expose port 8080
EXPOSE 8080

4
distillery/resources/compose/runtime/conf/ports.conf Normal file
View file

@ -0,0 +1,4 @@
# This file configures where apache should listen.
# Because we are running as a limited user, we want to listen on a high port.
# For this we use port 8080
Listen 8080

22
distillery/resources/compose/runtime/conf/wisski.conf Normal file
View file

@ -0,0 +1,22 @@
<VirtualHost *:8080>
# the document root -- /var/www/data/project/web
DocumentRoot /var/www/data/project/web
<Directory /var/www/data/project/web>
# add types for .owl and .rdf
AddType application/rdf+xml .owl
AddType application/rdf+xml .rdf
# Rewrite the 'ontology' directory
ReWriteRule ^(ontology/[^/]+/).+ $1 [R=303,L]
ReWriteRule ^(ontology/[^/]+)/$ sites/default/files/$1.owl [L]
# Allow overrides of symlinks
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
ErrorLog /dev/stderr
CustomLog /dev/stdout combined
</VirtualHost>

24
distillery/resources/compose/runtime/docker-compose.yml Normal file
View file

@ -0,0 +1,24 @@
version: "3.7"
services:
runtime:
build: .
restart: always
environment:
# port and hostname for this image to use
VIRTUAL_HOST: ${VIRTUAL_HOST}
VIRTUAL_PORT: 8080
# optional letsencrypt email
LETSENCRYPT_HOST: ${LETSENCRYPT_HOST}
LETSENCRYPT_EMAIL: ${LETSENCRYPT_EMAIL}
# the volumes to
volumes:
- ${REAL_PATH}/.composer:/var/www/.composer
- ${REAL_PATH}/data:/var/www/data
networks:
default:
external:
name: distillery

11
distillery/resources/compose/runtime/scripts/entrypoint.sh Executable file
View file

@ -0,0 +1,11 @@
#!/bin/bash
# This script contains
# chown the volumes to make sure they can be read and written by the limited user
chown www-data:www-data /var/www
chown www-data:www-data /var/www/.composer
chown www-data:www-data /var/www/data/
# run the original entrypoint
docker-php-entrypoint "$@"

126
distillery/resources/compose/runtime/scripts/provision_container.sh Normal file
View file

@ -0,0 +1,126 @@
#!/bin/bash
set -e
function log_info() {
echo -e "\033[1m$1\033[0m"
}
function log_ok() {
echo -e "\033[0;32m$1\033[0m"
}
log_info " => Reading configuration variables"
INSTANCE_DOMAIN="$1"
echo "INSTANCE_DOMAIN=$INSTANCE_DOMAIN"
shift 1
MYSQL_DATABASE="$1"
echo "MYSQL_DATABASE=$MYSQL_DATABASE"
MYSQL_USER="$2"
echo "MYSQL_USER=$MYSQL_USER"
MYSQL_PASSWORD="$3"
echo "MYSQL_PASSWORD=$MYSQL_PASSWORD"
shift 3
GRAPHDB_REPO="$1"
echo "GRAPHDB_REPO=$GRAPHDB_REPO"
GRAPHDB_USER="$2"
echo "GRAPHDB_USER=$GRAPHDB_USER"
GRAPHDB_PASSWORD="$3"
echo "GRAPHDB_PASSWORD=$GRAPHDB_PASSWORD"
shift 3
DRUPAL_USER="$1"
echo "DRUPAL_USER=$DRUPAL_USER"
DRUPAL_PASS="$2"
echo "DRUPAL_PASS=$DRUPAL_PASS"
shift 2
USE_DRUPAL_9="$1"
echo "USE_DRUPAL_9=$USE_DRUPAL_9"
shift 1
log_info " => Preparing installation environment"
BASE_DIR="/var/www/data"
COMPOSER_DIR="$BASE_DIR/project"
WEB_DIR="$COMPOSER_DIR/web"
ONTOLOGY_DIR="$WEB_DIR/sites/default/files/ontology"
log_info " => Creating '$COMPOSER_DIR'"
mkdir -p "$COMPOSER_DIR"
cd "$COMPOSER_DIR"
function drupal_sites_permission_workaround() {
chmod -R u+w "$WEB_DIR/sites/" || true
}
# Create a new composer project.
log_info " => Creating composer project"
if [ -z "${USE_DRUPAL_9}" ]; then
composer create-project 'drupal/recommended-project:^8.9.0' .
else
composer create-project 'drupal/recommended-project:^9.0.0' .
fi
# Install drush so that we can automate a lot of things
log_info " => Installing 'drush'"
composer require drush/drush
# Use 'drush' to run the site-installation.
# Here we need to use the username, password and database creds we made above.
log_info " => Running drupal installation scripts"
drush site-install standard --yes --site-name=${INSTANCE_DOMAIN} \
--account-name=$DRUPAL_USER --account-pass=$DRUPAL_PASS \
--db-url=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@sql/${MYSQL_DATABASE}
drupal_sites_permission_workaround
# create a directory for ontologies.
log_info " => Creating '$ONTOLOGY_DIR'"
mkdir -p "$ONTOLOGY_DIR"
# Install the Wisski packages.
log_info " => Installing Wisski packages"
cd "$COMPOSER_DIR"
# install the development version when requested
if [ -z "${USE_DRUPAL_9}" ]; then
composer require 'drupal/wisski'
else
composer require 'drupal/wisski:2.x-dev'
fi
drupal_sites_permission_workaround
composer require drupal/inline_entity_form
drupal_sites_permission_workaround
composer require drupal/imagemagick
drupal_sites_permission_workaround
composer require drupal/image_effects
drupal_sites_permission_workaround
composer require drupal/colorbox
log_info " => Enable Wisski modules"
drush pm-enable --yes wisski_core wisski_linkblock wisski_pathbuilder wisski_adapter_sparql11_pb wisski_salz
drupal_sites_permission_workaround
log_info " => Provisioning is now complete. "
log_ok "Your installation details are as follows:"
function printdetails() {
echo "URL: http://$INSTANCE_DOMAIN"
echo "Username: $DRUPAL_USER"
echo "Password: $DRUPAL_PASS"
log_info " => Your GraphDB details (for WissKI Salz) are: "
echo "Read URL: http://triplestore:7200/repositories/$GRAPHDB_REPO"
echo "Write URL: http://triplestore:7200/repositories/$GRAPHDB_REPO/statements"
echo "Username: $GRAPHDB_USER"
echo "Password: $GRAPHDB_PASSWORD"
echo "Writable: yes"
echo "Default Graph URI: http://$INSTANCE_DOMAIN/#"
echo "Ontology Paths: (empty)"
echo "SameAs property: http://www.w3.org/2002/07/owl#sameAs"
}
printdetails

32
distillery/resources/compose/sql/docker-compose.yml Normal file
View file

@ -0,0 +1,32 @@
version: "3.7"
services:
sql:
image: mariadb
volumes:
- "./data/:/var/lib/mysql"
environment:
# This combination of environment variables will configure a passwordless root user
# that can only connect to the container from 'localhost'.
# This means we can only connect using 'docker-compose exec sql mysql -C '...' '.
- "MYSQL_ALLOW_EMPTY_PASSWORD=yes"
- "MYSQL_ROOT_HOST=localhost"
restart: always
phpmyadmin:
image: phpmyadmin/phpmyadmin
environment:
- "PMA_HOST=sql"
- "HIDE_PHP_VERSION=true"
# phpmyadmin running on localhost:8080 so that we can easily access the system graphically.
# By default no admin account is created, so initial shell access to make one is needed.
ports:
- 127.0.0.1:8080:80
depends_on:
- sql
restart: always
networks:
default:
external:
name: distillery

0
images/triplestore/Dockerfile → distillery/resources/compose/triplestore/Dockerfile
View file

17
distillery/resources/compose/triplestore/docker-compose.yml Normal file
View file

@ -0,0 +1,17 @@
version: "3.7"
services:
triplestore:
build: .
ports:
- "127.0.0.1:7200:7200"
volumes:
- './data/data:/opt/graphdb/data'
- './data/work:/opt/graphdb/work'
- './data/logs:/opt/graphdb/logs'
restart: always
networks:
default:
external:
name: distillery

0
images/triplestore/entrypoint.sh → distillery/resources/compose/triplestore/entrypoint.sh
View file

45
distillery/resources/compose/web/docker-compose.yml Normal file
View file

@ -0,0 +1,45 @@
version: "3.7"
services:
nginx-proxy:
image: nginxproxy/nginx-proxy:alpine
ports:
- "80:80"
- "443:443"
volumes:
- "vhost:/etc/nginx/vhost.d"
- "htpasswd:/etc/nginx/htpasswd"
- "html:/usr/share/nginx/html"
- "/var/run/docker.sock:/tmp/docker.sock:ro"
- "certs:/etc/nginx/certs"
labels:
com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: true
restart: always
networks:
- default
letsencrypt-nginx-proxy-companion:
image: jrcs/letsencrypt-nginx-proxy-companion
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "htpasswd:/etc/nginx/htpasswd"
- "vhost:/etc/nginx/vhost.d"
- "html:/usr/share/nginx/html"
- "/var/run/docker.sock:/tmp/docker.sock:ro"
- "certs:/etc/nginx/certs"
restart: always
networks:
- default
depends_on:
- nginx-proxy
volumes:
vhost:
html:
certs:
htpasswd:
networks:
default:
external:
name: distillery

28
distillery/resources/templates/bookkeeping/create.sql Normal file
View file

@ -0,0 +1,28 @@
-- create the database if it doesn't exist yet
CREATE DATABASE IF NOT EXISTS `${DATABASE}`;
-- use the database we just created
USE `${DATABASE}`;
-- create the bookkeeping table
CREATE TABLE IF NOT EXISTS `${TABLE}`(
-- automatically created fields
pk INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
created DATETIME DEFAULT CURRENT_TIMESTAMP,
-- slug of the website
slug TEXT NOT NULL UNIQUE,
-- local file path
filesystem_base TEXT NOT NULL,
-- sql access credentials
sql_database TEXT NOT NULL,
sql_user TEXT NOT NULL,
sql_password TEXT NOT NULL,
-- graphdb credentials
graphdb_repository TEXT NOT NULL,
graphdb_user TEXT NOT NULL,
graphdb_password TEXT NOT NULL
);

0
factory/resources/templates/graphdb-repo.ttl → distillery/resources/templates/repository/graphdb-repo.ttl
View file

0
factory/resources/templates/graphdb-user.json → distillery/resources/templates/repository/graphdb-user.json
View file

17
distillery/resources/templates/runtime-config/environment Normal file
View file

@ -0,0 +1,17 @@
#######################
# Meta Settings
#######################
# Real path for volumes to be stored
REAL_PATH=${REAL_PATH}
#######################
### Web Server settings
#######################
# the hostname for the website
VIRTUAL_HOST=${VIRTUAL_HOST}
# optional letsencrypt support
# when blank, ignore
LETSENCRYPT_HOST=${LETSENCRYPT_HOST}
LETSENCRYPT_EMAIL=${LETSENCRYPT_EMAIL}

89
distillery/system_install.sh Normal file
View file

@ -0,0 +1,89 @@
#!/bin/bash
set -e
# read the lib/shared.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
# Read the 'GRAPHDB_ZIP' argument from the command line.
# If it's not set, throw an error.
GRAPHDB_ZIP=$1
if [ -z "$GRAPHDB_ZIP" ]; then
log_error "Usage: system_install.sh GRAPHDB_ZIP"
exit 1;
fi;
# print some general info on the screen
log_info "=> Preparing system to become a WissKI Distillery"
echo "This script will install or upgrade this system to become a WissKI distillery. "
echo "It is idempotent and can safely be run multiple times. "
sleep 5
# Install default system upgrades.
log_info "=> Installing system updates"
apt-get update
apt-get upgrade -y
# install docker dependencies.
log_info "=> Installing docker installer dependencies"
apt-get update
apt-get install -y curl
# install docker using an automated script.
log_info "=> Installing docker"
curl -fsSL https://get.docker.com -o - | /bin/sh
# install docker-compose dependencies.
log_info "=> Install docker-compose installer dependencies"
apt-get update
apt-get install -y python3-pip
# install docker-compose.
log_info "=> Installing docker-compose"
pip3 install --upgrade docker-compose
log_info "=> Creating docker-compose directories"
mkdir -p "$COMPOSER_INSTANCES_DIR"
mkdir -p "$COMPOSER_WEB_DIR"
mkdir -p "$COMPOSER_TRIPLESTORE_DIR"
mkdir -p "$COMPOSER_SQL_DIR"
log_info "=> Creating 'distillery' network"
docker network create distillery || true
log_info "=> Creating 'docker-compose' files for the 'web'. "
install_resource_dir "compose/web" "$COMPOSER_WEB_DIR"
# copy over the directory
log_info "=> Creating 'docker-compose' files for the 'triplestore'. "
install_resource_dir "compose/triplestore" "$COMPOSER_TRIPLESTORE_DIR"
# copy the graphdb.zip
echo "Writing \"$COMPOSER_TRIPLESTORE_DIR/graphdb.zip\""
cp "$GRAPHDB_ZIP" "$COMPOSER_TRIPLESTORE_DIR/graphdb.zip"
# create data (volume) location
mkdir -p "$COMPOSER_TRIPLESTORE_DIR/data/data/"
mkdir -p "$COMPOSER_TRIPLESTORE_DIR/data/work/"
mkdir -p "$COMPOSER_TRIPLESTORE_DIR/data/logs/"
# copy over the sql resource directory, then ensure the data diretory for sql exists.
log_info "=> Creating 'docker-compose' files for the 'sql'. "
install_resource_dir "compose/sql" "$COMPOSER_SQL_DIR"
mkdir -p "$COMPOSER_SQL_DIR/data/"
# Run all the updates via system_update.sh
log_info " => Running 'system_update.sh'"
bash "$SCRIPT_DIR/system_update.sh"
log_info "=> Waiting for sql to come up"
wait_for_sql
log_info "=> Creating '$DISTILLERY_BOOKKEEPING_DATABASE' database and '$DISTILLERY_BOOKKEEPING_TABLE' table"
load_template "bookkeeping/create.sql" "DATABASE" "$DISTILLERY_BOOKKEEPING_DATABASE" "TABLE" "$DISTILLERY_BOOKKEEPING_TABLE" | \
dockerized_mysql
log_info "=> System installation finished, ready to distill. "

31
distillery/system_update.sh Normal file
View file

@ -0,0 +1,31 @@
#!/bin/bash
set -e
# read the lib/shared.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
# update_stack fully updates a docker-compose stack in the given location.
function update_stack() {
cd "$1"
docker-compose pull
docker-compose build --pull
docker-compose up -d
}
log_info "=> Rebuilding and restarting 'web' stack"
update_stack "$COMPOSER_WEB_DIR"
# build and start the triplestore
log_info "=> Rebuilding and restarting 'triplestore' stack"
update_stack "$COMPOSER_TRIPLESTORE_DIR"
# build and start the triplestore
log_info "=> Rebuilding and restarting 'sql' stack"
update_stack "$COMPOSER_SQL_DIR"
# TODO: Iterate over all the instance
# and a pull_and_update
log_info "=> System up-to-date. "

212
factory/provision.sh
View file

@ -1,212 +0,0 @@
#!/bin/bash
# This script will provision a new Drupal instance and make it available to apache.
# Usage: sudo ./provision.sh $SLUG
# In case the installation fails, it will bail out and leave you with an incomplete installation.
# To delete an incomplete installation, use the ./purge.sh script, or try fixing the error manually.
set -e
# read the lib/shared.sh and read the slug argument.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
# A global flag 'USE_DRUPAL_9' can be set to enable drupal 9 support.
# We print out the value of the flag here
if [ -z "${USE_DRUPAL_9}" ]; then
log_info " => Will install stable Drupal 8 (Use 'USE_DRUPAL_9=1' for Drupal 9). "
else
log_info " => Will install experimental Drupal 9 version ('USE_DRUPAL_9' was set)"
fi
log_info " => Validing configuration"
# If the base directory already exists, we might have accidentally picked a name that already exists.
# In that case we bail out for safety reasons.
if [ -d "$BASE_DIR" ]; then
echo "'$BASE_DIR' already exists. "
echo "Aborting provisioning, please make sure you picked a unique name. "
exit 1
fi
# Check that the apache2 config is correct.
# This is a sanity test so that we don't randomly fail later because of bad config.
log_info " => Checking apache configuration"
apache2ctl configtest > /dev/null
# Create a system user and group
log_info " => Creating system user and group '$SYSTEM_USER'"
addgroup --system "$SYSTEM_USER"
adduser --home "$BASE_DIR" --system --disabled-password --disabled-login --ingroup "$SYSTEM_USER" "$SYSTEM_USER"
# Make directory for the composer project to live in
log_info " => Making composer directory '$COMPOSER_DIR'"
sudo -u "$SYSTEM_USER" mkdir -p "$COMPOSER_DIR"
cd "$COMPOSER_DIR"
# Write out a new apache configuration file into /etc/apache2/sites-available.
# We will need to substitute in some configuration directories.
log_info " => Writing new apache configuration file"
load_template "wisski-site.conf" \
"PUBLIC_PORT" "${PUBLIC_PORT}" \
"WEB_DIR" "${WEB_DIR}" \
"INSTANCE_DOMAIN" "${INSTANCE_DOMAIN}" \
"SYSTEM_USER" "${SYSTEM_USER}" \
"WISSKI_COMMON_PATH" "${WISSKI_COMMON_PATH}" \
> "${APACHE_CONFIG_SITE_AVAILABLE}"
# Create a new composer project.
log_info " => Creating composer project"
if [ -z "${USE_DRUPAL_9}" ]; then
composer create-project 'drupal/recommended-project:^8.9.0' .
else
composer create-project 'drupal/recommended-project:^9.0.0' .
fi
composer require drush/drush
# Randomly generate the database name and user we will configure.
# Use the 'randompw' alias for this.
log_info " => Generating new MySQL password"
MYSQL_PASSWORD="$(randompw)"
# Initialize the SQL database with those credentials.
log_info " => Intializing new SQL database '${MYSQL_DATABASE}' and user '$MYSQL_USER'. "
mysql -e "CREATE DATABASE \`${MYSQL_DATABASE}\`;"
mysql -e "CREATE USER \`${MYSQL_USER}\`@localhost IDENTIFIED BY '${MYSQL_PASSWORD}';"
mysql -e "GRANT ALL PRIVILEGES ON \`${MYSQL_DATABASE}\`.* TO \`${MYSQL_USER}\`@localhost;"
mysql -e "FLUSH PRIVILEGES;"
# Generate some more random credentials, this time for drupal.
# We again make use of the randompw alias.
log_info " => Generating new drupal credentials"
DRUPAL_USER="admin"
DRUPAL_PASS="$(randompw)"
# Use 'drush' to run the site-installation.
# Here we need to use the username, password and database creds we made above.
log_info " => Running drupal installation scripts"
drush site-install standard --yes --site-name=${INSTANCE_DOMAIN} \
--account-name=$DRUPAL_USER --account-pass=$DRUPAL_PASS \
--db-url=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@localhost/${MYSQL_DATABASE}
drupal_sites_permission_workaround
# Create a new repository for GraphDB.
# Use the template for this.
log_info " => Generating new GraphDB repository '$GRAPHDB_REPO'"
load_template "graphdb-repo.ttl" "GRAPHDB_REPO" "${GRAPHDB_REPO}" "INSTANCE_DOMAIN" "${INSTANCE_DOMAIN}" | \
curl -X POST \
http://127.0.0.1:7200/rest/repositories \
--header 'Content-Type: multipart/form-data' \
-F "config=@-"
# Generate a random password for the GraphDB user
log_info " => Generating a new GraphDB password"
GRAPHDB_PASSWORD="$(randompw)"
# Create the user and grant them access to the creatd database.
log_info " => Creating GraphDB user '$GRAPHDB_USER'"
load_template "graphdb-user.json" "GRAPHDB_USER" "${GRAPHDB_USER}" "GRAPHDB_REPO" "${GRAPHDB_REPO}" | \
curl -X POST \
"http://127.0.0.1:7200/rest/security/user/${GRAPHDB_USER}" \
--header 'Content-Type: application/json' \
--header 'Accept: text/plain' \
--header "X-GraphDB-Password: $GRAPHDB_PASSWORD" \
-d @-
# create a directory for ontologies.
log_info " => Creating '$ONTOLOGY_DIR'"
mkdir -p "$ONTOLOGY_DIR"
# Install the Wisski packages.
log_info " => Installing Wisski packages"
cd "$COMPOSER_DIR"
drupal_sites_permission_workaround
# install the development version when requested
if [ -z "${USE_DRUPAL_9}" ]; then
composer require 'drupal/wisski'
else
composer require 'drupal/wisski:2.x-dev'
fi
drupal_sites_permission_workaround
composer require drupal/inline_entity_form
drupal_sites_permission_workaround
composer require drupal/imagemagick
drupal_sites_permission_workaround
composer require drupal/image_effects
drupal_sites_permission_workaround
composer require drupal/colorbox
log_info " => Installation is now technically complete. "
log_ok "Some things below may fail. If that is the case, run: "
log_ok "$ a2ensite \"${INSTANCE_DOMAIN}\""
log_ok "$ systemctl reload apache2"
log_ok "$ $SCRIPT_DIR/shell.sh $SLUG"
log_ok "Your installation details are as follows:"
function printdetails() {
echo "URL: http://$INSTANCE_DOMAIN"
echo "Username: $DRUPAL_USER"
echo "Password: $DRUPAL_PASS"
log_info " => Your GraphDB details (for WissKI Salz) are: "
echo "Read URL: http://127.0.0.1:7200/repositories/$GRAPHDB_REPO"
echo "Write URL: http://127.0.0.1:7200/repositories/$GRAPHDB_REPO/statements"
echo "Writable: yes"
echo "Default Graph URI: http://$INSTANCE_DOMAIN/#"
echo "Ontology Paths: (empty)"
echo "SameAs property: http://www.w3.org/2002/07/owl#sameAs"
}
printdetails
function alldetails() {
echo "# Automatically generated WissKi details"
echo "# generated $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
echo "SLUG=$SLUG"
echo "INSTANCE_DOMAIN=$INSTANCE_DOMAIN"
echo "# System"
echo "SYSTEM_USER=$SYSTEM_USER"
echo "# Drupal"
echo "DRUPAL_USER=$DRUPAL_USER"
echo "DRUPAL_PASSWORD=$DRUPAL_PASSWORD"
echo "# MySQL"
echo "MYSQL_USER=$MYSQL_USER"
echo "MYSQL_PASSWORD=$MYSQL_PASSWORD"
echo "MYSQL_DATABASE=$MYSQL_DATABASE"
echo "# GraphDB"
echo "GRAPHDB_USER=$GRAPHDB_USER"
echo "GRAPHDB_PASSWORD=$GRAPHDB_PASSWORD"
echo "GRAPHDB_REPO=$GRAPHDB_REPO"
}
# put installation details in ENV_FILE
log_info " => Storing installation details in $ENV_FILE"
alldetails > "$ENV_FILE"
chown "$SYSTEM_USER:$SYSTEM_USER" "$ENV_FILE"
chmod o-r "$ENV_FILE"
# Enable the WissKI modules.
log_info " => Enable Wisski modules"
drush pm-enable --yes wisski_core wisski_linkblock wisski_pathbuilder wisski_adapter_sparql11_pb wisski_salz
drupal_sites_permission_workaround
# Because of a regresssion in EasyRDF and Tomcat, we need to manually patch EasyRDF
EASYRDF_RESPONSE="$COMPOSER_DIR/vendor/easyrdf/easyrdf/lib/EasyRdf/Http/Response.php"
log_info " => Patching '$EASYRDF_RESPONSE'"
load_template "easyrdf.patch" | patch "$EASYRDF_RESPONSE"
# Finally enable the apache2 config.
# And then reload to start serving it.
log_info " => Enabling and reloading apache configuration"
a2ensite "${INSTANCE_DOMAIN}"
systemctl reload apache2
# and done!
log_info " => Finished, your Drupal details are: "
printdetails

44
factory/purge.sh
View file

@ -1,44 +0,0 @@
#!/bin/bash
set -e
# TODO: Delete system user
# read the lib/shared.sh and read the slug argument.
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
# Delete the apache configuration files first.
# This prevents drupal from being served.
log_info " => Removing apache configuration files"
rm "$APACHE_CONFIG_SITE_ENABLED" || true
rm "$APACHE_CONFIG_SITE_AVAILABLE" || true
# Reload apache to apply the configuration.
log_info " => Reloading apache"
systemctl reload apache2
# Delete the MySQL database next.
log_info " => Deleting MySQL database '$MYSQL_DATABASE' and user '$MYSQL_USER'. "
mysql -e "DROP DATABASE IF EXISTS \`${MYSQL_DATABASE}\`;" || true
mysql -e "DROP USER IF EXISTS \`${MYSQL_USER}\`@localhost;" || true
mysql -e "FLUSH PRIVILEGES;"
# Clear the GraphDB repository.
log_info " => Deleting GraphDB repository '$GRAPHDB_REPO'"
curl -X DELETE http://127.0.0.1:7200/rest/repositories/$GRAPHDB_REPO/
log_info " => Deleting GraphDB user '$GRAPHDB_USER'"
curl -X DELETE http://127.0.0.1:7200/rest/security/user/$GRAPHDB_USER/
log_info " => Deleting system user and group '$SYSTEM_USER'"
deluser "$SYSTEM_USER" || true
delgroup "$SYSTEM_USER" || true
# Finally remove any trace of the repository by removing the base directory.
log_info " => Removing directory '$BASE_DIR'"
rm -rf "$BASE_DIR"
log_info " => Finished, '$INSTANCE_DOMAIN' has been purged. "

4
factory/resources/templates/easyrdf.patch
View file

@ -1,4 +0,0 @@
281c281
< if (preg_match("|^HTTP/([\d\.x]+) (\d+) ([^\r\n]+)|", $status, $m)) {
---
> if(preg_match("|^HTTP/([\d\.x]+) (\d+) ([^\r\n]*)|", $status, $m)) {

17
factory/resources/templates/graphdb.service
View file

@ -1,17 +0,0 @@
# This file contains a systemd service for GraphDB.
# It was tailed to a WissKI service and may or may not work in general.
[Unit]
Description=GraphDB
[Service]
Type=simple
# use the special graphdb user and group, so that other users can not just access this system.
User=graphdb
Group=graphdb
# The options here make it listen only on 127.0.0.1; this prevents external users from attempting access.
ExecStart=/opt/graphdb/bin/graphdb –Xmx6g -Dgraphdb.connector.address=127.0.0.1
[Install]
WantedBy=multi-user.target

16
factory/resources/templates/wisski-site.conf
View file

@ -1,16 +0,0 @@
# This file was generated automatically.
# It contains configuration for the WissKI site:
# ${INSTANCE_DOMAIN}
# Do not edit unless you know exactly what you are doing.
<VirtualHost *:${PUBLIC_PORT}>
DocumentRoot ${WEB_DIR}
ServerName ${INSTANCE_DOMAIN}
AssignUserId ${SYSTEM_USER} ${SYSTEM_USER}
<Directory ${WEB_DIR}>
include "${WISSKI_COMMON_PATH}/wisski-common-dir.conf"
</Directory>
include "${WISSKI_COMMON_PATH}/wisski-common.conf"
</VirtualHost>

14
factory/resources/wisski-apache-common/wisski-common-dir.conf
View file

@ -1,14 +0,0 @@
# This file is included inside the 'directory' configuration of all WissKI sites.
# add types for .owl and .rdf
AddType application/rdf+xml .owl
AddType application/rdf+xml .rdf
# Rewrites the ontology directory
ReWriteRule ^(ontology/[^/]+/).+ $1 [R=303,L]
ReWriteRule ^(ontology/[^/]+)/$ sites/default/files/$1.owl [L]
# Allow overrides of symlinks
Options Indexes FollowSymLinks
AllowOverride All
Require all granted

4
factory/resources/wisski-apache-common/wisski-common.conf
View file

@ -1,4 +0,0 @@
# This file is included inside the 'virtualhost' of a WissKI site.
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

17
factory/shell.sh
View file

@ -1,17 +0,0 @@
#!/bin/bash
set -e
# read the lib/shared.sh and lib/slug.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
log_info " => Opening shell in '$COMPOSER_DIR'"
# cd into the right directory.
cd "$COMPOSER_DIR"
# add /usr/local/bin (for composer) and the vendor bin (for drush) to path
# and open a bash shell as www-data there.
sudo -u "$SYSTEM_USER" PATH="$COMPOSER_DIR/vendor/bin:/usr/local/bin:$PATH" /bin/bash

108
factory/system_install.sh
View file

@ -1,108 +0,0 @@
#!/bin/bash
set -e
# read the lib/shared.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
# This script will prepare a server to become a factory for Drupal Instances.
# Even though it assumes a clean server, it *should* be idempotent.
log_info "=> Preparing system to serve as a factory. "
# Read the 'GRAPHDB_ZIP' argument from the command line.
# If it's not set, throw an error.
GRAPHDB_ZIP=$1
if [ -z "$GRAPHDB_ZIP" ]; then
log_error "Usage: system_install.sh GRAPHDB_ZIP"
exit 1;
fi;
# Make a temporary directory to use for various tasks during this script.
log_info " => Making temporary directory"
tmpdir="$(mktemp -d)"
log_ok "Made $tmpdir"
# fetch new package versions, then upgrade everything we already have.
# This isn't technically neccessary, but it means it'll work on an otherwise untouched system.
log_info " => Installing package updates ..."
apt-get update
apt-get dist-upgrade -y
# Install composer, by downloading it using curl and then run it with php to install it in /usr/local/bin.
log_info " => Installing composer"
apt-get install -y curl php-cli php-mbstring git unzip
curl -sS https://getcomposer.org/installer -o "$tmpdir/composer-setup.php"
php $tmpdir/composer-setup.php --install-dir=/usr/local/bin --filename=composer
# Install required php extensions for Drupal and WissKI.
log_info " => Installing required php extensions"
apt-get install -y php-xml php-gd php-mysql php-common php-xmlrpc php-soap php-gd php-intl php-mysql php-zip php-curl php-ssh2
# Install the mariadb kernel.
log_info " => Installing mariadb"
apt-get -y install mariadb-server
# Install apache and required php extensions.
log_info " => Installing apache2, php and auth modules"
apt-get install -y apache2 libapache2-mod-php libapache2-mpm-itk
# Install apache and required php extensions.
log_info " => Enabling apache 'rewrite' module"
a2enmod rewrite
# Copy over the wisski templates
log_info " => Copying over Apache Templates"
mkdir -p "$WISSKI_COMMON_PATH"
cp -v $SCRIPT_DIR/resources/wisski-apache-common/* "$WISSKI_COMMON_PATH"
# Make the directory for all drupal instances to live in.
log_info " => Making root directory for Drupal Installations"
mkdir -p "$DRUPAL_ROOT"
# Install java for GraphDB.
# We use the 'headless' package to prevent installing anything graphical on a headless server.
log_info " => Installing java"
apt-get install -y default-jre-headless
# Next we have to check if we need to install graphdb.
# If '/opt/graphdb' exists, assume that the installation has already been performed.
if [ -d "/opt/graphdb" ]; then
log_info " => 'opt/graphdb' exists, skipping setup step. ";
else
# Unzip the GraphDB sources into a temporary directory.
echo " => Unzipping GraphDB into temporary directory"
unzip "$GRAPHDB_ZIP" -d "$tmpdir/graphdb"
# Then move them into /opt/graphdb.
# Here we need to make sure that the first subdirectoy is renamed appropriately during the move.
echo " => Moving GraphDB into /opt/graphdb"
mv "$tmpdir/graphdb"/* /opt/graphdb
fi
# Next make a system group 'graphdb' and system user 'graphdb'.
# And also chown the /opt/graphdb directory to that user.
# As the user might already exist, we surpress errors of the commands.
log_info " => Making GraphDB group and user"
addgroup --system graphdb || true
adduser --home "/opt/graphdb" --system --no-create-home --disabled-password --disabled-login --ingroup graphdb graphdb || true
chown -R graphdb:graphdb /opt/graphdb
# Create a service file to use graphdb with systemd.
# This file uses the users created above, and also hard-codes listening address and maximum memory.
# This avoids having to write the config file using bash hacks.
log_info " => Making 'graphdb.service'"
load_template 'graphdb.service' > /etc/systemd/system/graphdb.service
# We just created a service, so now start it and put it into autostart mode.
log_info " => Starting and enabling graphdb.service"
systemctl enable graphdb
systemctl start graphdb
# Finally remove the temporary directory we created above.
log_info " => Removing temporary directory"
rm -rf "$tmpdir"
log_info " => Server is now ready to become a factory. "

14
factory/update.sh
View file

@ -1,14 +0,0 @@
#!/bin/bash
set -e
# read the lib/shared.sh and lib/slug.sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd "$DIR"
source "$DIR/lib/lib.sh"
require_slug_argument
# TODO: Figure out if this is enough.
echo " => Running 'composer update'"
cd "$COMPOSER_DIR"
drupal_sites_permission_workaround
composer update