by Robert Pastel, Mark Woodford and Travis Forester for Grail 2. Revised for Grails 3 and 5 by Robert Pastel. Automated deployment by Robert Pastel.

Deploying a web app is the task of packaging the files in your development environment and uploading the package to the server, but there are details of the application that you will need to configure so that it runs properly on the server. Preparing for deployment and deploying is a typically Development Operations (DevOps) task, so you will need to

• Understand the technology and terminology because DevOps is a specialized domain in CS and has its own language.
• Configure the application for deployment because the environment is different in production then development.
• Manually deploy the application to understand the process and debug deployment.
• Automate deployment so that all development team members can easily deploy.

This is the typical process for DevOps specialists and system administrators. 

Terminology

Environments

Software developing applications for users requires understanding the difference between “development” and “production  environments”. 

• “Development environment” is the machine, operating system, and process for running the application while coding.
• Production environment is the machine, operating system, and process for the user to run the application.

Grails can be configured to run differently in different “environments.” See

http://grails.github.io/grails-doc/latest/guide/conf.html#environments

Grails has three different environments:

• development
• test
• production

For a web application, the development environment is the environment while you are programming on your computer at home. The production environment is the server that is accessed by the public. The test environment is a server that is setup to test integration of many services and web apps. The primary difference between the environments is the data sources. In the development environment, you use an in-memory H2 database and in the production environment you might use a MySQL server.

Although, a development process should proceed from the development environment to the test environment and then finally made public in the production environment, we will shorten the process and use only the development and production environment. This implies that you will do final testing in the production environment.

WAR File

Applications are typically packaged for production to ensure that all the components of applications are together and so that the user can easily run the application. Java EE web applications are packaged into a  “WAR” file. WAR stands for “Web application ARchive.” It is basically a JAR file that has a specific directory structure for web applications.

http://en.wikipedia.org/wiki/WAR_(file_format)

For Java EE web apps, deployment proceeds by making a WAR file of your application and uploading the WAR file to a special directory on the server. The server un-packages the WAR, called exploding, knows how to interpret the files in your applications, and consequently can serve the files to the public. There are several JSP servers, see the list at Wikipedia

http://en.wikipedia.org/wiki/JavaServer_Pages

We will the Apache Tomcat server

http://en.wikipedia.org/wiki/Apache_Tomcat

Tomcat is just another Java application. There is no special hardware associated with the server, but for the server to be public it needs to be connected to the Internet and registered in a Domain Name Server (DNS).

Tomcat Directories

Domain

The domain name for the machine with the tomcat instances is

ui-dev.cs.mtu.edu

Catalina Base Directories

Catalina is the core component of Tomcat. Catalina is the Tomcat’s servlet container. Catalina is an implementation of the Java Servlet 

https://www.mulesoft.com/tcat/tomcat-catalina

http://stackoverflow.com/questions/32985051/what-are-the-tomcat-component-what-is-catalina-and-coyote

The Catalina base directories for the teams are located at

• /var/lib/tomcats/2023_ui_1/
• /var/lib/tomcats/2023_ui_2/
• /var/lib/tomcats/2023_ui_3/
• …

The form of the path for the Catalina base directories is

/var/lib/tomcat/<tomcat instance name>

Where the “tomcat instance name” is the location and name for the specific tomcat that will serve your web app. Note that tomcats/ has a “s”. Each team has access only to their tomcat instance’s Catalina base directories. The Catalina base directory contains subdirectories: 

• bin/ – scripts for managing the the tomcat
• conf/ –  configuration files for your tomcat instance
• lib/ – jars for running the tomcat
• logs/ – log files written by your tomcat instance
• temp
• webapp/ – this is the “appBase” directory and contains war files and exploded directories for  apps running on the tomcat instance..
•  work

We will talk more about these directories later, in particular the webapp/ and logs/ directories.

Configuration

Server Context Path

If you wish, you can set server context path for your app should be set to your application name, but it is not necessary. Edit grails-app/conf/application.yml by adding to the bottom of the file:

server:
    servlet:
        context-path: /<app name>

Where <app name> is the name of your app.

Request and File Sizes

If the app uploads files or posts large data sets, the controllers need to be configured for maximum file size and request size. At the bottom of grails-app/conf/application.yml, assure the sizes are configured:

---
grails:
    controllers:
        upload:
            maxFileSize: <max file size>
            maxRequestSize: <max request size>

Where <max file size> and <max request size> are sizes in bytes. These must be integers and not an expression. Also be sure to include the three dashes, “- - -“, above the configuration.  

Database

The GORM and Hibernate packages can use many database options, for example H2 (default), HSQLDB, MySQL, etc. In addition, the database server can be either hosted by a server or embedded into the Grails application:

• hosted on a server: a service such as mysqld, running separately from the web application and possibly on a different machine, handles requests from the application to create, read, update, or delete data within the database
• embedded: the database is located within the application itself, and all operations to the database are done directly by the application

For rapid prototyping and database accesses are infrequent  (i.e. there are only a few connections at once) an embedded database will work just fine. For small projects, embedded databases have the advantage of being easy to initialize and maintain.

Another choice is where the database should store the data, in files on the hard drive or in memory. If the database is expected to grow very large, then it would be better to store the database in files on the hard drive to prevent out-of-memory errors. Keeping the database in memory allows faster access times, but if something goes wrong, all the data will be lost. It makes sense that during development, you would use a database storing data in memory, so that you can quickly make and use the database. But during production, the database should store the data so that the data will persist across deployment.

Grails uses “environments” to configure different databases for development, test, and production in grails-app/conf/application.yml. Near the bottom of application.yml, locate the environment-specific settings for the datastore. You should see code structure like:

  development:
        dataSource:
            dbCreate: create-drop
            url: jdbc:h2:mem:devDb;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    test:
        dataSource:
            dbCreate: update
            url: jdbc:h2:mem:testDb;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
    production:
        dataSource:
            dbCreate: none
            url: jdbc:h2:./prodDb;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE
            properties:

You need to make changes to the production settings.  In the production settings, change the “dbCreate” setting to “create”.

dbCreate: create

The “create” setting will make a new database each time the application is deployed. If later you wish to save the database and the data across deployments then change “create” to “none”. 

See Grails Gorm documentation

http://docs.grails.org/latest/guide/conf.html#dataSource

In the production settings, also change the url, to specify the path to the database.

url: jdbc:h2:file:/var/lib/tomcats/<tomcat instance name>/db/<app name>;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE

For example, if your tomcat instance name is 2020_hci_3 then the settings should be

url:jdbc:h2:file:/var/lib/tomcats/2020_hci_3/db/traveler;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE

Finally, comment out all the “properties” options. Don’t delete them because you might need them later.

These configurations will set the Grails application to use and connect to an H2 database storing the data in a file name <app name>.mv.db in  /var/lib/tomacts/<tomcat instance name>/db/ directory on the server. The database file will be created when you deploy your application if it does not already exist.  For more details about how to configure the data source see the grails documentation

http://grails.github.io/grails-doc/latest/guide/conf.html#dataSource

Note that you may need to make sure that tomcat has access to the database. So you may need to make the path to the database and assign all directories to the “tomcat” group. In this case, in the Catalina base directory, enter

mkdir db
chmod 755 db
chgrp tomcat db

First try deploying without making the db directory. If there is a problem then make the directory.

Apps without Databases 

Some apps do not use a database, but the grails-app/conf/application.yml must still be edited. Just comment out the production dataSource. Another technique is to configure the production dataSource to use an embedded in-memory database. Grails can then create the database, and never use it. In application.yml, copy the development dataSource setting to the production dataSource settings.

H2-console 

The h2-console is not enabled by default in the production environment. In grails-app/conf/application.yml,  add at the bottom:

---
environments:
    production:
        spring:
            h2:
                console:
                    enabled: true
                    settings.web-allow-others: true

These settings  enable the h2-console and allow remote connections. Now on the production server, you can access the h2-console  by pointing the browser to: 

https://ui-dev.cs.mtu.edu:<port number>/<app-name>/h2-console

You should make sure that access to the h2-console is secured by granting access only to the admin role. In application.groovy, add the static rule:

grails.plugin.springsecurity.controllerAnnotations.staticRules = [
...
   [pattern: '/h2-console/**',   access: ['ROLE_ADMIN']],
...
]]

When using h2-console in production, make sure that the h2-console login fields are correct.

• Driver Class: org.h2.Driver
• JDBC URL: jdbc:h2:file:/var/lib/tomcats/<tomcat instance name>/db/<your app name> – This is the same URL that you used to specified the database file location.
• User Name: what you specified for the database username. Default is “sa”
• Password: default is blank

War filename

By default Grails names the WAR file with a version number. When deploying to an external tomcat, we do not want the version number. The WAR file name must match the project/app name, otherwise the web app will not deploy properly. You can edit the WAR file name if you want without hurting the contents, but you’ll have to edit the WAR filename everytime, you deploy, An easy technique to remove the version number from the WAR filename is to make the version number an empty string

Near the top of the build.gradle file, change the line

version "0.1"

to 

version ""

Mail Service

(Robert Pastel: needs checking with Grails 5.)

If your web application uses a mail service then you will need to configure the mail for the production mail server. If your application requires users to register in order to use the app then your web app should be using the mail server to send a key to the user’s email address so that they can finish the registration. This is done for two reasons. The email address is a unique identifier for the user. Also, responding to the email address assures that the user is a “real” person and not a robot because a real user can click on the url specified in email. Finally the mail service is used to send a key to the user when they forget their passwords and need to reset their passwords. If you are using Spring Security UI plugin in your application then the above registration and forgotten password procedures are implement in your web app, and your web app is using a mail server and the Grails Mail plugin.

By default the Mail plugin is configured at localhost on port 25. The Mail plugin can be reconfigured in Config.groovy file. On the production server your application will use a special mtu email account. The email address is “mushroommail@mtu.edu”  with a password I’ll give you later. In the bottom of the application.groovy file add the following code:

// Spring Security Email Configurations
grails {
    	mail {
        		host = "smtp.gmail.com"
        		port = 465
        		username = "mushroommail@mtu.edu"
        		password = "actual-password"
        		props = ["mail.smtp.auth":"true",
            			"mail.smtp.socketFactory.port":"465",
            			"mail.smtp.socketFactory.class":"javax.net.ssl.SSLSocketFactory",
            			"mail.smtp.socketFactory.fallback":"false"]
     }
}

Note that you will have to replace “actual-password” with a password that I will give you via google docs.

You can read more about configuring and using the Mail plugin at

http://grails.org/plugins/mail

File paths 

Some applications use the server file system to store files. For example uploaded images could be saved as a file on the server and the database stores only the path to the image file. Another example is the app could use read JSON file on the server’s file system to configure an aspect of the app. In this case, file paths will be different for production and development, and we need configure the file path for the different environments. Below is a technique to make it convenient to have different file paths.

First consider where uploaded files should be kept on the production server. Although it is possible to locate the files in the exploded web app folder that tomcat makes, this would not be appropriate because redeploying the app would cause the app to lose the files. Although the uploaded files can be stored any location on the server that the tomcat has permission to access, I have found it convenient to locate the uploaded in the webapps directory in a subdirectory named:

/<catatina path>/webapps/<app name>_content

This clearly indicates that the subdirectory is for the web app’s content files and the intended app. Note that this location is also accessible to ftp, so that programmers can add content after deployment.

Now we should consider where to locate the files on the development machine. If you are using a Linux machine for development the directory can be anywhere as long as tomcat has read and write permission in the directory. But a windows development machine will not let the web app write into any directory outside the project folder. Consequently the best location on the develop machine is to use a relative path to directory, for example at the project root (adjancent to the grails-app directory) create the directories

web-apps/<app name>_content/

The example below is a slightly modified solution I used for uploading JSON configuration files for my “vmr” web app. My development machine is a window machine. During development, JSON files are uploaded to the “vmr_content” subdirectory in the web-app/ directory. While on the production machine the JSON files are uploaded to the “vmr_content” subdirectory in the webapps folder.

This code in the init/BootStrap.groovy files makes it easy to find and change the paths if necessary, without delving into the code throughout the app:

package vmr

import grails.util.Environment

class BootStrap {
    private static final contentName = 'vmr_content'
    // System's separator 
    String sep = System.getProperty('file.separator')
    // Catalina/webapp/ Full Path
    String webappPath = "${sep}var${sep}lib${sep}tomcats${sep}2017_hci_1${sep}webapp"

   def init = { servletContext ->

     // File location based on environment
   
        switch (Environment.current) {
           case Environment.DEVELOPMENT:
                servletContext.files = [
                    server: "web-app${sep}${contentName}",
                    client: "${sep}${contentName}"
                ]
                break
             case Environment.PRODUCTION:
                 servletContext.files = [
                     server: "${webappPath}${sep}${contentName}",
                     client: "${sep}${contentName}"
                 ]
                 break
             default:
                 println 'Files switch default. Should not get here.'
       }

    ....
}

Note the code uses Grails utility Environment property to switch between the different variables. Also because the deployment and production machine are different systems, the System’s getProrperty(‘file.separator’) is used to determine whether to use ‘/’ for Linux machines or ‘\’ for Windows machines.

The servletContext variable is injected into the controllers. So while uploading the file, the controller would use servletContext.files.server to specify the path to save the file. For example to upload a json file and write it to the vmr_content directory, the controller code could be:

class SomeController {
   String sep = System.getProperty('file.separator')

   def uploadFile(){
      def json = request.JSON
      Sting jsonName = "SpecificFileName.json"
      String filePath = "${serveletContext.files.server}${sep}${jsonName}
      File file = new File(filePath)
      file.delete() // just in case already exist
      file.createNewFile()
      file.text = json.toString()
   }
   ...
}

If a jpg file is to be displayed in a view, the controller should add serveletContext.files.client to the model for the view.

class ShowFileController {
   String sep = System.getProperty('file.separator')

   def show(){
      fileName = "SpecificFileName.jpg"
      model.filePath = "serveletContext.files.client${sep}${fileName}
      return model 
   }
   ...   
}

To display the image, the gsp file would then use

<img src="${filePath}" >

The tomcat should have +rwx permissions on the vmr_content directory in question.

SSH Keys

During deployment, it is convenient not to login into the server. To avoid logging in every time, we can set up SSH keys. Note that this section is optional, but having SSH keys setup is convenient.

The SSH protocol uses asymmetric keys to encrypt and authenticate the user (client) access and communication to a server (host). The client keys are called the “private key” and the “public key”. The authentication is partially two way, and the host has a key pair. The host’s public key is called the “host key”.

The essentials of asymmetric key encryption (called public key encryption) is that a “secret message” can be encrypted with the public key. Only the owner of the private key can decrypt the “secret message”. This is all it does. It does not provide information about who sent the “secret message”, i.e. it does not “sign” the message, so alternative techniques must be used to confirm the sender of the “secret message”.

https://en.wikipedia.org/wiki/Public-key_cryptography

The ssh connection and authentication is a complex communication between the client and server. A simplification of the ssh protocol is:

1. Client initiates contact with the server
2. Server sends its host key to the client
3. The client verifies the server from the host key
4. Client and server negotiate a secure channel and pass a season ID over the channel.
5. The client initiates authentication by telling the server which public key to use.
6. The server checks if it has the public key and uses it to encrypt a “secret message”
7. The server sends the encrypted message to the client over the secured channel.
8. The client uses its private key to decipher the “secret message”. 
9. The client combines the “secret message” with the session ID and hashes them.
10. The client sends the hashed message to the server over the secured channel. 
11. The server hashes the “secret message” and session ID and compares it with the hashed message from the client. If they matched, the server authenticates the client.

References

• https://www.digitalocean.com/community/tutorials/ssh-essentials-working-with-ssh-servers-clients-and-keys
• https://www.ssh.com/academy/ssh#the-ssh-protocol
• https://web.mit.edu/rhel-doc/4/RH-DOCS/rhel-rg-en-4/s1-ssh-conn.html 

Several files are involved in the ssh authentication. 

On the client machine, in  /home/<user account>/.ssh/, there are: 

• The private key file 
• known_hosts contains public keys from servers (host keys)
• configuration files 

On the server machine, in /home/<user account>/.ssh/, there is:

• authorized_keys contains the client public keys 

In addition, the system administrator has typically made in /etc/ssh/:

• configuration files
• host private and public keys. 

To set up ssh authentication, the user is responsible for making the key pair and distributing the keys. The process is

1. Generate client key pair using ssh-keygen.
2. Copy the public key into the servers authorized_keys file using ssh-copy-id
3. Get the host keys using ssh-keyscan and copy them into known_hosts
4. If necessary configure ssh

The following is a detailed description for setting up the ssh authentication. 

Generate Key Pair

The key pair can be generated on any machine in any directory. But we don’t want the private key to be part of the git repository. So I make a directory adjacent to the workspace directory for the project, called ssh-key-pair. In the ssh-key-pair/ directory, open a bash terminal, and run

ssh-keygen -t ed25519

This will run a script in the terminal that will ask two questions. The first question is “Enter file in which to save the key:”. Enter

./ui-dev

This will name the key files. The second question is “Enter the passphrase (empty for no passphrase):”. The passphrase is a password to use the key. We will not want a passphrase, so just hit the “enter” key. You are asked to confirm the passphrase. Hit the “enter” key again. 

This will generate two files in the ssh-key-pair/ directory, “ui-dev” and “ui-dev.pub”. The “ui-dev” file is the private key and “ui-dev.pub” is the public key. 

Copy Public Key to Authorized_Keys

In the bash terminal, enter

ssh-copy-id -i ui-dev <user name>@ui-dev.cs.mtu.edu

where <user name> is the account on the ui-dev.cs.mtu.edu for your project. You are prompted for the account’s password. Enter it. 

Git Bash SSH Config

You will need to configure ssh on the client to know the location of the private key. If you are using a “git bash” terminal, in any git bash terminal enter 

cd ~/.ssh
ls config

If no config file, then create by entering

touch config

Open the ~/.ssh/config in any text editor. The “nano” editor is available in git bash, and is fairly easy to use. To use nano, enter

nano config

In the editor, add

Host ui-dev.cs.mtu.edu
  Hostname ui-dev.cs.mtu.edu
  IdentityFile <path to>/ssh-key-pair/ui-dev

Where <path to> is the path to the ssh-key-pair/ directory. Save the file. In nano, use ctrl-x and save the buffer. 

https://gist.github.com/jherax/979d052ad5759845028e6742d4e2343b

Note that there can be more than one IdentityFile line for a host. They will all work.

https://linux.die.net/man/5/ssh_config

Make known_hosts

For interactive SSH, you don’t have to make the known_hosts file prior to logging onto the server because on first access, SSH will ask if you trust the server and add the host keys for you. But for automated deployment, we cannot respond to the question, so it is best to try it now, and it is easy. In a bash terminal at ~/.ssh directory, enter

ssh-keyscan ui-dev.cs.mtu.edu >> known_hosts

This will append the host keys for ui-dev.cs.mtu.edu to the known_hosts file. If there is not a known_hosts file, it will make the file and add the host keys. 

Test SSH Authentication

Test the ssh configuration, by entering in a bash terminal

ssh <user name>@ui-dev.cs.mtu.edu

Where <user name> is the user name for the account on the server for your project. You should be authenticated and have access without having to login or answer any questions.

Manual Deployment

Tomcat servers have a special directory called the appBase. The appBase on ui-dev.cs.mtu.edu for your project is

/var/lib/tomcats/<tomcat instance>/webapps/

Where <tomcat instance> is the tomcat instance name in the credential file that I have given you.  When a WAR file is uploaded to the appbase directory, the  tomcat will then expand, called explode, the WAR file into a subdirectory in  the appbase directory. The web application will then be live. If there is already a WAR file with the same name then the old WAR file should be deleted before uploading the new WAR file. Tomcat will detect that the WAR file is deleted and delete the corresponding exploded web application directory. I call this de-explode. You must de-explode before redeploying.

So manually deploying is a multi step process

1. Package war
2. Delete old war on server
3. Upload new war to server
4. Restart Tomcat
5. Test deployment

The following sections describe the steps in more detail. 

Package War

There are two techniques. You can use the Grails war command or the Gradle assemble task. In a bash terminal enter either ./grailsw war or ./gradlew assemble. This will make two war files in the project “build/libs/” directory:

• <app name>.war <- This is the Tomcat war file. Use it.
• <app name>-plan.war <- This is the Micronaut war file. Don’t use it. 

I always delete the old WAR file before building it, so that I know for certain that a new WAR file is made. The first time you build a WAR file there will not be a “build/libs/” directory. Gradle will make it.

War Packaging for App Development using React Profile

Note that if you are using the React profile with client and server projects, the process is different. You need to create a Gradle task to combine the projects, besides other configuration changes. Follow the direction in the “Combining the React profile projects” Grails Guide:

http://guides.grails.org/react-combined/guide/index.html

Note that the guide is for Grails 3 which uses an older version of Gradle, so some modifications to the Gradle task need to be made. 

Delete Old War 

Using ssh, navigate to the appbase and delete the war file.

rm <app name>.war

Where <app name> is the name of your application. Wait a while to check that Tomcat has deleted the old expand web application using ls:

ls -la 

This can take several seconds. So if the <app name> directory still appears in the list, check again.

I like to use WinSCP to inspect the appBase directory and delete the old war file. See the “Upload War” section for setting up and using WinSCP. Using WinSCP, you will need to refresh the view by clicking the green recycle button in the toolbar to see if the exploded directory has been removed, de-exploded. 

Upload War

I like to use a FTP client (capable of SCP or SFTP protocol) to upload the war file. If your home machine’s operating system is Windows, I recommend WinSCP. It is free, easy to use, and feature rich. If your home machine is Linux or iOS, FileZilla is a good choice, but not as good as WinSCP.

In the FTP client, set up the connection using 

• File protocol: SFTP or SCP
• Host name: ui-dev.cs.mtu.edu
• Port number: 22
• User name: <user name> from the credential file

To use the ssh keys, you will need to point the FTP client to the private key. In WinSCP, double click the “Advanced” button. In the window that appears, click “Authentication” in the list that appears on the left side of the window. Click the three periods, “…”, in the “Private key file” text field. Point the window that opens to the private key you made.  It is OK to let WinSCP convert and create a new private key in a file format it prefers, “.ppk”. 

Save the setting, and click the “Login” button. The FTP client will open in the home directory. Navigate to the appbase directory

/var/lib/tomcats/<tomcat instance>/webapps/

Drag and drop the war file in your project “build/libs/” subdirectory on your home machine into the appbase directory on the server. 

Wait and refresh to the WinSCP’s view. You should see the exploded directory. This can take a while, so if you don’t see it, try again. 

Special Note about the number of WAR files

JSP and especially Grails web applications use lots of memory. They can also have a memory leak if they are not deployed properly. The tomcat instances for the class have enough memory for only one additional web app besides the web applications that comes standard with tomcat.

In your appbase you’ll notice that there are already some subdirectories which are web applications. They are:

• ROOT/ – web app for bare domain
• docs/ – documents for how to use tomcat
• examples/ – example web applications

Leave these web apps in your appBase. They do not consume much memory. You can access ROOT, docs, and examples through your browser. You should do so in order to check that your tomcat instance is running properly. To get to the ROOT app use the URL

https://ui-dev.cs.mtu.edu:<port number>/

This page tells you about the tomcat. It also specifies the tomcat version number.

Remember: ONLY add one more web app to your appBase, otherwise your tomcat might run out of memory and crash.

Restart Tomcat

Tomcat needs to be restarted for larger web apps, especially applications using a database. Without restarting your tomcat, you might get a 404, when you browse to you web app. You use the “systemctl” command to issue commands to your tomcat. The systemctl command is a tool used by system administrators to manage services on a Linux machine. Consequently the command needs to be preceded by the “sudo”, which stands for “super user do”. 

MTU IT has exposed four tomcat commands that you can use to control the tomcat. The four commands are

• start – to start tomcat
• stop – to spot tomcat
• restart – to stop and start
• status – to check tomcat status

To execute a command enter in a ssh terminal

sudo systemctl <command> tomcat@<tomcat instance name>

Note the command uses the <tomcat instance name>, not the <app name>.

To restart the the 2022_ui_1 tomcat enter

sudo systemctl restart tomcat@2022_ui_1

If you are using WinSCP, the client has a button to “open a terminal”. You can issue the restart command in that window. 

Tomcat typically takes a few seconds to restart. To check that tomcat has started, you can navigate your browser to the ROOT application for the tomcat.

https://hci-dev.cs.mtu.edu:<port number>/

Test Deployment

Because Tomcat is a Java application, the ui-dev.cs.mtu.edu server can have multiple instances of Tomcat. So  that the Tomcat servers do not interfere with each other while serving pages, the different instances are configured to receive their request and send their responses through different ports. The ports are specified in the URL by the port number. Recall that while you were programming and testing your individual assignments you would point your browser to the URL

http://localhost:8080/cs4760progassign/

• http: – represents the protocol, in this case hypertext transfer protocol
• localhost – is the domain name for the server, in this case it means the current machine
• 8080 – is the port number

The port number is specified in the credential file. 

The tomcat servers on ui-dev.cs.mtu.edu are configured to be secure using SSL. This is the https protocol. When browsing to your web app on ui-dev.cs.mtu.edu, you need to specify https, not http. You can not just type the domain into the browser’s URL window, you must add “https://”.

The URL to your app is

https://ui-dev.cs.mtu.edu:<port number>/<app name>

Using a browser navigate to your web app and thoroughly test it.  If you have issues during deployment or after deploying, please see the “Debug” section at the end of this document.

Automated Deployment

Automated deployment, part of “continuous deployment”, encourages developers to make frequent improvements to the project and keep them deployed. Now that you have configured the project for deployment and manually deployed, there is not much work to automate deployment. We will use GitHub Actions to automate deployment. Then any developer on your team can deploy by pushing to GitHub. 

You can learn more about GitHub Actions at:

https://docs.github.com/en/actions

In particular, you should work through the “Quickstart” tutorial.

https://docs.github.com/en/actions/quickstart

And you should read the “Understanding GitHub Actions”:

https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions

We will use “Self-Hosted Runners” to run the workflow actions. You can read about runners:

• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners
• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-an-organization

To set up automated deployment using GitHub Actions, we will need to

1. Email Robert Pastel informing him that you are preparing to automate deployment.
2. Write the workflow yml and scripts to describe the workflow
3. Commit and push to the “deploy” branch on GitHub
4. Check deployment

Email Robert Pastel

Self-hosted runners have several advantages compared to GitHub runners. Besides providing exact control of the build environment, self-hosted runners avoid some complexities of authorization because the runner polls GitHub rather than GitHub uploading (SCP) the War file on to Tomcat server. In other words, the self-hosted runs on the server, periodically checks GitHub if a workflow needs to run. When a workflow is triggered then the self-hosted runner, downloads the repository and runs the workflow.

The instructor for the course needs to install the Self-hosted Runner for the GitHub organization. So email me telling me that you are preparing to automate deployment.

In addition the runner will disconnect and stop polling GitHub if there has not been activity for two weeks. So the instructor will need to check that the runner is still online.

Workflow and Scripts

On your development machine, open your project code. Make sure that your code is uptodate, use

git status

Check that no files need committing and that the main branch is up to date with “origin/main”.

Before we get started I should define two directories:

• Repository root
• Project root

The repository root is the top most directory of your repository and the project root is directory that has gradlew and gradle.bat files and typically the grails-app/ folder but not always. Sometimes the repository and project root directories are one in the same directory. But if you cloned the repository that I gave your team and then executed some like grails create-app <app-name> in the directory that downloaded when your cloned the repository then Grails has made a subdirectory in the repository root, called <app-name>. That directory is the project root.

build-deploy-test.yml

In the repository root directory, make the new directories called “.github/workflows/”. In the new workflow directory, make a new file called “build-deploy-test.yml”. Note that the “.github/” directory needs to be in the repository root for GitHub to find the workflows.

Copy the code below and paste it into the new file.

name: Build deploy and test project
run-name: ${{ github.actor }} is building
on:
  push:
    branches:
      - deploy

env:

  PROJECT_PATH: test_deploy
  BUILD_PATH: build/libs
  APP_NAME: test_deploy

  TOMCATS_DIR: /var/lib/tomcats/
  TOMCAT_INSTANCE: test_deploy

  SERVER_ACCOUNT: pastel
  SERVER_URL: ui-dev.cs.mtu.edu
  SERVER_PORT: 8112

  RESTART_WAIT: 5

jobs:
  Build-Deploy:
    runs-on: self-hosted
    timeout-minutes: 10
    steps:
      - name: Checkout project sources
        uses: actions/checkout@v4
      - name: Environment check and setup
        run: |
          pwd
          ls -la
          cd $PROJECT_PATH
          which java
          java -version
          chmod 775 gradlew
          ./gradlew -v
      - name: Run build with Gradle Wrapper
        run: |
          cd $PROJECT_PATH
          ./gradlew --console='verbose' assemble
      - name: List war
        run: |
          cd $PROJECT_PATH
          ls -laR ./$BUILD_PATH
      - name: Delete old war
        run: |
          chmod 775 .github/scripts/delete-wait.sh
          echo "Run delete-wait.sh"
          .github/scripts/delete-wait.sh $TOMCATS_DIR $TOMCAT_INSTANCE $APP_NAME
      - name: Copy war to Tomcat
        run: |
          cp ./$PROJECT_PATH/$BUILD_PATH/$APP_NAME.war $TOMCATS_DIR/$TOMCAT_INSTANCE/webapps/
          echo "Copied war to $TOMCAT_INSTANCE Tomcat"
      - name: Restart Tomcat
        run: |
          chmod 775 .github/scripts/wait-restart.sh
          echo "Run wait-restart.sh"
          .github/scripts/wait-restart.sh $TOMCATS_DIR $TOMCAT_INSTANCE $APP_NAME $RESTART_WAIT
  
  Test-Deploy:
    needs: Build-Deploy
    runs-on: self-hosted
    timeout-minutes: 1
    steps:
      - name: Test website with curl
        run: curl -fsSI https://$SERVER_URL:$SERVER_PORT/$APP_NAME

You need to edit this file. I’ll explain the code while you edit it. Be careful while you edit the file because the yml parser is very particular. The lines 

on:
  push:
    branches:
      - deploy

specifies that this workflow should run on push to the deploy branch. The next lines of the workflow, 

env:

  PROJECT_PATH: test_deploy
  BUILD_PATH: build/libs
  APP_NAME: test_deploy

  TOMCATS_DIR: /var/lib/tomcats/
  TOMCAT_INSTANCE: test_deploy

  SERVER_ACCOUNT: pastel
  SERVER_URL: ui-dev.cs.mtu.edu
  SERVER_PORT: 8112

  RESTART_WAIT: 5

specify the values for environment variables. You need to change the values of 

• PROJECT_PATH from “test_deploy” to the path of the project root from your repository root. If the repository and project roots are the same then replace “test_deploy” with a period, “.”. If you created the Grails app using “grails create-app <app-name>” then replace “test_deploy” with “<app-name>”.
• APP_NAME from “cs4760progassign” to the app name for your project. This should be same as the same server context path specified in grails-app/conf/application.yml. 
• TOMCAT_INSTANCE from “2022_test_1” to the tomcat instance name for your project.
• SERVER_ACCOUNT from “pastel” to your project account name on ui-dev.cs.mtu.edu.
• SERVER_PORT from “8101” to the port number for your tomcat.

All the other values can remain. The next section lists jobs for the workflow. There are two workflows

• Build-Deploy
• Test-Deploy

The “steps” in the workflow are an array of commands. Each array element begins with a name, specified

- name

In the command named “Checkout project sources”: 

      - name: Checkout project sources
        uses: actions/checkout@v4

Uses a GitHub action to download the repository into the runners “_work” directory.

The next step

      - name: Environment check and setup
        run: |
          pwd
          ls -la
          cd $PROJECT_PATH
          which java
          java -version
          chmod 775 gradlew
          ./gradlew -v

Will first change into the project root and then print the Java version, change the permissions to execute and print the Gradle version. The versions are printing can be used for debugging the workflow. 

The next step

      - name: Run build with Gradle Wrapper
        run: |
          cd $PROJECT_PATH
          ./gradlew --console='verbose' assemble

will execute the “gradlew assemble”. It will need to download the Gradle version that you project uses and then execute the Gradle command. So the assemble make take several minutes.

The next step

      - name: List war
        run: |
          cd $PROJECT_PATH
          ls -laR ./$BUILD_PATH

just lists the war for debugging. 

The next step

      - name: Delete old war
        run: |
          chmod 775 .github/scripts/delete-wait.sh
          echo "Run delete-wait.sh"
          .github/scripts/delete-wait.sh $TOMCATS_DIR $TOMCAT_INSTANCE $APP_NAME

Changes the permission of the delete-wait.sh script and then runs the script. You will make this script soon.

The next step

      - name: Copy war to Tomcat
        run: |
          cp ./$PROJECT_PATH/$BUILD_PATH/$APP_NAME.war $TOMCATS_DIR/$TOMCAT_INSTANCE/webapps/
          echo "Copied war to $TOMCAT_INSTANCE Tomcat"

Just copies the assembled WAR file to the Tomcat webapps folder.

The final step

      - name: Restart Tomcat
        run: |
          chmod 775 .github/scripts/wait-restart.sh
          echo "Run wait-restart.sh"
          .github/scripts/wait-restart.sh $TOMCATS_DIR $TOMCAT_INSTANCE $APP_NAME $RESTART_WAI

Change the permissions of the wait-restart.sh script and run it. The script waits for war file to explode and then restarts the tomcat. 

The next job, Test-Deploy, sets up a new environment and runs curl to check that application is running. You can learn more about curl at

https://everything.curl.dev/

Note the line

    needs: Build-Deploy

It states that the job needs to wait for the Build-Deploy job before running. By default, GitHub Actions will run all jobs in parallel.

delete-wait.sh

In the “.github” directory make a new subdirectory called “scripts”. In the scripts/ directory, make a new file called “delete-wait.sh”. Copy and paste the code below:

#!/bin/bash

##################################################################################
# Script filename: delete-wait.sh
# Author: Robert Pastel
# Date: 11/2022
##################################################################################
# This script deletes the old war file and waits for tomcat to delete the app.
# This file runs before uploading new war file.
#
# Script arguments:
# * $1 Tomcats directory
# * $2 Tomcat instance name
# * $3 App name

echo Checking for old war file.
if [ -e $1$2/webapps/$3.war ]
  then
  rm $1$2/webapps/$3.war
  echo Deleted old war file.
  echo -n  Waiting for tomcat to remove app.
  while [ -d $1$2/webapps/$3 ]
  do
    echo -n " Waiting."
    sleep 1
  done
else
  echo No old war file to delete.
fi

echo
echo Done.

exit 0

This is a bash script. There are lots of resources for learning bash on the web. A fairly good tutorial is 

https://ryanstutorials.net/bash-scripting-tutorial/

You do not need to edit the script, and I will only describe some of the more obscure aspects. The line

if [ -e $1$2/webapps/$3.war ]

specifies a “if” clause. The “-e” is a test for file existence. 

The line

  while [ -d $1$2/webapps/$3 ]

specifies a while loop. The “-d” is a test for a directory existence. The body of the while-loop

  do
    echo -n " Waiting."
    sleep 1
  done

echos to standard out “Waiting.” and then sleeps for a 1 second. 

wait-restart.sh

Make another file in “.github/scripts/” called “wait-restart.sh”. Copy and paste the code below into the file.

#!/bin/bash

##################################################################################
# Script filename: wait-restart.sh
# Author: Robert Pastel
# Date: 11/2022
##################################################################################
# This script waits for new war file to explode and reboots tomcat.
# This file runs after uploading new war file.
#
# Script arguments:
# * $1 Tomcats directory
# * $2 Tomcat instance name
# * $3 App name
# * $4 restart wait

echo Checking for new war file.
if [ -e $1$2/webapps/$3.war ]
  then
    echo -n Waiting for war file to explode.
    while [ ! -d $1$2/webapps/$3 ]
    do
      echo -n " waiting."
      sleep 1
    done
    echo
    echo "Waiting $4 seconds to restart tomcat"
    sleep $4
    echo Restarting tomcat.
    echo $(sudo systemctl restart tomcat@$2)
    echo "Waiting $4 seconds before testing"
    sleep $4
else
  echo No war file. Check upload.
fi

echo Done.

exit 0

You do not need to edit the file. I’ll also let you decipher the code.

Commit and Push

In a bash terminal, run

git add .
git commit -am "prepared github actions"
git checkout -b deploy
git push -u origin deploy

The last git command will create a tracking “upstream branch” on GitHub and push your project code to GitHub. GitHub Action Runner will notice that you pushed the code to the deploy branch and run the workflow. 

Next time you are ready to deploy. You will need to update the deploy branch on your development machine and run “git push”.

Check Deployment

In your browser go to your GitHub repository, and select the “Actions” tab. You should see a list of prior workflow and the current workflow running.  Click the running workflow, and you will see the standard output. It is an accordion, so you can click on lines to expand them. After the “Test-Deploy” job runs, expand the “Test website with curl” line. You should see the output from the curl command. If the result is

HTTP/1.1 302

Everything is good. The 302 status code is a “redirect”. Access to ui-dev.cs.mtu.edu is through a firewall so the browser gets a redirect to the actual location. Browsers know to follow the redirect, but not curl.

If you get a status code of 404, something is wrong. 

Even if you get a 302, navigate your browser to your deployed web app.

https://ui-dev.cs.mtu.edu:<port number>/<app name> 

Thoroughly check your app. If something is wrong be sure to check in the developer tools on the browser and the tomcat log files on the server in logs/ directory adjacent to the appBase directory.

Debugging

The above procedure can go wrong. Naturally while developing you should practice “incremental development” and test your code after every modification. When the code breaks, check your modifications. Recall that “must bugs are stupid mistakes, not something subtile”. 

Ques 

You can get ques of what broke by checking

• Workflow output on GitHub, if a failure then the workflow is broken or a step in the workflow broke, for example “./gradlew assemble.”
• The war file exploded, if not and the workflow or the Gradle assemble task was successful, then something is wrong with the app or the app configuration. 
• ROOT app, if the tomcat root app does not run then something is wrong with the tomcat. 
• Bowsering to the web app, if 404 or 500 then something is broken with the app or the app configuration. A 500 typically means something wrong with the database. If there is an exploded war file directory, try restarting the Tomcat. “Never forget to reboot.”

Also check the tomcat log files.  They are located in the Catalina base adjacent to the webapps/ directory. There are many log files, but the best file to look at is the log file for the day, they are named with the format

catalina.<year>-<month>-<day>.log

The files are large, but you will find time stamped traces. Go to the bottom of the file for the day and check the most recent traces. Pay attention to “SEVERE” errors.  You will want to look at the beginning of the trace and the bottom of the trace. See if you can recognize anything. For example, if you see “hibernate” then you suspect that something is wrong with the database. 

Common Bugs

GitHub Workflow Errors

The GitHub Actions page will indicate a failed job with a red icon and the step that failed will be in red. Expand the step and you should find an output indicating the type of error. The message will be cryptic. Remember that the YML parser is not very tolerant of formatting or syntax errors. 

404 Browser Error

Check the URL.

•  Make sure you are using https protocol.
• The app name is correct. 

Database Problems 

First make sure that production dbCreate is set to “create” in the application.yml. Otherwise the database will not be made. Later you may want to deploy a new WAR without changing the database then make sure that BootStrap.groovy does not add to the database and change dbCreate  to “none” in the application.yml.

Check that the db/ directory is made. It should be made in the Catalina base directory. Examine the directory for files. If your app is still having trouble accessing the database you could delete these files and restart tomcat.

If the db/ directory is not created. Try creating the db/ directory and redeploying. In the Catalina base directory, enter

mkdir db
chmod 755 db
chgrp tomcat db

Class Not Found Error

If you see error like

Caused by: java.lang.ClassNotFoundException: ACLIP.Role

The tradition is for the package name to be lower case. But the package name in the error message is upper case. For some reason this causes confusion after deployment, even though it works in development. The solution is to refactor the package name and app name to be all lower case.

Apps without Domain Objects

If your app does not have Domain objects then it is not using a database. Even if the app is not using a database the production data source configuration in application.yml must be correct because the app will try to connect to the data source during start up. I solve the problem by copying the data source configuration for development into production.

Problems with Styling?

I have had my CSS styling broken on the production server when it worked fine on development. I have fixed this by changing the asset pipeline settings in “build.gradle” to disable CSS minifying. Search for the assets section in “build.gradle” and change it to

assets {
    minifyJs = true
    minifyCss = false
}

Problems with Stopping and Starting Tomcat

Sometimes IT admin forgets to grant sudo or all the necessary permissions to the user account. If you have problems with sudo commands, you can check what you can do with sudo by

sudo -l -U <username>

Problems with Tomcat versions

Sometimes tomcat minor versions conflict. You can check the version of your Tomcat by navigating to the provided ROOT page (the URL for the Tomcat with just the port). The page should show the Tomcat version running. If you believe that the mismatch of Tomcat versions is the problem, you can change the Tomcat version by specifying the “ext” directive with the correct version  in the build.gradle.

Good luck