Grails Photo Moderation using Azure Services

Introduction

User uploaded image files are a concern for application developers and web hosters, especially when those uploaded images are available to the public. The concern is that the image may contain hateful, violent or sexual content. Developers and hosters need to moderate the files. To moderate manually is time consuming and/or expensive, consequently manual moderation is often neglected. An automated moderation using an AI model could be more reliable than manual moderation. Azure offers a cloud service for text and image moderation:

https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview

For many citizen science applications, the upload file should not contain human images or selfies. Azure offer another service for face detection:

https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/overview-identity

This lecture demonstrates integrating Azure Content Safety and Azure Face services into a Grails app to ensure that user uploaded images do not contain hateful, violent or sexual content. In addition, this lecture demonstrates saving the files in the server’s file system and displaying them in the browser. 

Azure Services

Before discussing the demo web application, you should study some of the tutorials about Content Safety. In particular the “QuickStart: Analyze text content” and the “QuickStart: Analyze image content” are excellent simple examples:

Work through these examples. Be sure you study the Java version. 

Azure does not offer a good tutorial using the Face service in Java, but take a quick look at the “Call the Detect API” in C#:

https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/how-to/identity-detect-faces

Application Architecture

Our application will use the Grails framework. Our goal is to integrate Azure services into the Grails web application. 

We should review the process used in Azure services before integrating them. The general process for both Content Safe and Face Detection is:

  1. Make the ContentSafe or Face client for the service
  2. Convert the image data  into a Azure BinaryData
  3. Call the client’s analyze method to get the results
  4. Process the results

Our web application does not need to make a client for every analysis. The web application only needs a single client. In addition, converting the data, analyzing the data, and processing the results should be a transaction, meaning that the app should be able to rollback the results if any part fails. 

Grails uses Spring’s Inversion of Control (IoC) which uses Spring Beans. Beans are Plain Old Java Objects (POJO). The Spring’s IoC then manages the beans, creating them and injecting beans into other beans. Nearly every object in Grails is a bean, this includes Domains, Controllers and Services. By default, Grails creates beans as singletons. In addition, the typical use of Grails Services is that their methods are transactional. This suggests that the application should use a Spring Bean to make the client and a Grails Service to convert the image data, call the analysis method, and process the results. 

We should consider how to call the Content Safety services by reviewing the Grails conventions for how Domains, Controllers and Services relate to each other. After the programmer makes a Domain object and runs the command  “./grailsw generate-all …”, Grails makes a Controller to manage the Views and a Service for the Controller to interface with the Domain. The relationship between them looks similar to a MVC:

Domain <-----> Service <-----> Controller <-----> Views

To study the program flow, consider the process of a user on the Controller’s index view wanting to create a new Domain entity. 

  1. In the index view, the user clicks the “new” button.
  2. The link for the “new” button, calls the Controller’s “create” method.
  3. The Controller’s “create” method makes an “unsaved” Domain entity and generates the “create” view which is a form for the entity’s data.
  4. The user enters the data and clicks the “create” (submit) button.
  5. The action for submit calls the Controller’s “save” method. 
  6. The Controller’s save method receives the domain entity and calls the Service’s “save” method.
  7. The Domain receives the entity with the data, validates it, and if successful, stores it in the datastore. If unsuccessful, the Domain throws an exception. Either way, the program returns to the Controller’s save method.
  8. If the transaction is successful then the Controller’s “save” method redirects to the “show” view. If the validation fails then the Controller save method shows the error message in the “create” view. 

Pictorially the process looks like:

View             Controller            Service           Domain

index ----------> create         
create ---------> save   --------------> save ----------> 
                                                         validate
show <----------  save <----------------------------------

The natural place to inject our Service for Content Safety is in the Controller and to call its analyze method just before the Domain Service save method is called by the Controller. THe Content Safety Service can throw an exception if the data is not safe.

So pictorially it will look like:

View             Controller             Service          Domain          Content Safe

index ----------> create         
create ---------> 
                   save  ------------------------------------------------->
                                                                          analyze
                       <---------------------------------------------------

                       -----------------> save ---------> 
                                                         validate
show <------------ save <--------------------------------

To summarize the design for integrating Azure services:

  • Azure clients will be Spring beans.
  • A Grails Service will use the beans to call the Azure services.
  • The Grails Service will be called in the Controller “save” method before the Domain Service is called.
  • The Grails Service throws an exception or adds errors to the domain entity if Content Safety Service indicates there is unsafe content. Otherwise it just returns. 

Our implementation has an additional challenge because we want the image saved in the server’s file system. The challenge is that Domain specifies the data stored and the “generate-all” command uses it to make the form in the “create” view. But we want the file path to be saved in the data store and the image stored in the file system. The implementation is not as simple as modifying the “create” view and retrieving the MultipartFile from the request in the Controller’s “save” method for two reasons. The first reason is ideally we would like the Domain to do some validation, for example assuring that the file is a JPEG. The second and more important reason is that all the messaging is handled through the Domain entity. So the Domain entity should have a field for the MultipartFile but should not save it in the datastore. Fortunately, GORM has a mechanism for handling this problem, transients.

https://grails.apache.org/docs/7.0.2/ref/Domain%20Classes/transients.html

We are finished with designing and ready to start implementing. 

Implementation

To get started reviewing the implementation, clone the repository

https://github.com/2026-UI-SP/safe

The project was created using “grails create-app safe” with Grails web profile. The grails version is 5.3.6 and the Java version is 8.0.392. 

To create the project, I first added the dotenv-java project to the “create-app” project to configure the Azure keys and endpoints.

https://github.com/cdimascio/dotenv-java

I also added dotenv-java package dependency to build.gradle

dependencies {

   // RLP adds:
    implementation "com.azure:azure-ai-contentsafety:1.0.0"
    implementation "com.azure:azure-ai-vision-face:1.0.0-beta.2"
    // implementation 'io.github.cdimascio:dotenv-java:3.2.0' // Requires java 11 or above
    implementation 'io.github.cdimascio:dotenv-java:2.3.2' // For java 8
    implementation "org.grails.plugins:async"
}

While I was editing build.groovy, I also added the dependencies the ContentSafety, Face, and Grails Async Plugin, which I will use later. Then I added a “.env” to “.gitignore” file:

.env
**/.env

I made the “.env” file and added the keys and endpoints. See “env.example” for an example.

To run the project, create a “free” Azure subscription. Even if you choose the “free” option, Azure will request a credit card account, but promises never to charge you.

After making the Azure subscription, create a Content Safety resource. 

https://aks.ms/acs-create

There is not much choice to fill out in the form but the “resource-group” name. Make the name memorable, eg “safe”.

Copy the key and endpoint for Content Safety into the “.env” file. You also need to add the Face API resource to the group resource and copy its key and endpoint to the “.env” file. 

Now you can boot the project by running

./gradlew bootRun

The app opens the home page with links to Controllers:

  • TextController – demonstrating Content Safety for text content
  • PhotoController – demonstrating Content Safety and Face Detection for photo upload

Click one of the links and explore. 

We will first study integrating Context Safety for the Text domain because it is much simpler than the Photo integration. 

Text Moderation

ContentSafe Bean

We need to write the class definition for the ContentSafe bean in “src/main/groovy/ContentSafe.groovy”.

There is not much to the constructor. It first retrieves the key and endpoint form “.env” and then creates the “client” using ContentSafetyClientBuilder and the credential key and the endpoint. The client is a public property of ContentSafe. 

Now the bean should be registered with Grails. The bean is registered in “grails-app/conf/spring/resources.groovy”.

import safe.ContentSafe
import safe.FaceDetect

beans = {
    contentSafe(ContentSafe)
    faceDetect(FaceDetect)
}

The registration format is simple

beans = {
	<bean name>(<bean constructor>)
}

The current resources.groovy is registering two beans, one for Content Safety and the other for Face Detect. 

You can learn more about bean registration at

https://grails.apache.org/docs/7.0.7/guide/spring.html

Text Domain

Now we can make the Text domain, by entering

./grailsw create-domain safe.Text

The Text domain is very simple:

package safe

class Text {
    String text
    
    static constraints = {
    }
}

To create the initial Controller, Service and Views, I entered

./grailsw generate-all safe.Text

SafeServer.textSafe

Now we can make the SafeService in “grails-app/services/services/SafeService.groovy”. Notice how the ContentSafe bean is injected by just adding the bean name as property of SafeService. Currently the SafeService code has two methods:

  • textSafe
  • moderatePhoto

For text content moderating, we only need to study the textSafe method. The textSafe method is passed a Text domain entity, “text”. The method calls the analysis from the contentSafe.client using the analyzeText 

AnalyzeTextResult response = contentSafe.client.analyzeText(new AnalyzeTextOptions(text.text));

The “analyzeText method only needs the text string. 

For a safe text, we want the severities for all the categories to sum to zero. So, if the severitySum is greater than zero code throws a ValidationExpection.

if (severitySum > 0) {
     text.errors.rejectValue("text", "text.unsafe", "Improper text entry. Please modify your text and try again.");
     throw new ValidationException("Text is not safe", text.errors);
}

Notice that an error is added to the text entity before the exception is thrown. The third argument in rejectValue is the error message that appears in the view.

TextController

Now we are ready to edit “TextController.groovy” in “grails-app/controllers/” There are only two things to do. Inject “safeService” and call it in the “save” method:

package safe

import grails.validation.ValidationException
import static org.springframework.http.HttpStatus.*

class TextController {

    TextService textService
    SafeService safeService
…
   def save(Text text) {
        if (text == null) {
            notFound()
            return
        }

        try {
            safeService.textSafe(text) // Check if text is safe. If not, safeService throws a ValidationException. 
            textService.save(text)  
        }
        catch (ValidationException e) {
            respond text.errors, view:'create'
            return
        }

        request.withFormat {
            form multipartForm {
                flash.message = message(code: 'default.created.message', args: [message(code: 'text.label', default: 'Text'), text.id])
                redirect text
            }
            '*' { respond text, [status: CREATED] }
        }
    }

Notice that the “safeService.textSafe” method is called with the “text” entity in the “try” block. If “safeService” throws an exception, it will be caught and the errors will appear in the “create” view. 

Try some text entries. Experiment with different texts. I’ll not make suggestions for text that will throw exceptions. I’m sure you can think of some.

Photo Moderation

Implementing image moderation is considerably more difficult because of two issues: We want:

  • to store the image file in the filesystem. 
  • add face detection Azure service

FaceDetect Bean

Adding Face Detection is not very difficult. All we have to do is write the FaceDetect class and register it. The “FaceDetect.groovy” is at “src/main/groovy/”. The class definition is nearly identical to the ContentSafe bean. You can study it on your own. Registering the bean in “grails-app/conf/spring/resources.groovy” is just as similar as for the ContentSafe bean. 

Photo Domain

The Photo domain has several tricks, some not intuitive.

package safe

import org.springframework.web.multipart.MultipartFile
class Photo {

    long createTime
    String photoPath
    String photoContentType
    MultipartFile photoFile

    static transients = ['photoFile']

    MultipartFile getPhotoFile() {
        return photoFile
    }

    void setPhotoFile(MultipartFile photoFile) {
        this.photoFile = photoFile
    }   

    static constraints = {
        createTime nullable: true
        photoPath nullable: true, blank: true
        photoContentType nullable: true, blank: true 
        photoFile  validator: { val, obj ->
            if ( val == null ) {
                return false
            }
            if ( val.empty ) {
                return false
            }

            ['jpeg', 'jpg'].any { extension -> 
                 val.originalFilename?.toLowerCase()?.endsWith(extension)
            }
        }
    }
}

First notice that org.springframework.web.multipart.MultipartFile is imported and used to add the “photoFile” property to the domain. Then photoFile is added to the transients:

   static transients = ['photoFile']

This assures that the photoFile will not be saved to the datastore, which is good because Hibernate does not know how to deal with a Multipart File. Also notice that photoFile getters and setters are added in order  to access the file.

The constraints are particularly interesting. 

   static constraints = {
        createTime nullable: true
        photoPath nullable: true, blank: true
        photoContentType nullable: true, blank: true 
        photoFile  validator: { val, obj ->
            if ( val == null ) {
                return false
            }
            if ( val.empty ) {
                return false
            }

            ['jpeg', 'jpg'].any { extension -> 
              val.originalFilename?.toLowerCase()?.endsWith(extension)
            }
        }
    }

First notice that even though photoFile is not saved to the datastore we  can still validate it. Second notice that although we do not want to save “photoFile” without saving the “createTime”, “photoPath” and “photoContentType”, we still set their constraints to nullable true. This is because if we don’t, the application will complain in the view that they are not provided. Our application logic will have to provide them. 

Application.yml

Because the application is uploading files and saving them to the filesystem, we should set a large file max size and the path for the content in application.yml.

---
grails:
    controllers:
        upload:
            maxFileSize: 26214400
            maxRequestSize: 26214400
---
environments:
    development:
        contentFolder: /home/pastel/data/research/photo-moderation/workspace/azure-learn/safe/content_safe

Note that the setting for the contentFolder is dependent on the environment, so that we can have different paths for development and production. Any class desiring to access the setting can implement the GrailsConfigurationAware interface to access the settings.

https://grails.apache.org/docs/5.3.5/guide/conf.html#config

Look for “GrailsConfigurationAware Interface”. 

SafeSever.moderatePhoto

The “moderatePhoto” is considerably more complicated than the “safeText” method because of three reasons.

  • a BinaryData should be created
  • Face Detect service needs to be called
  • Face Detect and Content Safety services should run simultaneously

Creating a BinaryData is not too difficult. There are three options to create a BinaryData. 

  • Using a File in the filesystem
  • Using an IOStream
  • Using a Byte array

We do not want to save the file to the filesystem before assuring that the content is safe, so I rejected that approach. An IOStream would be good, especially for large images, but a Stream can only be used once, but two service calls need to be made, one to Content Safety and the other to Face Detect. So I opt for using a Byte array. A Byte array is stored in memory, so it can be reused, but it requires that the server have sufficient memory.

        byte[] photoBytes = photo.photoFile.getBytes();
        BinaryData photoData = BinaryData.fromBytes(photoBytes);

IMPORTANT: Let me know if the server has issues with your files, so I can request more memory for the server.

With our choice made, the implementation is simple:

   faceDetect.client.detect(
         photoData,
         FaceDetectionModel.DETECTION_03, 
         FaceRecognitionModel.RECOGNITION_04,
         false, // returnFaceId, do not need.
         null, // FaceAttributeTyes, do not need. 
         false, // returnFaceLandmarks, do not need.
         false, // returnFaceAttributes, do not need.
         60); // faceIdTimeToLive in seconds. The shortest time.

The above arguments are minimal for detecting faces in a image. It returns a list of bounding boxes locating the faces in the image. See

https://learn.microsoft.com/en-us/dotnet/api/azure.ai.vision.face.faceclient.detect?view=azure-dotnet-preview#azure-ai-vision-face-faceclient-detect(system-binarydata-azure-ai-vision-face-facedetectionmodel-azure-ai-vision-face-facerecognitionmodel-system-boolean-system-collections-generic-ienumerable((azure-ai-vision-face-faceattributetype))-system-nullable((system-boolean))-system-nullable((system-boolean))-system-nullable((system-int32))-system-threading-cancellationtoken)

During my initial development and testing, I discovered that the process of calling the Content Safety and Face Detect services sequentially was very slow, more than 5 seconds. So I researched making the calls concurrently in Grails. 

https://grails.apache.org/docs/latest/guide/async.html

You only need to read the first section about Promises. To call two services simultaneously, all that needs to be done is create two “Promise.tasks” and then “waitAll” for them. 

        Promise<AnalyzeImageResult> contentResults = task{ 
            ContentSafetyImageData image = new ContentSafetyImageData();
            image.setContent(photoData);  
            contentSafe.client.analyzeImage(new AnalyzeImageOptions(image)) 
        };

        
        Promise<List<FaceDetectionResult>> faceResults = task{ 
            faceDetect.client.detect(
                photoData,
                FaceDetectionModel.DETECTION_03, 
                FaceRecognitionModel.RECOGNITION_04,
                ...)
            };

        waitAll(contentResults, faceResults);

This reduced the response time to 3 seconds, barely acceptable. After the app has the results, it can process them and throw exceptions if needed. 

        // Check content safety results
        int severitySum = 0;
        for (ImageCategoriesAnalysis result : contentResults.get().getCategoriesAnalysis()) {
            severitySum += result.getSeverity()
        }
        if (severitySum > 0) {
            throw new ValidationException("photo", "photo.unsafe", "Improper photo. Please use a different photo and try again.");
        }
        // Check face detection results
        if (faceResults.get().size() > 0) {
            photo.errors.rejectValue("photoFile", "photo.unsafe", "Photo contains a face. Please use a different photo and try again.");
            throw new ValidationException("Photo is not safe", photo.errors);
        }

PhotoController

The “PhotoController” “save” method is similar to the “Text” “save” method. 

def save(Photo photo) {
        //println ""; println ""; println "save called with photo: ${photo}"; println "save called with params: ${params}"
        if (photo == null) {
            notFound()
            return
        }
        // The Domain validation does not actually throw a ValidationException, 
        // so we need to check for errors 
        if (photo.hasErrors()) {  
            // Domain validation does not throw a ValidationException, 
            // so we need to check for errors 
            respond photo.errors, model: [photo: photo], view:'create'  
            return
        }

        try { 
            // This can throw a ValidationException if the photo is not safe.
            safeService.moderatePhoto(photo)  
        } catch (ValidationException e) {
            respond photo, model: [photo: photo], view:'create'  
            return
        }

        // Save the file and enter the photo metadata in the DB.
        photo = photoUploadService.uploadFile(photo)

        if (photo == null || photo.hasErrors()) {
            respond photo?.errors ?: [photo: photo], model: [photo: photo], view:'create'  
            return
        }

        request.withFormat {
            form multipartForm {
                flash.message = message(code: 'default.created.message', args: [message(code: 'photo.label', default: 'Photo'), photo.id])
                redirect photo
            }
            '*' { respond photo, [status: CREATED] }
        }
    }

There are three changes to the “save” method. First we check the domain entity, “photo”, for errors before calling the Service. Recall that the save action is only invoked after the user has submitted the form so that Grails data binding will check validation. The data binding validations are specified in the Domain constraints. The only constraint is that a JPEG should have been submitted. If “photo” passes the constraint then the “photoFile” is added to it. 

Next the safeService.moderatePhoto is called with the “photo” in the “try” block, similar as what was done for text. 

The third change to the “save” method is that PhotoUploadService is used to save the file to the server’s filesystem. Finally the code checks if uploading the file did not create errors. 

PhotoUploadService

We separate the PhotoUploadService from the Controller because there is significant logic and opportunity for failure. Note that the PhotoUploadService implements the GrailsConfigurationAware interface and overrides the “setConfiguration” method  to set the “photoFolder” property. The “uploadFile” first checks for the content folder, if it does not exist it makes it. Then it names the file with the current time and the file extension. It uses the “transferTo” in the Spring.MultipartFile to write the file to the file system. 

Now it can set the “photo” persistent properties, so it can be saved. The “photoService” is used to save the “photo” entity to the datastore. Finally it checks if any errors have occurred during this process. If there were errors, it responds to the “create” view with the errors. 

Views and Messages

The implementation above is most of the effort for the functionality of validating the photo, but I was not happy with the views. 

First the data binding validation error after submitting a file not a JPEG was not very human readable. So, I added a message to “grails-app/i18n/message.propertise”.

# RLP added for photoFile validation error
photo.photoFile.validator.invalid=Please upload a photo in JPEG format.

It actually took a lot of guess work to determine the proper key for the message.

Second I prefer that the save and index view displays an image rather than a filepath. Just as we have done in the programming assignments, I made a PhotoController action, “photoImage”.

   def photoImage(Long id) {
        Photo photo = photoService.get(id)
        if (!photo) {
            notFound()
            return
        }
        def file = new File(photo.photoPath)
        render file: file, contentType: photo.photoContentType
    }

I also modified the show view, so that it displayed the image rather than the path.

<f:display bean="photo" />
   <g:if test="${this.photo.photoPath}">
      <img src="<g:createLink controller="photo" action="photoImage" id="${this.photo.id}"/>" width="400"/>
   </g:if>

I also did not like the Photo index view. I thought that list in the index view could be used by the scientists to quickly check the files. So I thought that the list of photos should only display the createTime and a small photo. This required editing the <f:table />.

https://grails.apache.org/docs-legacy-views/snapshot/fields/ref/Tags/table.html

Fortunately Grails make this easy. An alternative table template is written in “grails-app/templates/_feilds/taple-p.gsp”

<table>
    <thead>
         <tr>
            <g:each in="${domainProperties}" var="p" status="i">
                <g:sortableColumn property="${p.property}" title="${p.label}" />
            </g:each>
        </tr>
    </thead>
    <tbody>
        <g:each in="${collection}" var="bean" status="i">
            <tr class="${(i % 2) == 0 ? 'even' : 'odd'}">
                <g:each in="${domainProperties}" var="p" status="j">
                    <g:if test="${j==0}">
                        <td><g:link method="GET" resource="${bean}"><f:display bean="${bean}" property="${p.property}" displayStyle="${displayStyle?:'table'}" theme="${theme}"/></g:link></td>
                    </g:if>
                    <g:else>
                        <td>
                            <!-- <f:display bean="${bean}" property="${p.property}"  displayStyle="${displayStyle?:'table'}" theme="${theme}"/> -->
                            <img src="<g:createLink controller="photo" action="photoImage" id="${bean.id}"/>" width="100"/>
                        </td>
                    </g:else>
                </g:each>
            </tr>
        </g:each>
    </tbody>
</table>

This was not so difficult because I only had to modify the default table provided by Grails.

https://github.com/grails-fields-plugin/grails-fields/blob/master/grails-app/views/templates/_fields/_table.gsp

That is it. Try some photo uploads. Experiment with different photos. I cannot make suggestions for photos that will throw exceptions. I did not have any.

Conclusions

Implementing photo moderation is a little more involved than text moderation. The steps are

  1. Store your keys and endpoints in “.env” and don’t commit them.
  2. Make the Photo domain using transients for MultipartFile
  3. In application.yml, increase the maxFileSize and set the contentFolder
  4. Define the beans in “src/main/groovy”
  5. Register the beans in “grails-app/conf/resources.groovy”
  6. Make the SafeService and PhotoUploadServices in “grails-app/services”
  7. Modify the PhotoController save method
  8. Improve the views by modifying the show and index views, and add to the “grails-app/i18n/messages.properties

There is plenty yet to do. The photo aspect ratio and resolution should probably be validated. Also the usability would be improved by adding a spinner during file upload. Also note that the update view and Controller methods have not been modified. 

References

GitHub

Azure

Grails

Appendix

List of Created and Edited Files

Web app

  • grails create-app safe

Environment Variables

  • sdk env init 
  • .env
  • .gitigonore
  • build.groovy

Text

  • domain/Text.groovy
  • ./grailsw generate-all safe.Text
  • src/groovy/ContentSafe.groovy
  • conf/spring/resources.groovy
  • services/SafeService.groovy
  • controllers/TextController.groovy

Photo

  • domain/Photo.groovy
  • ./grailsw generate-all safe.Photo
  • conf/application.yml
  • src/groovy/FaceDetect.groovy
  • conf/spring/resources.groovy
  • services/SafeServices.groovy
  • services/PhotoUploadService.groovy
  • controllers/PhotoController.groovy
  • views/photo/show.gsp
  • views/template/_fields/_table.gsp
  • views/photo/index.gsp
  • i18n/messages.properties