Image Interaction using Leaflet

A client requested a website that could assist medical doctors explaining the association of pain in skin with nerve fibers originating from the spine. In the medical field this is called dermatome:

https://en.wikipedia.org/wiki/Dermatome_%28anatomy%29

On the website, the doctor can identify areas of the body by hovering the cursor over different areas of the image. The area would highlight and a tooltip would name the never fiber. To learn more, the doctor can click on the highlighted area, and information about the area should be displayed in a side panel. 

Consider the interactions with the images, there are three:

  1. Cursor hovering highlights the area
  2. Cursor hovering displays a tooltip
  3. Clicking on the highlighted area displays information

These are the basic image interactions that we want to implement. 

Implementation Approaches

During my research, I discovered three different approaches for implementing the interaction:

  1. Using canvas API  and a JavaScript image library
  2. Using HTML image map
  3. Using GIS software such as Leaflet

Using HTML canvas

The first approach that I researched deeply was using canvas API.

https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API

The technology seemed perfectly matched. The image would be loaded into the canvas element. And areas of the body would be represented by SVG elements.

Developers would need to make the SVGs by tracing the areas of body using a drawing software such as Inkscape:

https://inkscape.org

Although the interaction could be implemented using SVG directly by accessing and changing its attribute, I suspected that developers would rather use a JavaScript image processing software such as FabricJS.

http://fabricjs.com/

The essential interaction is that the user should be able to click the image of the SVG and the event would identify the proper SVG as the target of the event. So I studied the “event inspector” demo:

http://fabricjs.com/events

All the events seem to be provided, but try this.  In the event inspector do this:

  • In the checkboxes above the canvas panel, make sure that only “all events”, “canvas”, “red” and “blue” are checked.
  • In the canvas panel, make the blue circle big enough so that the bounding box for the blue circle overlaps the red square without the circle overlapping. 

Now move and click the cursor into the corner of the red square that overlaps the circle’s bounding box. You’ll notice that the red events do not occur in the log. This is because of two reasons:

  • The area indicated as the target of an event is the bounding box of SVG not the visible geometry of the SVG. 
  • The target of the event is the top SVG, and events do not have an array of targets. 

The implication is that development would require listening to all events and then identify the polygon that contains the click point.  There are algorithms for determining if a point is in a polygon.

The algorithm is not trivial and probably computationally expensive. Also the algorithm’s are based on absolute coordinates, the output from Inkscape using the “pencil” tool  uses relative coordinates.

https://developer.mozilla.org/en-US/docs/Web/SVG/Tutorial/Paths

This approach, although feasible, appeared to be very error prone during development. So I searched for another approach. 

Using HTML image map

The second approach that I researched was using HTML image map.

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/map

I came across the above webpage by chance. Try the demo. Clicking different areas in the image will navigate the browser to a new page. Study the code in the demo, you’ll notice that the shape attribute can be “poly”, a polygon. So the interaction can identify exact areas in the image. 

Although the exact area clicked is identified, navigating to another page or anchor within the page is not what we are looking for. So I tried another approach.

Using GIS Software 

Actually, using GIS Software was the approach that I thought of first because I knew Leaflet could implement all the user interactions. Consider this Leaflet’s example interactive choropleth map

https://leafletjs.com/examples/choropleth/example.html

Hovering over a state highlights the state and updates the information panel. Also clicking on a state zoom into the state. In addition the user can pan and zoom the map.

I initially rejected the approach because using GIS software to implement image interaction seemed excessive and a bad match. Besides, GIS software uses GeoJSON or Shapefile to express geographic information. 

I asked the client for an example of a similar application. The sent me this link:

https://www.innerbody.com/image/nervov.html

Notice that moving the cursor over the body highlights parts of the body. Also if you click on the body area the side panel gives information about that part of the body.  It has all the user interactions that the application needs. So I tried to reverse engineer the application. Opening the developer tools and digging deep into the html tags. I notice a div with id “image-map”, and below this div with class “leaflet-container” and other “leaflet” like classes. This is the div that Leaflet creates when it loads the map onto the webpage. So they used Leaflet. 

But there are challenges using Leaflet:

  • Leaflet accepts GeoJSON files to load on the map, but drawing softwares outputs SVG.
  • Leaflet uses latitude and longitude coordinates while SVG uses x and y coordinates. 
  • Also I did not want to use the “control panel” provided by Leaflet to display the information. 

Nonetheless, armed with the knowledge that using Leaflet can implement the interactions, I decided to pursue this approach. 

After the decision, I conducted a quick google search for “SVG to GeoJSON” and discovered that there exist some. So I progressed to implementation. 

Implementation

Using Leaflet to implement user interactions with images will require require three steps:

  1. Trace the nerve fibers using Inkscape
  2. Convert SVG to GeoJSON
  3. Writing the Image Interaction application using Leaflet

I explain in detail the steps in the above order.

Tracing Dermatome

No matter the approach, there is no way to avoid the tedious steps of tracing the body areas, dermatome, to create SVG files representing the body area. Inkscape is a good software for drawing and generating SVG files, and it is free. 

https://inkscape.org

I will assume that you have study these two tutorials:

I studied the “Bitmap Tracing tutorial” 

https://inkscape.org/doc/tutorials/tracing/tutorial-tracing.html

But I did not find it useful. The tracer command generated too many nodes and the image that we want to trace is not high enough quality. 

Tracing was not as tedious as I thought it would be. Inkscape has some tools to ease the process. I will introduce the tools as I explain the workflow of tracing and making the SVG file. 

First, download the image from wikipedia..

Note the image name is Gant_1962_663.png, and the size is 2292 x 2480 px where 2292 px is the width and 2480 px is the height. 

Now open the Inkscape, and in the opening screen, click the “Open” button not the “New document” button. This will create a new Inkscape project with the size matching the image size. 

After the image loads, check the size is 2922 px wide and 2480 px heigh  in the “Document Properties” window (File -> Document Properties -> Display tap). 

Also make sure that the “Layers and Object” side panel is open (Layers -> Layers and Object). In the side panel you should see the “Image” layer. Click the “Image” down button to expose the objects for that area, and you should see the “image1” button. You should lock it so you don’t move the object. 

Now we can start tracing. You choose a dermatome, nerve fiber area, say L4. Notice that L4 appears in several areas on the figure, along the side of the leg, medial, lateral, and around the waist, at the ankle and sole of the foot in the back view. You’ll need to trace all these areas. 

Choose an area and zoom in. With the pencil tool, trace around the area. Make sure that you terminate the path on itself. The initial node node will highlight when the pencil returns to it. The drawing does not need to be exact. You’ll correct it. When you finish the drawing you should see the “path1” object listed in the “Layers and Objects” panel. 

With the path selected, click the “node edit” tool. You see that there are a lot of nodes. To reduce the number of nodes click “simplify” (Path -> Simplify). You’ll want to click the simplify command several times until there are a reasonable number of nodes. Now working around the path, click and drag a node to correct position. To get a smooth shape, you’ll also to adjust the “node handles” so their circles are on the line in the image that you are tracing. At sharp corners in the image, I found it useful to position a node on the corner. You may want to delete individual nodes or add nodes. Make sure that the path does not loop pack onto itself. You’ll work around the path adjusting all nodes and node handles. The process is actually quick. 

When you are satisfied with the trace, and this should only take a few minutes, we want the path to be slightly inside the area so they do not overlap with adjacent areas. Click the “Inset” command (Path -> Inset). Do this once or twice. You’ll need to experiment. 

You are done with the area. Lock the “path1” object in the “Layers and Objects” panel, so you’ll not accidentally edit it. You may want to look at the XML for the path. You can do this by opening the “XML Editor” window (Edit -> XML Editor). You can click on the path object to see the attributes and also see the path SVG  commands that compose the path. 

Now you repeat the steps above for each of the body areas associated with the nerve fiber. 

When you have traced around all the areas, you can save the drawing. First delete the “image1” from the drawing using the “Layers and Objects” panel. Then click “Save as” (File -> Save as). Rename the file to nerve fiber, L4, and select plain SVG.  

You are done with this dermatome. 

Convert SVG to GeoJSON

We need to convert the L4.svg to a GeoJSON. The specification for GeoJSON is at

https://geojson.org/.

It is basically a JSON. The spec for JSON is at

https://www.json.org/json-en.html.

The GeoJSON specification is significantly different from the specification for SVG.

https://en.wikipedia.org/wiki/SVG.

Inkscape uses a SVG path to specify an arbitrary shape.  An SVG is  described as a sequence of pen movement typically in relative positions (i.e. dx and dy) coordinates  from the previous position. While in GeoJSON, an arbitrary shape is a polygon described by longitude and latitude coordinates in WGS84 datum. You can read about WGS at 

https://en.wikipedia.org/wiki/World_Geodetic_System.

With these fundamental differences it is hard to believe that a SVG to GeoJSON software exists, but modern geographers need drawing software to produce maps. Because we’ll be using JavaScript to implement the application, I searched for a JavaScript application to convert SVG to GeoJSON in Node Package Manager (npm). 

https://www.npmjs.com/package/svg2geojson

When selecting a Node package, I check the weekly downloads. Approximately a 100 is not bad for a very specialized software in JavaScript. I also check the dependencies. There should not be too many. Six is reasonable and they are reliable packages. Finally, I check the repository,

https://github.com/Phrogz/svg2geojson.

The last commit was 3 years ago. That is not good, but nevertheless it is the best software  I could find.

Note that to use svg2geojson you need to insert a “Prognoz MetaInfo” element into the SVG. 

I downloaded the package and tried it. It sort of worked, “sort of”. Leaflet displayed the GeoJSON on the “map”, but it was inverted and the latitude position was wrong. Hoping that the correction was easy I tried many values for the “GeoItem” element, but none improved the transformation. I studied the SVG and JSON specification and realized that positive y-coordinate is up  in GeoJSON, while the y-coordinate in SVG is positive down the screen display. I studied the svg2geojson code and determined that the correction had to be made in the code. I made the modification and tested it. It worked. The code is at 

https://github.com/2024-UI-SP/isvg2geojson

Make sure you have Node and Git on your development machine and clone the repository. 

To use the isvg2geojson, you will need to insert the “Prognoz MetaInfo” element into the SVG file. Open L4.svg in a text editor, Visual Studio Code works well.   Insert the  “Prognoz MetaInfo” element just above the <defs> tag in the SVG. 

    <MetaInfo xmlns="http://www.prognoz.ru"><Geo>
        <GeoItem X="0" Y="0" Latitude="0" Longitude="0"/>
        <GeoItem X="2292" Y="2480" Latitude="2480" Longitude="2292"/>  
    </Geo></MetaInfo>

Run isvg2geojson.

 $ ./bin/isvg2geojson -t=10000 -p=2 -i ../../images/nerves/L4.svg

Note the options, especially the “-i” option which run the code I added. The program will create the file L4.geojson in the same directory as L4.svg. The structure of the file is a “FeatureCollection” with an array of features. Each “Feature” in the GeoJSON, corresponds to the “path” in the SVG file. Also each feature has a “properties” attribute. 

We need to edit the GeoJSON file. We want to combine all the individual features into a single feature and give that feature values for the “properties” attribute. 

Open L4.geojson in a text editor. In the first feature properties insert values. For example I added these values.

{
 "type"    : "FeatureCollection",
 "creator" : "svg2geojson v0.7.2",
 "features": [
  {
   "type"      : "Feature",
   "properties": {
    "name" : "L4",
    "description" : "This is a description of L4"
   },
   "geometry"  : {
    "type"       : "Polygon",
    "coordinates": [
...

Now scan through the file searching for other features. If you used high tolerance, “-t” option, to create the GeoJSON the file will not be too big. The next features element will look like:

   ...
            [1703.15,1475.60],
            [1703.16,1475.60]
         ]
        ]
       }
      },
      {
        "type"      : "Feature",
        "properties": null,
        "geometry"  : {
            "type"       : "Polygon",
            "coordinates": [
            [
              [1723.76,506.21],
              [1724.12,498.67],
    ...

Combine the next feature with the first feature by deleting the tags and adding a comma. The result should look like this.

    ...
      [1703.15,1475.60],
      [1703.16,1475.60]
     ]
     ,
     [
      [1723.76,506.21],
      [1724.12,498.67],
    ...

Note the terminating square bracket, “]”, the intermediate comma, “,” and the initiating square bracket, “[“. This will keep the two polygons (i.e. paths) separated.

The GeoJSON file is ready to be used in the application.

image-interaction

The demo image interaction code is at

https://github.com/2024-UI-SP/image-interaction

You should  clone the repository and open it in an IDE such as Visual Studio Code, so you can examine the code as I explain the salient aspects of the code. 

ViteJS

The code is written in JavaScript for the browser using ViteJS for the development server. The documentation for ViteJS is at

I used the ViteJS vallina template to start implementation. If you are not familiar with ViteJS, you may want to create one and run it. 

ViteJS uses RollupJS to bundle the JavaScript before serving the website. The documentation for RollupJS is at

You can run the program by entering 

$ npm install
$ npm run dev

The HTML file that is served is at the project root directory, index.html. The entrance JavaScript file, main.js, is also at the project root directory.  The PNG image file and GeoJSON files are in the public folder. 

Open main.js, and note the “info” and “imagemap” divs.

import createImageMap from './src/imagemap/imagemap'
import { createInfo } from './src/info/info'
...
  <div id="info-map">
    <div id="info">      </div>
    <div id="imagemap">  </div>
  </div>
...
createInfo(document.querySelector('#info'));
createImageMap(document.querySelector('#imagemap'));

The imagemap div is for mounting the Leaflet map component, and the info div is for mounting the panel that will display information about what was clicked. 

The rest of the JavaScript and CSS files are in the src folder. I like to organize the source by features. There are basically three major features:

  • imagemap/ – The Leaflet map using an image
  • geojson/ – Interacting with the image using GeoJSON
  • info/ – The panel displaying information about the clicked area

The rest of my description of the code assumes that you have read the Leaflet tutorials

imagemap

Look at the src/imagemap/imagemap.js file. The code should look familiar. It is basically the code from the “Non-geographical maps” tutorial applied to the Grant_1962_663.png image. 

Look at the  src/imagemap/imagemap.css file. The styling just fixes the dimensions of the leaflet-container div (that Leaflet will add) and puts a border around it. 

geojson

Look at the src/geojson/geojson.js file. This code is not trivial. Most of it comes from the “Interactive Choropleth Map” tutorial. I will go through the differences. 

First the code is responsible for fetching all the GeoJSON file and combining them into a single GeoJSON. 

 * Combine multiple geojson files into one.
 * fetches the geojson files and combines them.
 * @param {Array} geojsons
 * @returns {Object} combinedJson
 */
async function combineGeoJsons(geojsons) {
    let combinedJson = {};
    // ToDO: should parallelize this. Use Promise.all. Maybe leave to the team.
    // A rule "Don't optimize until it is obviously slow".
    for (let i = 0; i < geojsons.length; i++) {
        // Fetch and read the response
        const geojson = geojsons[i];
        const response = await fetch(path + geojson + '.geojson');
        const data = await response.json();
        if (i === 0) { // clone the first geojson file
            // This may take a long time. Consider using:
            // https://www.npmjs.com/package/fast-json-stringify
            // https://www.npmjs.com/package/fast-json-parse
            // or some other efficient cloning library.
            combinedJson = JSON.parse(JSON.stringify(data));
            // console.log('in i===0 combinedJson', combinedJson);
        } else {  // for the rest copy the features
            // need to test
   
            combinedJson.features.push(data.features[0]);
            // console.log('in i > 0 combinedJson', combinedJson);
        }
    }
    return combinedJson;
}

Note that the GeoJSON files are fetched, so the code must use asynchronous functions and the “await” command. The code uses a for-loop to fetch the GeoJSON files sequentially. This is not the most efficient method, but it works fine for fetching just two files. As you add more GeoJSON files, you may need to parallelize the fetch using Promise.all

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

You may want to read the MDN guide about asynchronous JavaScript.

https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous

I wanted the info panel to reset to the default value when the user clicks on an area of the image that is not in a  feature of a GeoJSON. So I had to add an eventhandler on the map. 

export default async function loadGeoJsons(map) {
    // This function takes time. So need to use await.
    const combinedJson = await combineGeoJsons(geojsons);
    geoJsonLayer =  loadGeoJson(map, combinedJson);


    // Add to detect when the map is clicked.
    map.on('click', function(e) {
        updateInfo();
    });
}

But then the propagation of the event must be stopped from reaching the map when the user clicks a feature. So I had to add a DomEvent.stopPropagation to the clickedFeature function.

function clickedFeature(e) {
    const properties = e.target.feature.properties;
    const title = properties.name;
    const description = properties.description;
    const content = {title, description};
    L.DomEvent.stopPropagation(e);  // Prevent the map from getting the event.
    updateInfo(content);
}

Info components will need to know when click occurs, so the geojson.js code needs to import the updateInfo function from info.js, and call it the clickedFunction function and the map’s eventhandler. 

I also notice when using Chrome and Edge that when I click a feature a bounding box around the feature appears. This did not occur for Firefox. I did not like the bounding box, so I added a rule to geojson.css.

/*
 * This rule removes the outline from the GeoJSON path elements
 * when they are focused. Works in chrome, firefox, and edge.
 * See https://stackoverflow.com/questions/710746/how-to-remove-border-outline-around-text-input-boxes-chrome
*/
path.leaflet-interactive:focus {
    outline: none;
}

info

Look at the src/info/info.js file. The code is fairly simple. The fillContent function is responsible for creating the HTML structure. The udateInfo function uses a conditional ternary operator to switch between the default content and the feature content.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Conditional_operator

The styling in src/info/info.css does not contain much. It styles the info panel similar to the imagemap and fixes the width.  

That is all the code. The code can be improved. Besides parallelizing the fetching of the GeoJSON files, I would like to make the geojson.js file smaller by modularizing it. But the geojson component will always be complex because it handles all the events on the map. 

The styling can definitely be improved.

Summary

The implementation workflow is:

  1. Make the SVG file using Inkscape
  2. Convert the SVG file to GeoJSON by editing and using isvg2geojson
  3. Add the GeoJSON file to image-interaction/public/ after editing
  4. Improve the styling and code in image-interaction 

This demo only implements the basic user interactions. Study the InnerBody website.

https://www.innerbody.com/htm/body.html

You’ll notice that it uses map icons and controls what features are shown and interactive by the zoom level. 

I have also considered interactions when dragging or distorting features, but these interactions  might be beyond the capabilities of Leaflet. 

Have fun coding. 

Afterword

I think that it is possible to implement the basic image interaction using directly HTML Canvas and SVG elements and avoiding implementing the algorithm for a point inside a polygon. The implementation would probably require adding event handlers on the appropriate elements and using CSS pointer-events property.

I wrote a simple web page to demonstrate or test the idea.

<!DOCTYPE html>
<html>
<head>
    <title>circle</title>
</head>
<body>
    <div>
        <svg
            version="1.1"
            id="svg1"
            xmlns="http://www.w3.org/2000/svg"
            xmlns:svg="http://www.w3.org/2000/svg">
                <g id="layer1" >
                    <circle
                        id="path1"
                        cx="50"
                        cy="50"
                        r="40"
                        pointer-events="fill"
                        fill="#ff0000"
                    />
                </g>
        </svg>
    </div>

    <script>
        var obj = document.getElementById('path1'); 
        var bbox = obj.getBBox();
        console.log('bbox', bbox);

        obj.addEventListener('click', event => {
            alert (event.target.tagName);
        });
    </script>
</body>
</html>

It works! Notice the pointer-event: fill property in the circle element.

References

Wikipedia 

Dermatome entry: https://en.wikipedia.org/wiki/Dermatome_%28anatomy%29

Dermatome Image:

World Geodetic System (WGS): https://en.wikipedia.org/wiki/World_Geodetic_System 

JSON

MDN

Canvas: https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API

SVG:

HTML Map: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/map

JavaScript:

Software

Inkscape:

FabricJS: 

LeafLet:

Inner Body:

svg2geojson:

isvg2geojson: https://github.com/2024-UI-SP/isvg2geojson

image-interaction: https://github.com/2024-UI-SP/image-interaction 

ViteJS:

RollupJS:

Algorithms