I’ve spent this bank holiday weekend building an improved tool for generating my Hugo short codes for responsive images on my blog. In the process I ran into a situation where I needed to call a Go library from Rust. In doing so I learned a fair bit about FFI and as there was no single resource that documented the process I thought I’d break down how to do it and highlight some of the intricacies.

Background

Responsive images are hard to get right, and at the start of the year I wrote a Node.js tool to help automate some of the steps in the process I used to make sure the ones on my blog are of a high quality ¹. Node.js made sense at the time as I wanted to create SVG placeholders and the original SQIP library was written in Node. Additionally it’s a great language for prototyping in, and it comes with near effortless async! Unfortunately, it’s brittle and, in my opinion pretty poorly suited for command line tools, so after a couple of months of frustration with it, I decided to rewrite it in Rust.

Unfortunately the SQIP library is pretty essential to the tool and there are no Rust ports I could have leveraged. I considered rewriting the whole library, but that would’ve added days to the project, and I’d likely need to rewrite a few transitive dependencies too. In short - probably too much to try at the time. It might have been possible to interface with the Node SQIP library but because JavaScript is an interpreted language that might involve embedding a V8 runtime in our program - not ideal and definitely not straightforward! Luckily, there was a Go port of the SQIP library and that seemed like it might be easier to interface with.

FFI

The common way programs of different languages can communicate is through what’s known as a Foreign Function Interface or FFI. By providing a C-compatible interface as a common denominator, languages are able to interop with each other by using C-like primitives as return values and function arguments. Whilst few languages (C / C++) use those primitives natively most languages have ways to coerce their own primitives to be represented in a C-like fashion and vice versa.

The coercion from language representation to C representation typically involves putting a little extra work in, but in our case Go provides tooling to do most of the heavy lifting.

Making Go Code Callable via a FFI

In order to provide an FFI for a Go library we need to do five things².

• Have all of our code in a main package
• Have an empty main function
• Mark the functions with an //export FunctionNameHere comment
• Import the pseudo-package C
• Use a special flag during the build to export either a shared library or a static library -buildmode=c-shared produces a .so file and -buildmode=c-archive produces a .a file

Unfortunately, none of this had been done for us by the Go SQIP port, so I created a wrapper library - a library that serves no purpose except to produce an FFI interface for an existing library.

Let’s see what that looks like with our four steps applied.

package main

import "C"
import (
 "github.com/denisbrodbeck/sqip"
)

//export MakeSVG
func MakeSVG(path string) *C.char {
    workSize := 256  // large images get resized to this - higher size grants no boons
    count := 8       // number of primitive SVG shapes
    mode := 1        // shape type
    alpha := 128     // alpha value
    repeat := 0      // add N extra shapes each iteration with reduced search (mostly good for beziers)
    workers := 1     // number of parallel workers
    background := "" // background color (hex)

    svg, _, _, err := sqip.Run(path, workSize, count, mode, alpha, repeat, workers, background)
    if err != nil {
    result := "Error " + err.Error()
    return C.CString(result)
    }
    return C.CString(svg)
}

func main() {}

Note that in order to save us a headache later we return a C-like string from the function, but that we still use a regular Go string as the argument to the function. It might be possible to accept *C.char as a function argument - I didn’t try. What’s important is to know what types we’re dealing with when we come to calling our library from Rust.

Rust

Rust has some kick-ass support for FFI but more often than not, as a low level systems language, Rust is the one being called via the FFI rather than the other way around. This made the documentation a little spotty in places!

Linking Against CGO

In order to link with our CGO binary we need to be aware of whether it’s a static or a dynamic binary. Dynamic binaries are expected to already exist on each user’s machine whereas a static binary is added to the binary the Rust compiler produces and therefore increases the size of the binary.

go build -buildmode=c-archive -o libsqip.a main.go produces a static binary, whereas go build -o libsqip.so -buildmode=c-shared main.go produces a dynamic one. I chose to build a static binary so our Rust program is one single fat binary which makes distribution easier. Whichever you choose note that Cargo expects your library’s filename to start with lib regardless of the type.

In order to give a heads-up to Cargo about needing a non-Rust dependency we have to add a build.rs file to the root of the project. This will be invoked by Cargo during build. By printing lines using build.rs additional information is passed to the compiler. The two additional flags we care about are cargo:rustc-link-search=native=./build which is the path Cargo should look for the binary to link against, in our case ./build - and cargo:rustc-link-lib=static={} which tells Cargo the type of binary to link against, in our case static, and the name of the library - in our case sqip. Notice that the name here is without the lib prefix.

Here is the build.rs I used.

fn main() {
    let path = "./build";
    let lib = "sqip";

    println!("cargo:rustc-link-search=native={}", path);
    println!("cargo:rustc-link-lib=static={}", lib);
}

Calling the FFI

In order to invoke the FFI function I created a separate module to contain some of the messiness. Calling out to an FFI requires the use of unsafe, a header block, and some ugly looking structs.

Here’s the full file, and I’ll walk through it bit by bit.

use std::ffi::{CStr, CString};
use std::os::raw::c_char;

use crate::error::AppError;
use base64::encode;

extern "C" {
    fn MakeSVG(path: GoString) -> *const c_char;
}

#[repr(C)]
struct GoString {
    a: *const c_char,
    b: i64,
}

pub fn make_sqip(path: &str) -> Result<String, AppError> {
    let c_path = CString::new(path).expect("CString::new failed");
    let ptr = c_path.as_ptr();
    let go_string = GoString {
        a: ptr,
        b: c_path.as_bytes().len() as i64,
    };
    let result = unsafe { MakeSVG(go_string) };
    let c_str = unsafe { CStr::from_ptr(result) };
    let string = c_str.to_str().expect("Error translating SQIP from library");
    match string.is_empty() || string.starts_with("Error") {
        true => Err(AppError::SQIP {}),
        false => Ok(encode(&string)),
    }
}

C works on the basis of header files, which contain a function signature, and .c files which are the implementation for the function signature. Here we do a similar thing.

extern "C" {
    fn MakeSVG(path: GoString) -> *const c_char;
}

The extern block lists all the functions in the foreign interface along with their type signature. Notice how the function requires a GoString as an argument but returns a C character pointer. That perfectly matches our signature from the Go library: func MakeSVG(path string) *C.char.

A GoString isn’t something that Rust understands, so we have to build a struct to represent it. If we look at the header files go build produces we can discover the definition of a GoString typedef struct { const char *p; ptrdiff_t n; } - in order to interface with this function we need to build a struct that matches that representation of a string at a byte level.

Fortunately, Rust allows us to define structs and then annotate them to ensure they’re represented in the same way a C struct would be.

#[repr(C)]
struct GoString {
    a: *const c_char,
    b: i64,
}

Here we attach a pointer to the first character in the string and provide a representation of its length excluding the null-termination byte that C strings have.

The last piece of magic is to wrap this unsafe interface in a safe signature.

pub fn make_sqip(path: &str) -> Result<String, AppError> {
    let c_path = CString::new(path).expect("CString::new failed");
    let ptr = c_path.as_ptr();
    let go_string = GoString {
        a: ptr,
        b: c_path.as_bytes().len() as i64,
    };
    let result = unsafe { MakeSVG(go_string) };
    let c_str = unsafe { CStr::from_ptr(result) };
    let string = c_str.to_str().expect("Error translating SQIP from library");
    match string.is_empty() || string.starts_with("Error") {
        true => Err(AppError::SQIP {}),
        false => Ok(encode(&string)),
    }
}

Now our Rust code can call this function safely and we limit the dodgy unsafe calls to two lines. It’s worth taking at line 6 above and noting that we do not count the null terminating byte in the length of the string. Doing so will cause segfaults!