Fun with OpenSSL and Erlang

March 28, 2022 7-minute read

openbsd • systems • tls • ssl • erlang • elixir • dev

Since OpenBSD 7.1 (and -current at the time of writing this post), LibreSSL includes some breaking changes made in order to replicate the struct visibility changes present in OpenSSL since version 1.1. Suddenly, my dev setup is broken. Oh my!

The problem with having these two crypto libraries breaking APIs at their own pace is that, sometimes, other libraries depending on them struggle to keep up. It is the case of the Erlang library fast_tls, which still assumes every LibreSSL implementation behaves like OpenSSL before 1.1, and thus fails to compile. While the patch is trivial to contribute, filling their paperwork is not, so while we wait for upstream to fix this issue themselves… LET’S PLAY.

UPDATE Apr 2022: fast_tls fixed this issue in version 1.1.14.

Installing OpenSSL in OpenBSD

OpenBSD provides packages for all major OpenSSL versions: 1.0, 1.1 and 3.0. We are going to install 1.1, since it is the option in the middle, and middle options are good(TM)¹.

1
2
3
4
5
6
7
8
9


ashen:~# pkg_add openssl
quirks-5.4 signed on 2022-03-26T23:20:55Z
Ambiguous: choose package for openssl
a       0: <None>
        1: openssl-1.0.2up4
        2: openssl-1.1.1np0
        3: openssl-3.0.2p0
Your choice: 2
openssl-1.1.1np0: ok

Awesome. Now, what exactly is inside that package?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ashen:~# pkg_info -L openssl
Information for inst:openssl-1.1.1np0

Files:
/usr/local/bin/c_rehash11
/usr/local/bin/eopenssl11
/usr/local/include/eopenssl11/openssl/aes.h
/usr/local/include/eopenssl11/openssl/asn1.h
/usr/local/include/eopenssl11/openssl/asn1_mac.h
/usr/local/include/eopenssl11/openssl/asn1err.h
/usr/local/include/eopenssl11/openssl/asn1t.h
( ... around 1 quadrillon lines more ... )

“I should have | lessed that”

Anyway, for development purposes we are only interested in headers and shared libraries:

1
2
3


ashen:~# pkg_info -L openssl | egrep 'opensslv.h|libssl.so'
/usr/local/include/eopenssl11/openssl/opensslv.h
/usr/local/lib/eopenssl11/libssl.so.11.6

Okay, so the OpenBSD devs are using the standard directories for third-party headers/libs (/usr/local/include + /usr/local/lib), but additionally “confining” everything inside an eopenssl11 directory in order to allow several OpenSSL versions installed at the same time. Sweet.

Now, Erlang.

Erlang + custom OpenSSL

Note: Erlang 24.3.2 seems to compile out of the box against LibreSSL with no problem. This section is just for reference.

We manage our local Elixir/Erlang installs with asdf. Elixir is just a wrapper, but Erlang has to be compiled, and that’s where we can tell it to use our own OpenSSL install. The asdf-erlang plugin uses kerl under the hood, so we can pass it the KERL_CONFIGURE_OPTIONS environment variable to reach Erlang’s configure stage. Unfortunately, the option Erlang provides to customize OpenSSL is rather coarse-grained:

--with-ssl=PATH - Specify base location of OpenSSL include and lib directories.

Hmmm… OK, I see. And what if the include/lib directories are separate, as in OpenBSD? The answer is potato. We are left with no options other than…

1
2
3


ashen:~# mkdir /usr/local/eopenssl11
ashen:~# ln -s /usr/local/include/eopenssl11 /usr/local/eopenssl11/include
ashen:~# ln -s /usr/local/lib/eopenssl11 /usr/local/eopenssl11/lib

}:|

Okay, that should work. Now just compile and install Erlang with a custom --ssl-dir and we’re done:

1
2


~ $ KERL_CONFIGURE_OPTIONS="--with-ssl=/usr/local/eopenssl11" \
    asdf install erlang 24.3.2

Erlang library (fast_tls) + custom OpenSSL

Suppose you have an Elixir project that depends on fast_tls either directly or indirectly. You can cd inside the project and try to compile it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


$ mix deps.compile fast_tls
===> Fetching pc v1.14.0
===> Analyzing applications...
===> Compiling pc
===> Compiling /home/user/sample/deps/fast_tls/c_src/fast_tls.c
===> /home/user/sample/deps/fast_tls/c_src/fast_tls.c:381:9: error: incomplete definition of type 'struct dh_st'
        DH_set0_pqg(dh, dh_p, NULL, dh_g);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/sample/deps/fast_tls/c_src/fast_tls.c:65:48: note: expanded from macro 'DH_set0_pqg'
#define DH_set0_pqg(dh, dh_p, param, dh_g) (dh)->p = dh_p; (dh)->g = dh_g
                                           ~~~~^
/usr/include/openssl/ossl_typ.h:116:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;
               ^
/home/user/sample/deps/fast_tls/c_src/fast_tls.c:381:9: error: incomplete definition of type 'struct dh_st'
        DH_set0_pqg(dh, dh_p, NULL, dh_g);
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/sample/deps/fast_tls/c_src/fast_tls.c:65:64: note: expanded from macro 'DH_set0_pqg'
#define DH_set0_pqg(dh, dh_p, param, dh_g) (dh)->p = dh_p; (dh)->g = dh_g
                                                           ~~~~^
/usr/include/openssl/ossl_typ.h:116:16: note: forward declaration of 'struct dh_st'
typedef struct dh_st DH;
               ^
2 errors generated.

Boom! Trying to access private struct members: not adapted to LibreSSL yet. Let’s try to point it to our custom OpenSSL 1.1 installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


$ CFLAGS="-I/usr/local/include/eopenssl11" \
  LDFLAGS="-L/usr/local/lib/eopenssl11" \
  mix deps.compile fast_tls

===> Fetching pc v1.14.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_tls.c
===> Compiling c_src/ioqueue.c
===> Compiling c_src/p1_sha.c
===> Linking /home/user/sample/deps/fast_tls/priv/lib/fast_tls.so
===> Linking /home/user/sample/deps/fast_tls/priv/lib/p1_sha.so
===> Analyzing applications...
===> Compiling fast_tls

Looks good! Except, it is not. As soon as you start fast_tls…

15:22:00.010 [error] Process #PID<0.13122.0> on node :"sample@127.0.0.1" raised an exception
** (ArgumentError) argument error
    (kernel 8.3.1) :erl_ddll.format_error_int({:load_failed, 'Failed to load NIF library /home/user/sample/_build/dev/lib/fast_tls/priv/lib/fast_tls: \'Cannot load specified object\''})
    (kernel 8.3.1) erl_ddll.erl:239: :erl_ddll.format_error/1
    (fast_tls 1.1.13) /home/user/sample/deps/fast_tls/src/fast_tls.erl:449: :fast_tls.load_nif/1
    (kernel 8.3.1) code_server.erl:1317: anonymous fn/1 in :code_server.handle_on_load/5

The “Failed to load NIF library” points to a problem loading the fast_tls dynamic library. In Erlang, NIF means Native Implemented Function – code written in C. Let’s check it out:

1
2
3
4


$ ldd _build/dev/lib/fast_tls/priv/lib/fast_tls.so
_build/dev/lib/fast_tls/priv/lib/fast_tls.so:
Cannot load specified object
_build/dev/lib/fast_tls/priv/lib/fast_tls.so: exit status 1

Oops. Definitely something wrong, there are some needed libraries that can not be found. Let’s inspect the dynamic section of the library:

1
2
3
4
5
6
7
8


$ readelf -d _build/dev/lib/fast_tls/priv/lib/fast_tls.so
Dynamic section at offset 0x8bd0 contains 22 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.11.6]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.11.6]
 0x0000000000000007 (RELA)               0x1988
 0x0000000000000008 (RELASZ)             1680 (bytes)
 ...

(By the way, for a great introduction to dynamic libraries check out this awesome blog post by Amir Rachum.)

So, either libssl.so.11.6 and libcrypto.so.11.6 cannot be found (it’s both, of course). That’s because, remember, our OpenSSL library is not in a standard location – “standard” here meaning “somewhere where our dynamic linker can find it at runtime”. Depending on how h4x0r we feel tonight we can do several things, so let’s try out all of them just for amusement.

The good

There’s this option you can pass to the linker (man 1 ld):

     --rpath=value, -R value
             Add a DT_RUNPATH to the output.

The RUNPATH is a colon-separated list of paths that get engraved on the ELF file and tell the dynamic linker where to look for extra libraries. We can pass that option via LDFLAGS after wrapping it into a -Wl, which is the way to tell the compiler to just pass that option to the linker:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


$ CFLAGS="-I/usr/local/include/eopenssl11" \
  LDFLAGS="-L/usr/local/lib/eopenssl11 -Wl,--rpath=/usr/local/lib/eopenssl11" \
  mix deps.compile fast_tls

===> Fetching pc v1.14.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_tls.c
===> Compiling c_src/ioqueue.c
===> Compiling c_src/p1_sha.c
===> Linking /home/user/sample/deps/fast_tls/priv/lib/fast_tls.so
===> Linking /home/user/sample/deps/fast_tls/priv/lib/p1_sha.so
===> Analyzing applications...
===> Compiling fast_tls

$ ldd _build/dev/lib/fast_tls/priv/lib/fast_tls.so
_build/dev/lib/fast_tls/priv/lib/fast_tls.so:
	Start            End              Type  Open Ref GrpRef Name
	000006295c841000 000006295c84e000 dlib  1    0   0      /home/user/sample/deps/fast_tls/priv/lib/fast_tls.so
	00000628ec7bf000 00000628ec86b000 rlib  0    1   0      /usr/local/lib/eopenssl11/libssl.so.11.6
	000006298d52d000 000006298d82d000 rlib  0    2   0      /usr/local/lib/eopenssl11/libcrypto.so.11.6

$ readelf -d _build/dev/lib/fast_tls/priv/lib/fast_tls.so
Dynamic section at offset 0x8bf0 contains 23 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/local/lib/eopenssl11]
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.11.6]
 0x0000000000000001 (NEEDED)             Shared library: [libcrypto.so.11.6]
 0x0000000000000007 (RELA)               0x19a8
 0x0000000000000008 (RELASZ)             1680 (bytes)
 ...

Wonderful.

The bad

Alternatively, if you’re too lazy to pass --rpath or you’re dealing with a really common library (OpenSSL arguably qualifies for this), you can add custom paths to your dynamic linker configuration. The OpenBSD one, ld.so(1), can be configured via ldconfig(1).

1
2
3


ashen:~# ldconfig -r | egrep 'libcrypto.so|libssl.so'
        143:-lcrypto.49.0 => /usr/lib/libcrypto.so.49.0
        280:-lssl.52.0 => /usr/lib/libssl.so.52.0

Those are the standard ones from LibreSSL.

Let’s add OpenSSL 1.1:

1
2
3
4
5
6
7


ashen:~# ldconfig -m /usr/local/lib/eopenssl11

ashen:~# ldconfig -r | egrep 'libcrypto.so|libssl.so'
        115:-lcrypto.49.0 => /usr/lib/libcrypto.so.49.0
        128:-lssl.52.0 => /usr/lib/libssl.so.52.0
        624:-lcrypto.11.6 => /usr/local/lib/eopenssl11/libcrypto.so.11.6
        625:-lssl.11.6 => /usr/local/lib/eopenssl11/libssl.so.11.6

Now, our custom libdir /usr/local/lib/eopenssl11 will be searched by ld.so(1) when some executable or dynamic library tries to load OpenSSL.

You will probably want to persist this change by editing some configuration file depending on your OS. In OpenBSD you can use the shlib_dirs parameter in rc.conf.local(8).

The ugly

Finally, the YOLO solution: modify the RUNPATH of the compiled fast_tls library… MANUALLY! BWAHAHAHAH

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


$ patchelf --set-rpath /usr/local/lib/eopenssl11 _build/dev/lib/fast_tls/priv/lib/fast_tls.so

$ ldd _build/dev/lib/fast_tls/priv/lib/fast_tls.so
_build/dev/lib/fast_tls/priv/lib/fast_tls.so:
	Start            End              Type  Open Ref GrpRef Name
	00000598720e2000 00000598720f1000 dlib  1    0   0      /home/user/sample/deps/fast_tls/priv/lib/fast_tls.so
	000005986fbdd000 000005986fc89000 rlib  0    1   0      /usr/local/lib/eopenssl11/libssl.so.11.6
	00000598f38c8000 00000598f3bc8000 rlib  0    2   0      /usr/local/lib/eopenssl11/libcrypto.so.11.6

$ readelf -d _build/dev/lib/fast_tls/priv/lib/fast_tls.so | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/local/lib/eopenssl11]

“With great power, don’t do it.” - Batman

Until OpenSSL 3.0 is mainstream… let them suffer the bugs first }:) ↩︎