Pages

Thursday 8 August 2013

Cross compiling uwsgi with buildroot

I really like uwsgi, and would like to see it on my raspberry PI, so I decided to create a buildroot environment, and add uwsgi as a package. It means cross compiling uwsgi. In this blog, I will log the process.
One thing to note, that the text was shortened with: 
s,/data/matelakat/repa/.workspace/buildroot-2013.05,$BUILDROOT,g
Let's check out a tagged version 
git checkout tags/1.9.9 -B build-fixes
And make sure, it builds on the host system: 
make
...
############## end of uWSGI configuration #############
*** uWSGI is ready, launch it with ./uwsgi ***

Cross compilation

First, I set the compiler and the precompiler, as suggested by Roberto: 
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
I got this error (newlines inserted for readability) 
*** uWSGI linking ***
$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc -o uwsgi ...
...
plugins/transformation_chunked/chunked.o -lpthread -lm -rdynamic -ldl -lz 
-L/usr/lib/x86_64-linux-gnu -lpcre -luuid -lssl -lcrypto 
-L/usr/lib/x86_64-linux-gnu -lxml2 -lpthread -ldl -lutil 
-lm /usr/lib/python2.7/config/libpython2.7.so -lutil -lcrypt
/data/matelakat/repa/.crosstool/x-tools/arm-unknown-linux-gnueabi/lib/gcc/
arm-unknown-linux-gnueabi/4.8.2/../../../../arm-unknown-linux-gnueabi/bin/
ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libpthread.so when searching for -lpthread
/usr/lib/x86_64-linux-gnu/libpthread.a: could not read symbols: File format not recognized
collect2: error: ld returned 1 exit status
*** error linking uWSGI ***
make: *** [all] Error 1
That seems to be an issue, my host system's libpthread will definitely fail to link with the arm binary. I added a print statement, and an assert False. 
*** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz', '-L/usr/lib/x86_64-linux-gnu -lpcre',
 '-luuid', '-lssl', '-lcrypto', '-L/usr/lib/x86_64-linux-gnu -lxml2', '-lpthread',
 '-ldl', '-lutil', '-lm', '/usr/lib/python2.7/config/libpython2.7.so', '-lutil', '-lcrypt']
Traceback (most recent call last):
  File "uwsgiconfig.py", line 1220, in 
    build_uwsgi(uConf(bconf))
  File "uwsgiconfig.py", line 401, in build_uwsgi
    assert False
AssertionError
make: *** [all] Error 1
It seems to be using my host's libpython2.7.so. But that's not the point (now). The real point is the '-L/usr/lib/x86_64-linux-gnu -lxml2' part. I need to find out from where does it come. I need to add some additional breakpoints. As my computer has 2 CPU cores, uwsgi build employs a compile queue. For now, as I would like to debug things, I want to run the build sequentially, so that it is easier to follow what's happening. So I cleaned the build, and set the CPUCOUNT environment variable: 
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
I created a shell script with these contents. The libs variable is used to collect the libraries during plugin compilation. Let's add a print statement to see when we modify this array 
./doit.sh  | grep ADDING
...
[ADDING LIBRARY] ['-lpthread', '-ldl', '-lutil', '-lm',
 '/usr/lib/python2.7/config/libpython2.7.so', '-lutil']
...
Oh, that doesn't look good, I bet, that's the python plugin. Let's print out the plugin's name as well 
./doit.sh  | grep ADDING
...
[ADDING LIBRARY - python] ['-lpthread', '-ldl', '-lutil',
 '-lm', '/usr/lib/python2.7/config/libpython2.7.so', '-lutil']
...
Okay, shelve it for now, and go back to the original problem, see from where does the '-L/usr/lib/x86_64-linux-gnu -lxml2' line come from. That might be lxml. It seems, that xml2-config --libs is called to detect the library path. The host's binary was called this time, instead of the cross tool's. So let's add the crosstool bin path to the path: 
#!/bin/bash

set -eux
PATH="$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
But as I modified my script, I ended up with this error: 
python uwsgiconfig.py --build
$BUILDROOT/output/
host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/
python: 1: $BUILDROOT/
output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python: 
Syntax error: word unexpected (expecting ")")
make: *** [all] Error 2
It's not an x86 executable: 
file $BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python
$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python: symbolic link to `python2'
file $BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python2.7
$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python2.7:
 ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs),
 for GNU/Linux 3.2.48, not stripped
that's why I have issues with that python. So I modify my shell script to start a host python built by buildroot: 
#!/bin/bash

set -eux
PATH="$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" \
$BUILDROOT/output/build/host-python-2.7.3/python uwsgiconfig.py --build
Note, that I also left out the make, and am directly calling uwsgiconfig.py. This is the next error that I get: 
*** uWSGI linking ***
...
$BUILDROOT/output/host/usr/lib/libz.so: file not recognized: File format not recognized
collect2: error: ld returned 1 exit status
*** error linking uWSGI ***

file $BUILDROOT/output/host/usr/lib/libz.so.1.2.7 
$BUILDROOT/output/host/usr/lib/libz.so.1.2.7:
 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
 BuildID[sha1]=0x5bbf6ecd4880bbbbb73c57e048dba2d36aa299e0, not stripped
well, that's interesting. That file is in the host directory, that's a host module of course. Maybe the library path is wrong? 
*** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz',
 '-L$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/lib -lpcre',
 '-luuid', '-lssl', '-lcrypto', '-lxml2 -lz -lm -ldl', '-lpthread', '-ldl',
 '-lutil', '-lm', '-lpython2.7', '-lcrypt']
There is the pcre library, which looks suspicious. So I added another entry to the path, which will find the target system's pcre config script. 
#!/bin/bash

set -eux
PATH="$BUILDROOT/output/build/pcre-8.32:$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" \
$BUILDROOT/output/build/host-python-2.7.3/python uwsgiconfig.py --build
The problem is still there. Let's look at the build output again: 
** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz', '-lpcre', '-luuid',
 '-lssl', '-lcrypto', '-lxml2 -lz -lm -ldl', '-lpthread', '-ldl',
 '-lutil', '-lm', '-lpython2.7', '-lcrypt']
...
$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc -o uwsgi -L$BUILDROOT/output/host/usr/lib
...
I need to find out from where does that come. As a first guess, I am printing out ldflags. 
LDFLAGS ['-L$BUILDROOT/output/host/usr/lib']
That could be an issue. Let's try to find out which plugin is doing this. 
./doit.sh | grep LDFLAG
...
[ADDING LDFLAG - python] ['-L$BUILDROOT/output/host/usr/lib']
...
As I looked at plugins/python/uwsgiplugin.py, found the include issue, and the libs. So I decided to do a really dirty hack to override libs (It's getting late here) That left me with another issue: 
plugins/python/python_plugin.o: In function `init_uwsgi_embedded_module':
python_plugin.c:(.text+0x1d6c): undefined reference to `Py_InitModule4_64'
collect2: error: ld returned 1 exit status
Let's hack the include directory as well, maybe that is causing the issue. as I was there, I also hacked the library paths. And this is my reward: 
make clean
./doit.sh
...
############## end of uWSGI configuration #############
*** uWSGI is ready, launch it with ./uwsgi ***
Posted by at

4 comments:

  1. Unknown30 July 2015 at 14:26

    Hi Mate,

    I'm going down the same path trying to cross-compile uwsgi and am running into a similar error "undefined reference to `Py_GetVersion'"

    Unfortunately, your links to commits in github no longer work to reveal the dirty hacks. Can you remember what you did or have any advice on less dirty hacks to get this working? I'm super new to buildroot so I'm just shooting in the dark trying to get this working.

    Thanks!
    -brian

    ReplyDelete
    Replies
    1. Mate Lakat9 August 2015 at 05:56

      Hi Brian,

      To be honest with you this exercise was done quite a long time ago, and in the end I did not use it. UWSGI comes with its own build system, and as you can imagine, when people think they can write a python script to replace make, the project fails badly. This is what happened in the uwsgi case. I looked at the build script, and that's when I gave up on it. It did not worth the investigation. The official answer was: compile it on the target system. It's rather sad that such a useful piece of software has such a rigid build system. I recommend you try to compile it on the target system, as they suggest.

      Delete
    2. Mate Lakat9 August 2015 at 08:39

      Hi Brian,

      I restored the github commits, so now you should be able to access them should you need it.

      Cheers,
      Mate

      Delete
  2. Unknown12 August 2015 at 13:07

    Thanks Mate! In the meantime I have gotten it working with two small patches to the config files, to pass in library paths to the right places, then adding two environment variables to UWSGI_BUILD_CMDS
    UWSGICONFIG_INCLUDEPATH=$(STAGING_DIR)/usr/include/python2.7 \
    UWSGI_PYTHON_CROSSLIB="$(STAGING_DIR)/usr/lib" \

    -brian


    Patch uwsgiconfig.py to read the UWSGICONFIG_INCLUDEPATH environment variable in order to avoid problems.

    Index: b/uwsgiconfig.py
    ===================================================================
    --- a/uwsgiconfig.py
    +++ b/uwsgiconfig.py
    @@ -425,7 +425,11 @@

    p_cflags = cflags[:]
    try:
    - p_cflags += up['CFLAGS']
    + buildroot_include_path = os.environ.get('UWSGICONFIG_INCLUDEPATH', '')
    + if buildroot_include_path:
    + p_cflags.append('-I' + buildroot_include_path)
    + else:
    + p_cflags += up['CFLAGS']
    except:
    pass




    diff --git a/plugins/python/uwsgiplugin.py b/plugins/python/uwsgiplugin.py
    index f9b4ddb..05e3c52 100644
    --- a/plugins/python/uwsgiplugin.py
    +++ b/plugins/python/uwsgiplugin.py
    @@ -16,7 +16,11 @@
    CFLAGS = ['-I' + sysconfig.get_python_inc(), '-I' + sysconfig.get_python_inc(plat_specific=True) ]
    LDFLAGS = []

    -if not 'UWSGI_PYTHON_NOLIB' in os.environ:
    +if 'UWSGI_PYTHON_CROSSLIB' in os.environ:
    + LDFLAGS.append("-L%s" % os.environ['UWSGI_PYTHON_CROSSLIB'])
    + LIBS = ['-lpython%s' % get_python_version()]
    +
    +elif not 'UWSGI_PYTHON_NOLIB' in os.environ:
    LIBS = sysconfig.get_config_var('LIBS').split() + sysconfig.get_config_var('SYSLIBS').split()
    # check if it is a non-shared build (but please, add --enable-shared to your python's ./configure script)
    if not sysconfig.get_config_var('Py_ENABLE_SHARED'):

    ReplyDelete

Subscribe to: Post Comments (Atom)