I really like uwsgi, and would like to see it on my raspberry PI, so I decided to create a buildroot environment, and add uwsgi as a package. It means cross compiling uwsgi. In this blog, I will log the process.
One thing to note, that the text was shortened with:
s,/data/matelakat/repa/.workspace/buildroot-2013.05,$BUILDROOT,g
Let's check out a tagged version
git checkout tags/1.9.9 -B build-fixes
And make sure, it builds on the host system:
make
...
############## end of uWSGI configuration #############
*** uWSGI is ready, launch it with ./uwsgi ***
Cross compilation
First, I set the compiler and the precompiler, as suggested by
Roberto:
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
I got this error (newlines inserted for readability)
*** uWSGI linking ***
$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc -o uwsgi ...
...
plugins/transformation_chunked/chunked.o -lpthread -lm -rdynamic -ldl -lz
-L/usr/lib/x86_64-linux-gnu -lpcre -luuid -lssl -lcrypto
-L/usr/lib/x86_64-linux-gnu -lxml2 -lpthread -ldl -lutil
-lm /usr/lib/python2.7/config/libpython2.7.so -lutil -lcrypt
/data/matelakat/repa/.crosstool/x-tools/arm-unknown-linux-gnueabi/lib/gcc/
arm-unknown-linux-gnueabi/4.8.2/../../../../arm-unknown-linux-gnueabi/bin/
ld: skipping incompatible /usr/lib/x86_64-linux-gnu/libpthread.so when searching for -lpthread
/usr/lib/x86_64-linux-gnu/libpthread.a: could not read symbols: File format not recognized
collect2: error: ld returned 1 exit status
*** error linking uWSGI ***
make: *** [all] Error 1
That seems to be an issue, my host system's libpthread will definitely fail to link with the arm binary.
I added a print statement, and an assert False.
*** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz', '-L/usr/lib/x86_64-linux-gnu -lpcre',
'-luuid', '-lssl', '-lcrypto', '-L/usr/lib/x86_64-linux-gnu -lxml2', '-lpthread',
'-ldl', '-lutil', '-lm', '/usr/lib/python2.7/config/libpython2.7.so', '-lutil', '-lcrypt']
Traceback (most recent call last):
File "uwsgiconfig.py", line 1220, in
build_uwsgi(uConf(bconf))
File "uwsgiconfig.py", line 401, in build_uwsgi
assert False
AssertionError
make: *** [all] Error 1
It seems to be using my host's libpython2.7.so. But that's not the point (now). The real point is the '-L/usr/lib/x86_64-linux-gnu -lxml2' part. I need to find out from where does it come. I need to add some additional breakpoints.
As my computer has 2 CPU cores, uwsgi build employs a compile queue. For now, as I would like to debug things, I want to run the build sequentially, so that it is easier to follow what's happening. So I cleaned the build, and set the CPUCOUNT environment variable:
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
I created a shell script with these contents.
The libs variable is used to collect the libraries during plugin compilation.
Let's add a print statement to see when we modify this array
./doit.sh | grep ADDING
...
[ADDING LIBRARY] ['-lpthread', '-ldl', '-lutil', '-lm',
'/usr/lib/python2.7/config/libpython2.7.so', '-lutil']
...
Oh, that doesn't look good, I bet, that's the python plugin.
Let's print out the plugin's name as well
./doit.sh | grep ADDING
...
[ADDING LIBRARY - python] ['-lpthread', '-ldl', '-lutil',
'-lm', '/usr/lib/python2.7/config/libpython2.7.so', '-lutil']
...
Okay, shelve it for now, and go back to the original problem, see from where does the '-L/usr/lib/x86_64-linux-gnu -lxml2' line come from. That might be lxml.
It seems, that xml2-config --libs is called to detect the library path. The host's binary was called this time, instead of the cross tool's. So let's add the crosstool bin path to the path:
#!/bin/bash
set -eux
PATH="$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" make
But as I modified my script, I ended up with this error:
python uwsgiconfig.py --build
$BUILDROOT/output/
host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/
python: 1: $BUILDROOT/
output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python:
Syntax error: word unexpected (expecting ")")
make: *** [all] Error 2
It's not an x86 executable:
file $BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python
$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python: symbolic link to `python2'
file $BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python2.7
$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin/python2.7:
ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs),
for GNU/Linux 3.2.48, not stripped
that's why I have issues with that python. So I modify my shell script to start a host python built by buildroot:
#!/bin/bash
set -eux
PATH="$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" \
$BUILDROOT/output/build/host-python-2.7.3/python uwsgiconfig.py --build
Note, that I also left out the make, and am directly calling uwsgiconfig.py. This is the next error that I get:
*** uWSGI linking ***
...
$BUILDROOT/output/host/usr/lib/libz.so: file not recognized: File format not recognized
collect2: error: ld returned 1 exit status
*** error linking uWSGI ***
file $BUILDROOT/output/host/usr/lib/libz.so.1.2.7
$BUILDROOT/output/host/usr/lib/libz.so.1.2.7:
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=0x5bbf6ecd4880bbbbb73c57e048dba2d36aa299e0, not stripped
well, that's interesting. That file is in the host directory, that's a host module of course. Maybe the library path is wrong?
*** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz',
'-L$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/lib -lpcre',
'-luuid', '-lssl', '-lcrypto', '-lxml2 -lz -lm -ldl', '-lpthread', '-ldl',
'-lutil', '-lm', '-lpython2.7', '-lcrypt']
There is the pcre library, which looks suspicious. So I added another entry to the path, which will find the target system's pcre config script.
#!/bin/bash
set -eux
PATH="$BUILDROOT/output/build/pcre-8.32:$BUILDROOT/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/usr/bin:${PATH}" \
CC="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc" \
CPUCOUNT=1 \
CPP="$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-cpp" \
$BUILDROOT/output/build/host-python-2.7.3/python uwsgiconfig.py --build
The problem is still there. Let's look at the build output again:
** uWSGI linking ***
['-lpthread', '-lm', '-rdynamic', '-ldl', '-lz', '-lpcre', '-luuid',
'-lssl', '-lcrypto', '-lxml2 -lz -lm -ldl', '-lpthread', '-ldl',
'-lutil', '-lm', '-lpython2.7', '-lcrypt']
...
$BUILDROOT/output/host/usr/bin/arm-unknown-linux-gnueabi-gcc -o uwsgi -L$BUILDROOT/output/host/usr/lib
...
I need to find out from where does that come. As a first guess,
I am printing out ldflags.
LDFLAGS ['-L$BUILDROOT/output/host/usr/lib']
That could be an issue.
Let's try to find out which plugin is doing this.
./doit.sh | grep LDFLAG
...
[ADDING LDFLAG - python] ['-L$BUILDROOT/output/host/usr/lib']
...
As I looked at plugins/python/uwsgiplugin.py, found the include issue, and the libs.
So I decided to do a really dirty hack to override libs (It's getting late here)
That left me with another issue:
plugins/python/python_plugin.o: In function `init_uwsgi_embedded_module':
python_plugin.c:(.text+0x1d6c): undefined reference to `Py_InitModule4_64'
collect2: error: ld returned 1 exit status
Let's hack the include directory as well, maybe that is causing the issue.
as I was there,
I also hacked the library paths.
And this is my reward:
make clean
./doit.sh
...
############## end of uWSGI configuration #############
*** uWSGI is ready, launch it with ./uwsgi ***
