ddddfang's Blog.

swig

swig python c
字数统计: 2.3k阅读时长: 14 min
2019/02/19 Share

1. 概述

swig可以将c/c++编写的code生成其他语言(python / js / java / c# / perl / lisp / ….)的接口,本质是一个代码生成工具.

2. 安装

swig install

1
2
3
4
5
6
7
8
9
10
11
12
$ sudo apt-get install automake
$ sudo apt-get install flex bison

$ git clone https://github.com/swig/swig.git
$ cd swig
$ ./autogen.sh
$ ./configure --prefix=/some/directory
$ make
$ sudo make install

export SWIG=/opt/swig
export PATH=$PATH:$SWIG/bin

3. 为c/c++代码添加python接口

添加 add.c :

1
2
3
4
5
6
7
8
//add.c
#include "add.h"
#include <stdio.h>
int add(int a,int b)
{
    printf("hello, in add\r\n");
    return a+b;
}

添加 add.h:

1
2
3
4
5
6
#ifndef ADD_H
#define ADD_H

int add(int,int);

#endif

添加 add.i (for swig):

1
2
3
4
5
6
7
%module add   /*模块名*/

%{
#include "add.h"
%}

int add(int,int);   /*add.h中的内容*/
1
2
3
4
#将生成 add.py & add_wrap.c
swig -python -py3 add.i
#-g -O0 -Wall -shared -fPIC
gcc -shared -fPIC -I/opt/python-3.6.5/include/python3.6m -L/opt/python-3.6.5/lib/ -lpython3.6m -o _add.so add.c add.h add_wrap.c

或者直接一个makefile搞定:

1
2
3
4
5
6
7
8
9
10
PUBLIC_FLAGS:=-g -O0 -Wall -shared -fPIC
LDFLAGS = -L/opt/python-3.6.5/lib/ -lpython3.6m
MODULE_INCLUDE = -I/opt/python-3.6.5/include/python3.6m

_add.so : add.c add.h add_wrap.c
	gcc $(PUBLIC_FLAGS) $(MODULE_INCLUDE) $(LDFLAGS) -o $@ $^
add_wrap.c : add.i
	swig -python -py3 $^
clean :
	-rm -rf _add.so add_wrap.c add.py

最后可以这样使用:

1
2
3
4
5
6
7
#! /opt/python-3.6.5/bin/python3.6

import sys
sys.path.append("./")

from _add import *
print("3+4=" + str(add(3,4)) )

4. 处理函数指针?

问题来源

4.1. 问 题

Here is a simplified example of what I want to do. Suppose I have the following c++ code in test.h

1
2
double f(double x);
double myfun(double (*f)(double x));

It doesn’t really matter for now what these functions do. The important thing is that myfun takes in a function pointer. After including the test.h file in my interface file, I compiled a python module “test” using SWIG. Now, in Python, I run the following commands:

1
2
import test
f = test.f

This creates a properly working function f, which takes in a double. However, when I try to pass “f” into myfun within python this is what happens:

1
2
myfun(f)
TypeError: in method 'myfun', argument 1 of type 'double (*)(double)'

How do I fix this? I figure I need a typemap declaration in my SWIG interface file, but I am not sure what the proper syntax is or where to put it. I tried

1
%typemap double f(double);

but that didn’t work. Any ideas?

4.2. 解决方案

Note: this answer has a long section on workarounds. If you simply want to use this skip straight to solution 5.
You’ve run into the fact that in Python everything is an object. Before we look at fixing things, first let’s understand what’s going on as is. I’ve created a complete example to work with, with a header file:

1
2
3
4
5
6
7
8
9
10
11
12
13
double f(double x) {
  return x*x;
}

double myfun(double (*f)(double x)) {
  fprintf(stdout, "%g\n", f(2.0));
  return -1.0;
}

typedef double (*fptr_t)(double);
fptr_t make_fptr() {
  return f;
}

The main changes I’ve made so far are adding a definition to your declarations so I can test them and a make_fptr() function that returns something to Python we know is going to be wrapped as a function pointer. With this the first SWIG module might look like:

1
2
3
4
5
6
7
%module test

%{
#include "test.h"
%}

%include "test.h"

And we can compile it with:

1
swig2.0 -Wall -python test.i && gcc -Wall -Wextra -I/usr/include/python2.6 -std=gnu99 -shared -o _test.so test_wrap.c

So now we can run this and ask Python about the types we have - the type of test.f and the type of the result of calling test.make_fptr()):

1
2
3
4
5
6
7
8
9
10
11
12
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> type(test.f)
<type 'builtin_function_or_method'>
>>> repr(test.f)
'<built-in function f>'
>>> type(test.make_fptr())
<type 'SwigPyObject'>
>>> repr(test.make_fptr())
"<Swig Object of type 'fptr_t' at 0xf7428530>"

So the problem as it stands should become clear - there’s no conversion from built-in functions to the SWIG type for function pointers, so your call to myfun(test.f) won’t work. The question then is how (and where) do we fix this? In fact there are at least four possible solutions we might pick, depending on how many other languages you target and how “Pythonic” you want to be.

Solution 1:

The first solution is trivial. We already used test.make_fptr() to return us a Python handle to a function pointer for the funciton f. So we can infact call:

1
2
f=test.make_fptr()
test.myfun(f)

Personally I don’t like this solution very much, it’s not what Python programmers expect and it’s not what C programmers expect. The only thing going for it is the simlicity of implementation.

Solution 2:

SWIG gives us a mechanism to expose function pointers to the target language, using %constant. (Normally this is used for exposing compile-time constants, but that’s esentially all function pointers really are in their simplest form anyway).

So we can modify our SWIG interface file:

1
2
3
4
5
6
7
8
9
10
%module test

%{
#include "test.h"
%}

%constant double f(double);
%ignore f;

%include "test.h"

The %constant directive tells SWIG to wrap f as a function pointer, not a function. The %ignore is needed to avoid a warning about seeing multiple versions of the same identifier. (Note: I also removed the typedef and make_fptr() function from the header file at this point) Which now lets us run:

1
2
3
4
5
6
7
8
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> type(test.f)
<type 'SwigPyObject'>
>>> repr(test.f)
"<Swig Object of type 'double (*)(double)' at 0xf7397650>"

Great - it’s got the function pointer. But there’s a snag with this:

1
2
3
4
>>> test.f(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'SwigPyObject' object is not callable

Now we can’t call test.f from the Python side. Which leads to the next solution:

Solution 3:

To fix this let’s first expose test.f as both a function pointer and a built-in function. We can do that by simply using %rename instead of %ignore:

1
2
3
4
5
6
7
8
9
10
%module test

%{
#include "test.h"
%}

%constant double f(double);
%rename(f_call) f;

%include "test.h"
1
2
3
4
5
6
7
8
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> repr(test.f)
"<Swig Object of type 'double (*)(double)' at 0xf73de650>"
>>> repr(test.f_call)
'<built-in function f_call>'

That’s a step, but I still don’t like the idea of having to remember if I should write test.f_call or just test.f depending on the context of what I want to do with f at the time. We can achieve this just by writing some Python code in our SWIG interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
%module test

%{
#include "test.h"
%}

%rename(_f_ptr) f;
%constant double f(double);
%rename(_f_call) f;

%feature("pythonprepend") myfun %{
  args = f.modify(args)
%}

%include "test.h"

%pythoncode %{
class f_wrapper(object):
  def __init__(self, fcall, fptr):
    self.fptr = fptr
    self.fcall = fcall
  def __call__(self,*args):
    return self.fcall(*args)
  def modify(self, t):
    return tuple([x.fptr if isinstance(x,self.__class__) else x for x in t])

f = f_wrapper(_f_call, _f_ptr)
%}

There are several functional bits here. Firstly we create a new, pure Python class to wrap a function as both a callable and a function pointer. It holds as members the real SWIG wrapped (and renamed) function pointer and function. These are now renamed to begin with an underscore as Python convention. Secondly we set test.f to be an instance of this wrapper. When it is called as a function it passes the call through. Finally we insert some extra code into the myfun wrapper to swap in the real function pointer rather than our wrapper, taking care not to alter any other arguments if there were any.

This does work as expected, for example with:

1
2
3
4
5
import test
print "As a callable"
test.f(2.0)
print "As a function pointer"
test.myfun(test.f)

We could make this a bit nicer, for example with a SWIG macro to avoid repeating the %rename, %constant and wrapper instance creation, but we can’t really escape from the need to use %feature("pythonprepend") everywhere we pass these wrappers back to SWIG. (If it is possible to do that transparently it’s well beyond my Python knowledge).

Solution 4:

The previous solution is somewhat neater, it works transparently as you’d expect (as both a C and Python user) and the mechanics of it is encapsulated with nothing but Python implementing it.

There is a gotcha still though, besides the need to use pythonprepend for every single usage of the function pointers - if you run swig -python -builtin it simply won’t work, because there’s no Python code to prepend in the first place! (You’d need to change the construction of the wrapper to be: f = f_wrapper(_test._f_call, _test._f_ptr), but that won’t be sufficient).

So we can work around that by writing some Python C API in our SWIG interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
%module test

%{
#include "test.h"
%}

%{
static __thread PyObject *callback;
static double dispatcher(double d) {
  PyObject *result = PyObject_CallFunctionObjArgs(callback, PyFloat_FromDouble(d), NULL);
  const double ret = PyFloat_AsDouble(result);
  Py_DECREF(result);

  return ret;
}
%}

%typemap(in) double(*)(double) {
  if (!PyCallable_Check($input)) SWIG_fail;
  $1 = dispatcher;
  callback = $input;
}

%include "test.h"

This is a little ugly for two reasons. Firstly it uses a (thread local) global variable to store the Python callable. That’s trivially fixable for most real world callbacks, where there’s a void* user data argument as well as the actual inputs to the callback. The “userdata” can be the Python callable in those cases.

The second issue is a little more tricky to solve though - because the callable is a wrapped C function the call sequence now involves wrapping everything up as Python types and a trip up and back from the Python interpreter just to do something that should be trivial. That’s quite a bit of overhead.

We can work backwards from a given PyObject and try to figure out which function (if any) it is a wrapper for:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
%module test

%{
#include "test.h"
%}

%{
static __thread PyObject *callback;
static double dispatcher(double d) {
  PyObject *result = PyObject_CallFunctionObjArgs(callback, PyFloat_FromDouble(d), NULL);
  const double ret = PyFloat_AsDouble(result);
  Py_DECREF(result);

  return ret;
}

SWIGINTERN PyObject *_wrap_f(PyObject *self, PyObject *args);

double (*lookup_method(PyObject *m))(double) {
  if (!PyCFunction_Check(m)) return NULL;
  PyCFunctionObject *mo = (PyCFunctionObject*)m;
  if (mo->m_ml->ml_meth == _wrap_f)
    return f;
  return NULL;
}
%}

%typemap(in) double(*)(double) {
  if (!PyCallable_Check($input)) SWIG_fail;
  $1 = lookup_method($input);
  if (!$1) {
    $1 = dispatcher;
    callback = $input;
  }
}

%include "test.h"

This does need some per function pointer code, but now it’s an optimisation rather than a requirement and it could be made more generic right a SWIG macro or two.

Solution 5:

I was working on a neater 5th solution that would use %typemap(constcode) to allow a %constant to be used as both a method and a function pointer. It turns out though that there’s already support in SWIG for doing exatly that, which I found when reading some of the SWIG source. So actually all we need to do is simply:

1
2
3
4
5
6
7
8
9
10
11
12
%module test

%{
#include "test.h"
%}

%pythoncallback;
double f(double);
%nopythoncallback;

%ignore f;
%include "test.h"

The %pythoncallback enables some global state that causes subsequent functions to be wrapped as to be usable a both a function pointer and a function! %nopythoncallback disables that.

Which then works (with or without -builtin) with:

1
2
3
import test
test.f(2.0)
test.myfun(test.f)

Which solves almost all the problems in one go. This is even documented in the manual too, although there doesn’t seem to be any mention of %pythoncallback. So the previous four solutions are mostly just useful as examples of customising SWIG interfaces.

There is still one case where solution 4 would be useful though - if you want to mix and match C and Python implemented callbacks you would need to implement a hybrid of these two. (Ideally you’d try and do the SWIG function pointer type conversion in your typemap and then iff that failed fallback to the PyCallable method instead).

refs

http://www.swig.org
http://www.swig.org/tutorial.html
http://swig.org/svn.html
https://www.cnblogs.com/dda9/p/8612068.html
https://segmentfault.com/a/1190000013219667

原文作者：ddddfang

原文链接：https://ddddfang.github.io/2019/02/19/swig/

发表日期：February 19th 2019, 7:10:20 pm

更新日期：February 28th 2019, 10:03:44 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 1. 概述
  2. 2. 2. 安装
  3. 3. 3. 为c/c++代码添加python接口
  4. 4. 4. 处理函数指针?
    1. 4.0.1. 4.1. 问 题
    2. 4.0.2. 4.2. 解决方案
      1. 4.0.2.1. Solution 1:
      2. 4.0.2.2. Solution 2:
      3. 4.0.2.3. Solution 3:
      4. 4.0.2.4. Solution 4:
      5. 4.0.2.5. Solution 5:
  • 5. refs
    • Total : 42
    2019
    binary-search bitmap caffe c++ 单例模式 common misc makefile 宏 顺序容器 智能指针 排序 shell linux command docker blog ffmpeg gdb gitbook p4 printf 可变参数 static and dynamic library 第k小的数 ubuntu setup 并查集 git draw.io markdown vscode office2016 密钥 键值对集合 链表 最大堆(大顶堆) swig python c vim 红黑树 skip-list hash-table
    algorithm-and-data-struct caffe common-misc cpp linux-daily ffmpeg