GlueGen is a tool which automatically generates the Java and JNI code necessary to call C libraries. It reads as input ANSI C header files and separate configuration files which provide control over many aspects of the glue code generation. GlueGen uses a complete ANSI C parser and an internal representation (IR) capable of representing all C types to represent the APIs for which it generates interfaces. It has the ability to perform significant transformations on the IR before glue code emission. GlueGen is currently powerful enough to bind even low-level APIs such as the Java Native Interface (JNI) and the AWT Native Interface (JAWT) back up to the Java programming language.
GlueGen is currently used to generate the JOGL interface to the OpenGL 3D graphics API and the JOAL interface to the OpenAL audio library. In the case of JOGL, GlueGen is used not only to bind OpenGL to Java, but also the low-level windowing system APIs on the Windows, X11 and Mac OS X platforms. The implementation of the JOGL library is thereby written in the Java programming language rather than in C, which has offered considerable advantages during the development of the library.
GlueGen is designed in modular form and can be extended to alter the glue code emission style or to generate interface code for other languages than Java.
This manual describes how to use GlueGen to bind new C libraries to the Java programming language.
GlueGen supports two basic styles of glue code generation: everything in one class, or a separate interface and implementing class. The first mode, "AllStatic", exposes the underlying C functions as a set of static Java methods in a concrete class. This is a straightforward binding mechanism, but has the disadvantage of tying users to a concrete class (which may or may not be a problem) and makes it more difficult to support certain kinds of call-through-function-pointer semantics required by certain C APIs. The second mode, "InterfaceAndImpl", exposes the C functions as methods in an interface and emits the implementation of that interface into a separate class and package. The implementing class is not intended to be in the public API; this more strongly separates the user from the implementation of the API. Additionally, because it is necessary to hold an instance of the implementing class in order to access the underlying C routines, it is easier to support situations where call-through-function-pointer semantics must be followed, in particular where those function pointers might change from instance to instance.
The generated glue code follows some basic rules in binding C APIs to Java:
int*
,
float*
) are bound to java.nio Buffer subclasses
(IntBuffer
, FloatBuffer
) and optionally
to Java arrays (int[]
, float[]
).
void*
are bound to java.nio.Buffer.
void*
argument will allow either a direct or non-direct java.nio
Buffer to be passed as argument. If the NioDirectOnly directive is specified,
however, only a direct Buffer will be accepted.
void*
return values apply to
those for pointers to typed primitives.
char*
may be bound to
java.lang.String using the ArgumentIsString or ReturnsString directives.
#define
statements in header files mapping names to
constant values are exposed as public static final constant
values in either the generated interface or AllStatic class.
JNIEnv*
and jobject
. The tool
understands that the JNIEnv*
argument is implicit
and that jobject
maps to java.lang.Object at the
Java programming language level. While this is most useful when
binding JDK-internal APIs such as the JAWT to Java, there may
be other JNI libraries which expose C functions taking these
data types, and GlueGen can very easily bind to them.
This section provides motivation for the design of the GlueGen tool and is not necessary to understand how to use the tool.
There are many tools available for assisting in the autogeneration of foreign function interfaces for various high-level languages. Only a few examples include Header2Scheme, an early tool allowing binding of a limited subset of C++ to the Scheme programming language; SWIG, a tool released at roughly the same time as Header2Scheme which by now supports binding C and C++ libraries to a variety of scripting languages; and JNIWrapper, a commercial tool automating the binding of C APIs to Java. Other language-specific tools such as Perl's XS, Boost.Python and many others exist.
GlueGen was designed with a few key principles in mind. The most fundamental was to support binding of the lowest-level APIs on a given platform up to the Java programming language. The intended goal, in the context of the JOGL project, was to allow subsets of the Win32 and X11 APIs to be exposed to Java, and to use those APIs to write the behind-the-scenes OpenGL context creation and management code in Java instead of C. This informed several other design goals:
GlueGen was designed with the Java programming language in mind, but is not necessarily restricted to generating glue code for the Java language. The tool is divided into separate parse and code generation phases, and the internal representation is fairly easy to iterate over. The core driver of GlueGen may therefore be useful in producing other tools which autogenerate foreign function interfaces to C libraries for other languages.
GlueGen accepts four kinds of command-line arguments:
#include
directives. Unlike most C preprocessors,
however, GlueGen has no default include path, so it is typically
necessary to supply at least one -I
option on the
command line in order to handle any #include
directives in the file being parsed.
com.sun.gluegen.GlueEmitter
interface. If this
option is not specified, a
com.sun.gluegen.JavaEmitter
will be used by default.
GlueGen contains and uses a minimal C preprocessor called the "Pseudo
C Pre-Processor", or PCPP. A slightly specialized C preprocessor is
required for correct glue code generation with most libraries.
Constant values intended for use by end users are defined in many C
libraries' headers using #define
s rather than constant
int declarations, and if the header is processed by a full C
preprocessor then the #define statements will be stripped become
unavailable for processing by the glue code generator.
PCPP is largely an invisible part of the glue code generation process; however, it has certain limitations which make it difficult to parse certain header files. First, it does not support macro evaluation in any form, so if a header relies on macro evaluation in order to generate code, PCPP will fail. It is possible that PCPP may fail silently in this situation, causing GlueGen to simply not produce code for the associated constructs. If GlueGen's output is not as expected and there is heavy use of the C preprocessor in the header, run PCPP against the header directly (PCPP takes simply the -I and filename arguments accepted by GlueGen) and examine the output.
Second, PCPP contains only limited support for #if
clauses. Generally speaking, its handling of #if defined(foo) ||
defined(bar)
constructs is limited to approximately what is
required to handle the OpenGL header files. If the header being parsed
relies on moderately complicated expressions being evaluated by the C
preprocessor, check the output from PCPP and ensure it is as expected.
Contributions to PCPP would be especially welcome. It would be very desirable to turn it into a full-blown C preprocessor with simply the option of passing through #define statements unchanged.
Error reporting by GlueGen's parser is currently less than ideal.
Because PCPP makes #include
directives disappear
completely with respect to the C parser (it appears that the
#line
directives it emits are not being consumed properly
-- an area which needs more investigation), the line numbers reported
in parse failures are incorrect in all but the simplest cases. This
makes it difficult to determine in exactly what header file and on
exactly what construct the C parser failed.
Fortunately, there is a relatively simple workaround. PCPP can be run
with all of the same -I arguments passed to GlueGen and the result
piped to a new .c file. GlueGen can then be invoked on that .c file
(now containing no #include
directives) and the line
numbers on any parse failures will be correct.
As much as is possible, GlueGen is intended to operate on unmodified C
header files, so that it is easy to upgrade the given C API being
bound to Java simply by dropping in a new set of header files.
However, most C headers contain references to standard headers like
stdio.h
, and if this header is parsed by GlueGen, the
tool will automatically attempt to generate Java entry points for such
routines as fread
and fwrite
, among others.
It is impractical to exclude these APIs on a case by case basis.
Therefore, the suggested technique to avoid polluting the binding with
these APIs is to "stub out" the headers.
GlueGen searches the include path for headers in the order the include
directories were specified to the tool. Placing another directory in
front of the one in which the bulk of the headers are found allows,
for example, an alternative stdio.h
to be inserted which
contains few or no declarations but which satisfies the need of the
dependent header to find such a file.
GlueGen uses a complete ANSI and GNU C parser written by John Mitchell and Monty Zukowski from the set of grammars available for the ANTLR tool by Terrence Parr. As a complete C parser, this grammar requires all data types encountered during the parse to be fully defined. Often a particular header will be included by another one in order to pick up data type declarations rather than API declarations. Stubbing out the header with a smaller one providing a "fake" type declaration is a useful technique for avoiding the binding of unnecessary APIs during the glue code process.
Here's an example from the JOGL glue code generation process. The
glext.h
header defining OpenGL extensions references
stddef.h
in order to pick up the ptrdiff_t
data type. We choose to not include the real stddef.h but instead to
swap in a stub header. The contents of this header are therefore as
follows:
#if defined(_WIN64) typedef __int64 ptrdiff_t; #elif defined(__ia64__) || defined(__x86_64__) typedef long int ptrdiff_t; #else typedef int ptrdiff_t; #endifThis causes the ptrdiff_t data type to be defined appropriately for the current architecture. It will be referenced during the glue code generation and cause a Java value of the appropriate type (int or long) to be used to represent it.
This is not the best example because it involves a data type which
changes size between 32- and 64-bit platforms, and there are otner
considerations to take into account in these situations (see the
section 32- and 64-bit considerations). Here's
another example, again from the JOGL source tree. JOGL binds the AWT
Native Interface, or JAWT, up to the Java programming language so that
the low-level code which binds OpenGL contexts to Windows device
contexts may be written in Java. The JDK's jawt_md.h
on
the Windows platform includes windows.h
to pick up the
definitions of data types such as HWND
(window handle)
and HDC
(handle to device context). However, it is
undesirable to try to parse the real windows.h
just to
pick up these typedefs; not only does this header contain thousands of
unneeded APIs, but it also uses certain macro constructs not supported
by GlueGen's minimal C preprocessor. To avoid
these problems, a "stub" windows.h
header is placed in
GlueGen's include path containing only the necessary typedefs:
typedef struct _handle* HANDLE; typedef HANDLE HDC; typedef HANDLE HWND;Note that it is essential that the type being specified to GlueGen is compatible at least in semantics with the real definition of the HANDLE typedef in the real
windows.h
, so that during
compilation of GlueGen's autogenerated C code, when the real
windows.h
is referenced by the C compiler, the
autogenerated code will compile correctly.
This example is not really complete as it also requires consideration of the size of data types on 32- and 64-bit platforms as well as a discussion of how certain opaque data types are described to GlueGen and exposed in its autogenerated APIs. Nonetheless, it illustrates at a basic level why using a stub header is necessary and useful in certain situations.
When binding C functions to the Java programming language, it is important that the resulting Java code support execution on a 64-bit platform if the associated native methods are compiled appropriately. In other words, the public Java API should not change if the underlying C data types change to another data model such as LP64 (in which longs and pointers become 64-bit).
GlueGen internally maintains two descriptions of the underlying C data model: one for 32-bit architectures and one for 64-bit architectures. These machine descriptions are used when deciding the mapping between integral C types such as int and long and the corresponding Java types, as well as when laying out C structs for access by the Java language. For each autogenerated C struct accessor, both a 32-bit and 64-bit variant are generated behind the scenes, ensuring that the resulting Java code will run correctly on both 32-bit and 64-bit architectures.
When generating the main class containing the bulk of the method
bindings, GlueGen uses the 64-bit machine description to map C data
types to Java data types. This ensures that the resulting code will
run properly on 64-bit platforms. Note that it also generally means
that C long
s will be mapped to Java long
s,
since an LP64 data model is assumed.
If Opaque directives are used to cause a given C integer or pointer data type to be mapped directly to a Java primitive type, care should be taken to make sure that the Java primitive type is wide enough to hold all of the data even on 64-bit platforms. Even if the data type is defined in the header file as being only a 32-bit C integer, if there is a chance that on a 64-bit platform the same header may define the data type as a 64-bit C integer or long, the Opaque directive should map the C type to a Java long.
Complex header files may contain declarations for certain data types that are either too complex for GlueGen to handle or unnecessarily complex from the standpoint of glue code generation. In these situations a stub header may be used to declare a suitably compatible typedef for the data type. An Opaque directive can be used to map the resulting typedef to a Java primitive type if it is undesirable to expose it as a full-blown Java wrapper class.
GlueGen hashes all typedefs internally down to their underlying primitive type. (This is probably not really correct according to the C type system, but is correct enough from a glue code generation standpoint, where if the types are compatible they are considered equivalent.) This means that if the parser encounters
typedef void* LPVOID;then an Opaque directive stating
Opaque long LPVOIDwill cause all
void*
or LPVOID
arguments in
the API to be mapped to Java longs, which is almost never desirable.
Unfortunately, it is not currently possible to distinguish between the
LPVOID typedef and the underlying void*
data type in this
situation.
A similar problem occurs for other data types for which Opaque
directives may be desired. For example, a Windows HANDLE equates to a
typedef to void*
, but performing this typedef in a stub
header and then adding the Opaque directive
Opaque long HANDLEwill cause all void* arguments to be exposed as Java longs instead of Buffers, which is again undesirable. Attempting to work around the problem by typedef'ing HANDLE to an integral type, as in:
typedef long HANDLE;may itself have problems, because GlueGen will assume the two integral types are compatible and not perform any intermediate casts between HANDLE and jlong in the autogenerated C code. (When casting between a pointer type and a JNI integral type such as jlong in C code, GlueGen automatically inserts casts to convert the pointer first to an "intptr_t" and then to the appropriate JNI type, in order to silence compiler warnings and/or errors.)
What is desired is to produce a new type name distinct from all others but still compatible with the pointer semantics of the original type. Then an Opaque directive can be used to map the new type name to, for example, a Java long.
To implement this in the context of the HANDLE example, the following typedef may be inserted into the stub header:
typedef struct _handle* HANDLE;This uses a pointer to an anonymous struct name to produce a new pointer type. This is legal ANSI C and is supported by GlueGen's parser without having seen a declaration for "struct _handle". Subsequently, an Opaque directive can be used to map the HANDLE data type to a Java long:
Opaque long HANDLENow HANDLEs are exposed to Java as longs as desired. A similar technique is used to expose XIDs on the X11 platform as Java longs.
In addition to the C headers, GlueGen requires a certain amount of metadata in the form of configuration files in order to produce its glue code. There are three basic reasons for this: first, GlueGen must be informed into which Java classes the C methods are to be bound; second, there are many configuration options for the generated glue code, and passing them all on the command line is infeasible; and third, there are ambiguities in many constructs in the C programming language which must be resolved before a Java binding can be produced.
The contents of the configuration file are dependent on the class of emitter specified to GlueGen. Currently there are three built-in emitter classes: JavaEmitter, which produces a basic, static Java binding of C functions; ProcAddressEmitter, which extends JavaEmitter by calling the underlying C functions through function pointers, resulting in more dynamic behavior and supporting C APIs with optional functionality; and GLEmitter, which specializes ProcAddressEmitter to support some OpenGL-specific constructs. The GLEmitter will be ignored in this manual as it is specialized for JOGL and provides very little additional functionality beyond the ProcAddressEmitter. The JavaEmitter and ProcAddressEmitter support many options in their configuration files. As the ProcAddressEmitter is a subclass of JavaEmitter, all of the constructs in the JavaEmitter's configuration files are also legal in the ProcAddressEmitter's configuration files.
The configuration files have a very simple line-by-line structure, and
are parsed by a very rudimentary, hand-written parser. Each
non-whitespace and non-comment line (note: comment lines begin with
'#') contains a directive like Package
,
Style
or JavaClass
followed by arguments to
that directive. There are a certain set of directives that are
required for any code generation; others are optional and their
omission results in some default behavior. Directives are
case-insensitive.
The following is an exhaustive list of the options currently supported by each of these emitters' configuration files. It is difficult to see exactly how to use the tool based simply on these descriptions, so the examples (FIXME) may be more helpful in seeing exactly how to structure a configuration file for proper glue code generation.
Note that only a very few of the following directives are specified as being "required" rather than "optional"; these indicate the minimal directives needed for a valid configuration file to begin to get glue code to be produced. In general, these are Package, ImplPackage, JavaClass, ImplJavaClass, and Style. Other directives such as NioDirectOnly are required in some circumstances for the glue code to be correct, and some such as ReturnedArrayLength, ReturnValueCapacity, and ReturnValueLength should be specified in some situations in order for certain return values to be useful at the Java level.
The following directives are specified in alphabetical order, although this is not necessarily the best semantic order.
AccessControl [method name] [ PUBLIC | PROTECTED |
PRIVATE | PACKAGE_PRIVATE ]
ArgumentIsString [function name]
[indices...]
where the first argument index is 0 char*
(or compatible data type) arguments, indicates that
those arguments are semantically null-terminated C strings rather than
arbitrary arrays of bytes. The generated glue code will be modified to
emit those arguments as java.lang.String objects rather than
byte[]
or ByteBuffer
.
ClassJavadoc [class name] [code...]
/**
and trailing */
must be included in the
correct place. Each line of Javadoc is emitted in the order
encountered during parsing of the configuration files.
CustomCCode [code...]
CustomJavaCode [class name] [code...]
EmitStruct [C struct type name]
Extends [Java interface name] [interface name to
extend]
HierarchicalNativeOutput true
Ignore [regexp]
IgnoreField [struct type name] [field name]
Implements [Java class name] [interface name to
implement]
ImplJavaClass [class name]
ImplPackage [package name]
Import [package name]
(no trailing semicolon)
Include [filename]
IncludeAs [prefix tokens] [filename]
IncludeAs CustomJavaCode
MyClass MyClass-CustomJavaCode.java
.
JavaClass [class name]
JavaEpilogue [C function name] [code...]
JavaOutputDir [directory name]
JavaPrologue [C function name] [code...]
ManuallyImplement [function name]
NativeOutputDir [directory name]
NioDirectOnly [function name]
void*
, float*
, etc.,
GlueGen will typically generate up to two overloaded Java methods, one
taking a Buffer
or Buffer
subclass such as
FloatBuffer
, and one taking a primitive array such as
float[]
. (In the case of void*
outgoing
arguments, GlueGen produces only one variant taking a Buffer.)
Normally the generated glue code accepts either a "direct" or
non-"direct" buffer (according to the New I/O APIs) as argument.
However, if the semantics of the C function are that it either expects
to hold on to this pointer past the point of the function call, or if
it can block while holding on to the pointer, the
NioDirectOnly
directive must be
specified for this C function in order for the generated glue code to
be correct. Failing to observe this requirement may cause JVM hangs or
crashes.
Opaque [Java primitive data type] [C data
type]
long
s. It is also
useful for forcing certain integral C data types to be exposed as e.g.
long
to Java to ensure 64-bit cleanliness of the
generated glue code. See the (FIXME) examples. The C data type may
be a multiple-level pointer type; for example Opaque long
void**
. Note that it is not currently supported to make a given
data type opaque for just a few functions; the Opaque directive
currently applies to all C functions in the headers being parsed.
This means that sweeping Opaque declarations like Opaque long
void*
will likely have unforseen and undesirable consequences.
Package [package name]
(no trailing
semicolon) RangeCheck [C function name] [argument number]
[expression]
RangeCheckBytes [C function name] [argument number]
[expression]
RenameJavaMethod [from name] [to name]
RenameJavaType [from name] [to name]
ReturnedArrayLength [C function name]
[expression]
where expression
is a legal Java
expression with MessageFormat specifiers such as "{0}". These
specifiers will be replaced in the generated glue code with the
incoming argument names where the first argument to the method is
numbered 0. XVisualInfo*
, indicates that the returned pointer is
to be treated as an array and specifies the length of the returned
array as a function of the arguments passed to the function. Note that
this directive differs subtly from ReturnValueCapacity and
ReturnValueLength. It is also sometimes most useful in conjunction
with the TemporaryCVariableDeclaration
and TemporaryCVariableAssignment directives.
ReturnsString [function name]
char*
or compatible type actually returns a
null-terminated C string which should be exposed as a
java.lang.String.
ReturnValueCapacity [C function name]
[expression]
Buffer
or
subclass wrapping a C primitive pointer such as char*
or
float*
being returned from a C function. Typically
necessary in order to properly use such pointer return results from
Java. As in the ReturnedArrayLength
directive, argument name substitution is performed on MessageFormat
expressions such as "{0}" where the first argument is index 0.
ReturnValueLength [C function name]
[expression]
RuntimeExceptionType [class name]
RuntimeException
.
StructPackage [C struct type name] [package
name]
. Package name contains no trailing semicolon. Style [ AllStatic | InterfaceAndImpl |
InterfaceOnly | ImplOnly ]
TemporaryCVariableAssignment [C function name]
[code...]
TemporaryCVariableDeclaration [C function name]
[code...]
Unignore [regexp]
Unimplemented [regexp]
The ProcAddressEmitter is a subclass of the core JavaEmitter which knows how to call C functions through function pointers. In particular, the ProcAddressEmitter detects certain constructs in C header files which imply that the APIs are intended to be called through function pointers, and generates the glue code appropriately to support that.
The ProcAddressEmitter detects pairs of functions and function pointer typedefs in a set of header files. If it finds a matching pair, it converts the glue code emission style for that API to look for the function to call in an autogenerated table called a ProcAddressTable rather than linking the autogenerated JNI code directly to the function. It then changes the calling convention of the underlying native method to pass the function pointer from Java down to C, where the call-through-function-pointer is performed.
The ProcAddressEmitter discovers the function and function pointer
pairs by being informed of the mapping between their names by the
user. In the OpenGL and OpenAL libraries, there are fairly simple
mappings between the functions and function pointers. For example, in
the OpenGL glext.h
header file, one may find the
following pair:
GLAPI void APIENTRY glFogCoordf (GLfloat); ... typedef void (APIENTRYP PFNGLFOGCOORDFPROC) (GLfloat coord);Therefore the mapping rule between the function name and the function pointer typedef for the OpenGL extension header file is "PFN + Uppercase(funcname) + PROC". Similarly, in the OpenAL 1.1 header files, one may find the following pair:
AL_API void AL_APIENTRY alEnable( ALenum capability ); ... typedef void (AL_APIENTRY *LPALENABLE)( ALenum capability );Therefore the mapping rule between the function name and the function pointer typedef for the OpenAL header files is "LP + Uppercase(funcname)".
These are the two principal function pointer-based APIs toward which the GlueGen tool has currently been applied. It may turn out to be that this simple mapping heuristic is insufficient, in which case it will need to be extended in a future version of the GlueGen tool.
Note that it is currently the case that in order for the
ProcAddressEmitter to notice that a given function should be called
through a function pointer, it must see both the function prototype as
well as the function pointer typedef. Some headers, in particular the
OpenAL headers, have their #ifdefs
structured in such a
way that either the declaration or the typedef is visible, but not
both simultaneously. Because the PCPP C
preprocessor GlueGen uses obeys #ifdefs
, it is in a
situation like this that the headers would have to be modified to
allow GlueGen to see both declarations.
The following directives are specified in alphabetical order, although this is not necessarily the best semantic order. The ProcAddressEmitter also accepts all of the directives supported by the JavaEmitter. The required directives are GetProcAddressTableExpr and ProcAddressNameExpr.
EmitProcAddressTable [true | false]
ForceProcAddressGen [function name]
GetProcAddressTableExpr [expression]
GetProcAddressTableExpr _context.getGLProcAddressTable()
.
In the JOAL project, the ProcAddressTables are currently held in a
separate class accessed via static methods, so one of the associated
directives is GetProcAddressTableExpr
ALProcAddressLookup.getALCProcAddressTable()
.
ProcAddressNameExpr [expression]
$UpperCase(arg)
converts the argument to
uppercase. "UpperCase" is case-insensitive.
$LowerCase(arg)
converts the argument to
lowercase. "LowerCase" is case-insensitive.
{0}
represents the name of the function.
PFN
$UPPERCASE({0}) PROC
. The ProcAddressNameExpr for the OpenAL
functions as described at the start of this section is LP
$UPPERCASE({0})
.
ProcAddressTableClassName [class name]
ProcAddressTablePackage [package name] (no
trailing semicolon)
SkipProcAddressGen [function name]