Inside the Danger Noodle

2022-08-06

In late 2021, I started putting together a native Python extension to our STEP and STEP-NC programming libraries. These C++ libraries are extensive (thousands of classes, millions of lines of code), under development for thirty years, and used in top CAD systems.

The APIs cover both high-level operations and low-level data class manipulation. We had previously done Node and .NET wrappers for the high level APIs, but I wanted to also open the data classes to Python so that a wider variety of applications could be created.

I considered SWIG and other generators but didn't think blindly pushing 4000 data class definitions through them would produce a useful result, even if they could handle it. They also seemed to do hinky things like stringitizing pointers. I figured that I could create a better result, that would feel more natural to python programmers, if I did my own from scratch.

And it worked.

I originally planned to create dynamic python types for all of these data classes, but ended up using our data dictionary calls to provide virtualized ARM (concept) and AIM (DB normalized) views of the data on a few Python classes. I also kept the link between the STEP and Python objects in a more seamless way, and eventually brought in the high level APIs in a more natural way than Node and .NET.

It is still a work in progress, but as a native extension, I have already read gigabyte-sized CAD files in a minute rather than hours or days, sliced models, created STEP-NC additive-manufacturing plans, and generated CNC control codes.

But getting to this point was an adventure that reminded me of the late-eighties, early-nineties before Google and StackOverflow replaced documentation with cut-and-paste coding. The internet is awash with examples of Python code, but relatively little about the underlying C implementation. Create an enum type dynamically with a metaclass? Plenty of Python-language examples. In C? Off into the wilderness boyo!

I had to do a lot of source diving, but over the past ten months I built up a lot of arcane knowledge which still remains arcane. It's a nice reminder of the time when the knowledge you carried in your head was not easily replaced by a quick web search!

Here is a bit of that arcana, so you can now get it with a quick web search :-)

/* How to create an enum dynamically using a metaclass in cpython */

PyObject * mod_enum = PyImport_ImportModule("enum");
if (!mod_enum) {
    Py_DECREF(m);
    return -1;
}

/* IntEnum with auto convert to int, while the plain Enum can have
 * any value so that it does not autoconvert.
 */
PyObject * type_enum = PyObject_GetAttrString(mod_enum, "IntEnum");
/*    PyObject * type_enum = PyObject_GetAttrString(mod_enum, "Enum"); */

/* Enum is actually a metaclass, so we call it with an argument
 * list to create a new instance of a type.  
 */
PyObject * dict = PyDict_New();
PyObject * val;

val = PyLong_FromLong(enumval_none);
PyDict_SetItemString(dict, "NONE", val);
Py_DECREF(val);

val = PyLong_FromLong(enumval_apples);
PyDict_SetItemString(dict, "APPLES", val);
Py_DECREF(val);

val = PyLong_FromLong(enumval_oranges);
PyDict_SetItemString(dict, "ORANGES", val);
Py_DECREF(val);

/* Pass two arguments, a string and an dict, N param means arglist
 * takes our reference so we do not need to decref the dict.
 */
PyObject * argList = Py_BuildValue(
    "sN", "FruitEnum", dict
    );

PyObject * my_enum = PyObject_CallObject(supobj, argList);
Py_DECREF(argList);
if (!my_enum) return -1;

PyObject * modname = PyUnicode_FromString("MyExtension");
(void) PyObject_SetAttrString(my_enum, "__module__", modname);
Py_DECREF(modname);

if (PyModule_AddObject(m, "FruitEnum", my_enum) < 0)
    return -1;

To get the Python enum object for a C/C++ integer enum value, you have to call your type object. To go the other way, just specify an integer value in the PyArg_ParseTuple() format string and the value will be extracted properly.

/* Find Python enumerator for integer value */

PyObject * argList = Py_BuildValue("(i)", enumval_apples);
PyObject * applesobj = PyObject_CallObject(my_enum, argList);
Py_DECREF(argList);