The Python/C API Release 3.5.1 Guido van Rossum and the Python development team February24,2016 PythonSoftwareFoundation Email: [email protected] CONTENTS 1 Introduction 3 1.1 IncludeFiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Objects,TypesandReferenceCounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.4 EmbeddingPython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.5 DebuggingBuilds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2 StableApplicationBinaryInterface 11 3 TheVeryHighLevelLayer 13 4 ReferenceCounting 19 5 ExceptionHandling 21 5.1 Printingandclearing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.2 Raisingexceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.3 Issuingwarnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.4 Queryingtheerrorindicator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 5.5 SignalHandling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5.6 ExceptionClasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.7 ExceptionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.8 UnicodeExceptionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.9 RecursionControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5.10 StandardExceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6 Utilities 33 6.1 OperatingSystemUtilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 6.2 SystemFunctions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 6.3 ProcessControl. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6.4 ImportingModules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6.5 Datamarshallingsupport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.6 Parsingargumentsandbuildingvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.7 Stringconversionandformatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.8 Reflection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.9 Codecregistryandsupportfunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 7 AbstractObjectsLayer 53 7.1 ObjectProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 7.2 NumberProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 7.3 SequenceProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 7.4 MappingProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 i 7.5 IteratorProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 7.6 BufferProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 7.7 OldBufferProtocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 8 ConcreteObjectsLayer 71 8.1 FundamentalObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 8.2 NumericObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 8.3 SequenceObjects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 8.4 ContainerObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 8.5 FunctionObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 8.6 OtherObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 9 Initialization,Finalization,andThreads 125 9.1 Initializingandfinalizingtheinterpreter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 9.2 Process-wideparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 9.3 ThreadStateandtheGlobalInterpreterLock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 9.4 Sub-interpretersupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 9.5 AsynchronousNotifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 9.6 ProfilingandTracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 9.7 AdvancedDebuggerSupport. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 10 MemoryManagement 139 10.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 10.2 RawMemoryInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 10.3 MemoryInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 10.4 CustomizeMemoryAllocators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 10.5 CustomizePyObjectArenaAllocator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 10.6 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 11 ObjectImplementationSupport 145 11.1 AllocatingObjectsontheHeap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 11.2 CommonObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 11.3 TypeObjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 11.4 NumberObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 11.5 MappingObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 11.6 SequenceObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 11.7 BufferObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 11.8 AsyncObjectStructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 11.9 SupportingCyclicGarbageCollection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 12 APIandABIVersioning 171 A Glossary 173 B Aboutthesedocuments 185 B.1 ContributorstothePythonDocumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 C HistoryandLicense 187 C.1 Historyofthesoftware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 C.2 TermsandconditionsforaccessingorotherwiseusingPython . . . . . . . . . . . . . . . . . . . . . 188 C.3 LicensesandAcknowledgementsforIncorporatedSoftware . . . . . . . . . . . . . . . . . . . . . . 191 D Copyright 205 Index 207 ii ThePython/CAPI,Release3.5.1 This manual documents the API used by C and C++ programmers who want to write extension modules or embed Python. Itisacompaniontoextending-index,whichdescribesthegeneralprinciplesofextensionwritingbutdoesnot documenttheAPIfunctionsindetail. CONTENTS 1 ThePython/CAPI,Release3.5.1 2 CONTENTS CHAPTER ONE INTRODUCTION TheApplicationProgrammer’sInterfacetoPythongivesCandC++programmersaccesstothePythoninterpreterat a variety of levels. The API is equally usable from C++, but for brevity it is generally referred to as the Python/C API.TherearetwofundamentallydifferentreasonsforusingthePython/CAPI.Thefirstreasonistowriteextension modules for specific purposes; these are C modules that extend the Python interpreter. This is probably the most commonuse. ThesecondreasonistousePythonasacomponentinalargerapplication; thistechniqueisgenerally referredtoasembeddingPythoninanapplication. Writinganextensionmoduleisarelativelywell-understoodprocess,wherea“cookbook”approachworkswell.There areseveraltoolsthatautomatetheprocesstosomeextent. WhilepeoplehaveembeddedPythoninotherapplications sinceitsearlyexistence,theprocessofembeddingPythonislessstraightforwardthanwritinganextension. Many API functions are useful independent of whether you’re embedding or extending Python; moreover, most ap- plicationsthatembedPythonwillneedtoprovideacustomextensionaswell,soit’sprobablyagoodideatobecome familiarwithwritinganextensionbeforeattemptingtoembedPythoninarealapplication. 1.1 Include Files Allfunction,typeandmacrodefinitionsneededtousethePython/CAPIareincludedinyourcodebythefollowing line: #include "Python.h" This implies inclusion of the following standard headers: <stdio.h>, <string.h>, <errno.h>, <limits.h>,<assert.h>and<stdlib.h>(ifavailable). Note: SincePythonmaydefinesomepre-processordefinitionswhichaffectthestandardheadersonsomesystems, youmustincludePython.hbeforeanystandardheadersareincluded. All user visible names defined by Python.h (except those defined by the included standard headers) have one of the prefixesPyor_Py. Namesbeginningwith_PyareforinternalusebythePythonimplementationandshouldnotbe usedbyextensionwriters. Structuremembernamesdonothaveareservedprefix. Important:usercodeshouldneverdefinenamesthatbeginwithPyor_Py.Thisconfusesthereader,andjeopardizes theportabilityoftheusercodetofuturePythonversions,whichmaydefineadditionalnamesbeginningwithoneof theseprefixes. The header files are typically installed with Python. On Unix, these are located in the directories prefix/include/pythonversion/ and exec_prefix/include/pythonversion/, where prefix and exec_prefix are defined by the corresponding parameters to Python’s configure script and version is sys.version[:3]. OnWindows,theheadersareinstalledinprefix/include,whereprefixistheinstal- lationdirectoryspecifiedtotheinstaller. 3 ThePython/CAPI,Release3.5.1 Toincludetheheaders,placebothdirectories(ifdifferent)onyourcompiler’ssearchpathforincludes. Donotplace theparentdirectoriesonthesearchpathandthenuse#include <pythonX.Y/Python.h>; thiswillbreakon multi-platform builds since the platform independent headers under prefix include the platform specific headers fromexec_prefix. C++usersshouldnotethatthoughtheAPIisdefinedentirelyusingC,theheaderfilesdoproperlydeclaretheentry pointstobeextern "C",sothereisnoneedtodoanythingspecialtousetheAPIfromC++. 1.2 Objects, Types and Reference Counts MostPython/CAPIfunctionshaveoneormoreargumentsaswellasareturnvalueoftypePyObject*. Thistypeis apointertoanopaquedatatyperepresentinganarbitraryPythonobject. SinceallPythonobjecttypesaretreatedthe samewaybythePythonlanguageinmostsituations(e.g.,assignments,scoperules,andargumentpassing),itisonly fittingthattheyshouldberepresentedbyasingleCtype.AlmostallPythonobjectsliveontheheap:youneverdeclare anautomaticorstaticvariableoftypePyObject,onlypointervariablesoftypePyObject*canbedeclared. The soleexceptionarethetypeobjects;sincethesemustneverbedeallocated,theyaretypicallystaticPyTypeObject objects. AllPythonobjects(evenPythonintegers)haveatypeandareferencecount. Anobject’stypedetermineswhatkindof objectitis(e.g.,aninteger,alist,orauser-definedfunction;therearemanymoreasexplainedintypes). Foreachof thewell-knowntypesthereisamacrotocheckwhetheranobjectisofthattype;forinstance,PyList_Check(a) istrueif(andonlyif)theobjectpointedtobyaisaPythonlist. 1.2.1 Reference Counts Thereferencecountisimportantbecausetoday’scomputershaveafinite(andoftenseverelylimited)memorysize;it countshowmanydifferentplacestherearethathaveareferencetoanobject. Suchaplacecouldbeanotherobject,or aglobal(orstatic)Cvariable,oralocalvariableinsomeCfunction. Whenanobject’sreferencecountbecomeszero, theobjectisdeallocated. Ifitcontainsreferencestootherobjects,theirreferencecountisdecremented. Thoseother objectsmaybedeallocatedinturn,ifthisdecrementmakestheirreferencecountbecomezero,andsoon. (There’san obviousproblemwithobjectsthatreferenceeachotherhere;fornow,thesolutionis“don’tdothat.”) Referencecountsarealwaysmanipulatedexplicitly.ThenormalwayistousethemacroPy_INCREF()toincrement an object’s reference count by one, and Py_DECREF() to decrement it by one. The Py_DECREF() macro is considerablymorecomplexthantheincrefone,sinceitmustcheckwhetherthereferencecountbecomeszeroandthen causetheobject’sdeallocatortobecalled.Thedeallocatorisafunctionpointercontainedintheobject’stypestructure. Thetype-specificdeallocatortakescareofdecrementingthereferencecountsforotherobjectscontainedintheobject ifthisisacompoundobjecttype,suchasalist,aswellasperforminganyadditionalfinalizationthat’sneeded.There’s nochancethatthereferencecountcanoverflow;atleastasmanybitsareusedtoholdthereferencecountasthereare distinctmemorylocationsinvirtualmemory(assumingsizeof(Py_ssize_t) >= sizeof(void*)). Thus, thereferencecountincrementisasimpleoperation. Itisnotnecessarytoincrementanobject’sreferencecountforeverylocalvariablethatcontainsapointertoanobject. In theory, the object’s reference count goes up by one when the variable is made to point to it and it goes down by onewhenthevariablegoesoutofscope. However,thesetwocanceleachotherout,soattheendthereferencecount hasn’tchanged. Theonlyrealreasontousethereferencecountistopreventtheobjectfrombeingdeallocatedaslong asourvariableispointingtoit. Ifweknowthatthereisatleastoneotherreferencetotheobjectthatlivesatleastas longasourvariable,thereisnoneedtoincrementthereferencecounttemporarily. Animportantsituationwherethis arisesisinobjectsthatarepassedasargumentstoCfunctionsinanextensionmodulethatarecalledfromPython;the callmechanismguaranteestoholdareferencetoeveryargumentforthedurationofthecall. However, a common pitfall is to extract an object from a list and hold on to it for a while without incrementing its referencecount. Someotheroperationmightconceivablyremovetheobjectfromthelist,decrementingitsreference count and possible deallocating it. The real danger is that innocent-looking operations may invoke arbitrary Python 4 Chapter1. Introduction ThePython/CAPI,Release3.5.1 codewhichcoulddothis;thereisacodepathwhichallowscontroltoflowbacktotheuserfromaPy_DECREF(), soalmostanyoperationispotentiallydangerous. A safe approach is to always use the generic operations (functions whose name begins with PyObject_, PyNumber_, PySequence_ or PyMapping_). These operations always increment the reference count of the objecttheyreturn. ThisleavesthecallerwiththeresponsibilitytocallPy_DECREF()whentheyaredonewiththe result;thissoonbecomessecondnature. ReferenceCountDetails ThereferencecountbehavioroffunctionsinthePython/CAPIisbestexplainedintermsofownershipofreferences. Ownership pertains to references, never to objects (objects are not owned: they are always shared). “Owning a reference”meansbeingresponsibleforcallingPy_DECREFonitwhenthereferenceisnolongerneeded. Ownership canalsobetransferred,meaningthatthecodethatreceivesownershipofthereferencethenbecomesresponsiblefor eventuallydecref’ingitbycallingPy_DECREF()orPy_XDECREF()whenit’snolongerneeded—orpassingon thisresponsibility(usuallytoitscaller). Whenafunctionpassesownershipofareferenceontoitscaller,thecalleris saidtoreceiveanewreference. Whennoownershipistransferred,thecallerissaidtoborrowthereference. Nothing needstobedoneforaborrowedreference. Conversely,whenacallingfunctionpassesinareferencetoanobject,therearetwopossibilities: thefunctionsteals areferencetotheobject,oritdoesnot. Stealingareferencemeansthatwhenyoupassareferencetoafunction,that functionassumesthatitnowownsthatreference,andyouarenotresponsibleforitanylonger. Fewfunctionsstealreferences;thetwonotableexceptionsarePyList_SetItem()andPyTuple_SetItem(), which steal a reference to the item (but not to the tuple or list into which the item is put!). These functions were designedtostealareferencebecauseofacommonidiomforpopulatingatupleorlistwithnewlycreatedobjects;for example, thecodetocreatethetuple (1, 2, "three") couldlooklikethis(forgetting abouterrorhandlingfor themoment;abetterwaytocodethisisshownbelow): PyObject *t; t = PyTuple_New(3); PyTuple_SetItem(t, 0, PyLong_FromLong(1L)); PyTuple_SetItem(t, 1, PyLong_FromLong(2L)); PyTuple_SetItem(t, 2, PyUnicode_FromString("three")); Here, PyLong_FromLong() returns a new reference which is immediately stolen by PyTuple_SetItem(). Whenyouwanttokeepusinganobjectalthoughthereferencetoitwillbestolen,usePy_INCREF()tograbanother referencebeforecallingthereference-stealingfunction. Incidentally, PyTuple_SetItem() is the only way to set tuple items; PySequence_SetItem() and PyObject_SetItem() refuse to do this since tuples are an immutable data type. You should only use PyTuple_SetItem()fortuplesthatyouarecreatingyourself. EquivalentcodeforpopulatingalistcanbewrittenusingPyList_New()andPyList_SetItem(). However, in practice, you will rarely use these ways of creating and populating a tuple or list. There’s a generic function, Py_BuildValue(), that can create most common objects from C values, directed by a format string. For example, the above two blocks of code could be replaced by the following (which also takes care of the error checking): PyObject *tuple, *list; tuple = Py_BuildValue("(iis)", 1, 2, "three"); list = Py_BuildValue("[iis]", 1, 2, "three"); It is much more common to use PyObject_SetItem() and friends with items whose references you are only borrowing,likeargumentsthatwerepassedintothefunctionyouarewriting. Inthatcase,theirbehaviourregarding 1.2. Objects,TypesandReferenceCounts 5 ThePython/CAPI,Release3.5.1 referencecountsismuchsaner,sinceyoudon’thavetoincrementareferencecountsoyoucangiveareferenceaway (“haveitbestolen”).Forexample,thisfunctionsetsallitemsofalist(actually,anymutablesequence)toagivenitem: int set_all(PyObject *target, PyObject *item) { Py_ssize_t i, n; n = PyObject_Length(target); if (n < 0) return -1; for (i = 0; i < n; i++) { PyObject *index = PyLong_FromSsize_t(i); if (!index) return -1; if (PyObject_SetItem(target, index, item) < 0) { Py_DECREF(index); return -1; } Py_DECREF(index); } return 0; } The situation is slightly different for function return values. While passing a reference to most functions does not changeyourownershipresponsibilitiesforthatreference,manyfunctionsthatreturnareferencetoanobjectgiveyou ownership of the reference. The reason is simple: in many cases, the returned object is created on the fly, and the referenceyougetistheonlyreferencetotheobject. Therefore,thegenericfunctionsthatreturnobjectreferences,like PyObject_GetItem()andPySequence_GetItem(),alwaysreturnanewreference(thecallerbecomesthe ownerofthereference). Itisimportanttorealizethatwhetheryouownareferencereturnedbyafunctiondependsonwhichfunctionyoucall only—theplumage(thetypeoftheobjectpassedasanargumenttothefunction)doesn’tenterintoit! Thus,ifyou extractanitemfromalistusingPyList_GetItem(),youdon’townthereference—butifyouobtainthesame itemfromthesamelistusingPySequence_GetItem()(whichhappenstotakeexactlythesamearguments),you doownareferencetothereturnedobject. Hereisanexampleofhowyoucouldwriteafunctionthatcomputesthesumoftheitemsinalistofintegers; once usingPyList_GetItem(),andonceusingPySequence_GetItem(). long sum_list(PyObject *list) { Py_ssize_t i, n; long total = 0, value; PyObject *item; n = PyList_Size(list); if (n < 0) return -1; /* Not a list */ for (i = 0; i < n; i++) { item = PyList_GetItem(list, i); /* Can't fail */ if (!PyLong_Check(item)) continue; /* Skip non-integers */ value = PyLong_AsLong(item); if (value == -1 && PyErr_Occurred()) /* Integer too big to fit in a C long, bail out */ 6 Chapter1. Introduction