Quantitative Analysis
Parallel Processing
Numerical Analysis
C++ Multithreading
Python for Excel
Python Utilities
Services
Author

I. Installation.
II. Threading primitives.
III. NonBlockingQueue.
1. NonBlockingQueue design.
2. Simplest example with NonBlockingQueue.
3. NonBlockingQueue prototypes.
4. Python-based acceptance test of NonBlockingQueue.
A. Basic Python API for NonBlockingQueue.
B. Multithreaded test of NonBlockingQueue.
IV. ThreadPool.
V. ThreadMaster.
VI. OTS Scheduler.
VII. Bibliography
Downloads. Index. Contents.

Basic Python API for NonBlockingQueue.


he following code exposes NonBlockingQueue to Python. We still restrict our attention to storing std::string.

Lines12-23. We can no longer use the template instantiation Data=std::string because std::string is copied by value somewhere inside the code that boost::python generates. We need to insure that a piece of Data, created in Python interpreter is passed by reference all the way into the C++ code and back. Therefore, we place std::string inside of a boost::noncopyable object.

Lines 30-42. These are functions for conversion of Queue into a string representation.

Lines 44-47. The function makeNode is introduced because boost::python does not process a constructor with the "volatile" type modifier. Hence, we cannot export the constructor of Node into Python directly.

Lines 49-72. This is a straightforward utilization of boost::python API. We export Wrapper, Queue and Node into Python interpreter.

The full code and project files are available at the download section.

1\#include <boost/python.hpp>

2\#include "PythonNonBlockingQueue.hpp"

3\#include "NonBlockingQueue.hpp"

4\#include <ots/otsConfig.hpp>

5\#include <ots/utils/toString.hpp>

6\#include <boost/function.hpp>

7\#include <boost/bind.hpp>

8\#include <ots/math/random/uniform.hpp>

9\

10\namespace ots { namespace scheduler { namespace testNonBlockingQueue {

11\

12\ template <class Data> class WrapperTmpl : boost::noncopyable

13\ {

14\ public:

15\ explicit WrapperTmpl(const Data& d) : theData(d) {}

16\ const Data theData;

17\ std::string toString() const

18\ {

19\ std::stringstream os;

20\ os<<theData<<'\0';

21\ return os.str();

22\ }

23\ };

24\

25\ typedef std::string Data;

26\ typedef WrapperTmpl<Data> Wrapper;

27\ typedef ots::scheduler::NonBlockingQueue<Wrapper> Queue;

28\ typedef Queue::Node Node;

29\

30\ std::pair<bool,std::string> wrapperToString( volatile Wrapper* w )

31\ {

32\ const Wrapper& ww=*const_cast<Wrapper*>(w);

33\ return std::pair<bool,std::string>(true,ww.toString());

34\ }

35\ std::string queueToString( const Queue& queue )

36\ {

37\ return queue.toString(wrapperToString);

38\ }

39\ std::string nodeToString( const Node& node )

40\ {

41\ return node.toString(wrapperToString);

42\ }

43\

44\ Node* makeNode( Wrapper& w )

45\ {

46\ return new Node(w);

47\ }

48\

49\ void pythonNonBlockingQueue()

50\ {

51\ using namespace boost::python;

52\ class_<Wrapper,boost::noncopyable>("Wrapper",init<std::string>())

53\ .def("__str__",&Wrapper::toString)

54\ ;

55\ class_<Queue,boost::noncopyable>("Queue")

56\ .def("__str__",&queueToString)

57\ .def("pop",

58\ &Queue::boostPythonPop,

59\ "Returning reference to something that was previously pushed.\n"

60\ "Relies on existence of the object\n",

61\ return_internal_reference<>()

62\ )

63\ .def("push",&Queue::push)

64\ ;

65\ class_<Node,boost::noncopyable>("Node")

66\ .def("__str__",&nodeToString)

67\ .def("remove",&Node::remove)

68\ ;

69\ def("makeNode",&makeNode,return_value_policy<manage_new_object>());

70\ }

71\

72\}}} //namespace ots,scheduler




We may experiment with the created classes with the following Python code.

1\import test

2\queue=test.Queue()

3\

4\def push(name,times):

5\ strings=[]

6\ data=[]

7\ nodes=[]

8\ for x in range(0,times) :

9\ s=name+x.__str__()

10\ strings.append(s)

11\ d=test.Wrapper(s)

12\ data.append(d)

13\ n=test.makeNode(d)

14\ nodes.append(n)

15\ queue.push(n)

16\ recentPushResult=(strings,data,nodes)

17\ return recentPushResult

18\

19\def pop(times):

20\ res=[]

21\ for x in range(0,times) :

22\ res.append(queue.pop())

23\ return res

24\

Line 1. We assume here that our C++ library is placed in a DLL-formatted file "test.pyd".

Lines 4-17. We create several strings, Wrappers and Nodes. We push them into queue and retain a pointer to all the data. Recall that NonBlockingQueue takes no ownership of the data that it receives. If we do not retain a pointer then the data disappears and we get undefined behavior.

Lines 19-23. Getting the data from the queue.

We place the above code into the file "test1.py". The following Python session illustrates the NonBlockingQueue behavior. We assume that the ipython interpreter is started from "NonBlockingQueueTest_nmake_win/release_gcc" directory of the installation.

In [1]: execfile('../../test1.py')

In [2]: ref=push('abc',10)

In [3]: print queue

Queue

theSwitch=10

Element

Node( abc7 )

Node( abc3 )

End of Element

Element

Node( abc8 )

Node( abc4 )

Node( abc0 )

End of Element

Element

Node( abc9 )

Node( abc5 )

Node( abc1 )

End of Element

Element

Node( abc6 )

Node( abc2 )

End of Element

End of Queue

In [4]: ref2=pop(5)

In [5]: [x.__str__() for x in ref2]

Out[5]:

['Node( abc6\x00 )',

'Node( abc7\x00 )',

'Node( abc8\x00 )',

'Node( abc9\x00 )',

'Node( abc2\x00 )']

In [6]: print queue

Queue

theSwitch=15

Element

Node( abc3 )

End of Element

Element

Node( abc4 )

Node( abc0 )

End of Element

Element

Node( abc5 )

Node( abc1 )

End of Element

Element

End of Element

End of Queue





Downloads. Index. Contents.


















Copyright 2007