diff --git a/doc/source/index.rst b/doc/source/index.rst index fd852ee0a..405d808d9 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -162,13 +162,14 @@ Reviewing reviewer_guide -Running benchmark -================= +Tutorials +========= .. toctree:: :maxdepth: 1 running_benchmark + writing_pipeline_stages Other resources =============== diff --git a/doc/source/writing_pipeline_stages.rst b/doc/source/writing_pipeline_stages.rst new file mode 100644 index 000000000..7b3fdf56c --- /dev/null +++ b/doc/source/writing_pipeline_stages.rst @@ -0,0 +1,225 @@ +======================================== +Writing stages for the storage pipelines +======================================== + +Introduction +~~~~~~~~~~~~ + +A pipeline is a set of stages needed to process a request. When a new request +comes to Zaqar, first the message goes through the transport layer pipeline and +then through one of the storage layer pipelines depending on the type of +operation of each particular request. For example, if Zaqar receives a +request to make a queue-related operation, the storage layer pipeline will be +``queue pipeline``. Zaqar always has the actual storage controller as the +final storage layer pipeline stage. + +By setting the options in the ``[storage]`` section of ``zaqar.conf`` +you can add additional stages to these storage layer pipelines: + +* **Claim pipeline** +* **Message pipeline** with built-in stage available to use: + + * ``zaqar.notification.notifier`` - sends notifications to the queue + subscribers on each incoming message to the queue, i.e. enables + notifications functionality. +* **Queue pipeline** +* **Subscription pipeline** + +The storage layer pipelines options are empty by default, because additional +stages can affect the performance of Zaqar. Depending on the stages, the +sequence in which the option values are listed does matter or not. + +You can add your own external stages to the storage layer pipelines. + +Things to know before writing the stage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Stages in the pipeline must implement storage controller methods they need +to hook. You can find all available to hook methods in the abstract classes in +``zaqar/storage/base.py``. For example, if you're looking for all methods +available to hook for the queue storage layer pipeline, see ``Queue`` +class in ``zaqar/storage/base.py``. As you can see, Zaqar's built-in stage +``zaqar.notification.notifier`` implements ``post`` method of +``zaqar.storage.base.Message`` abstract class. + +A stage can halt the pipeline immediate by returning a value that is not +None; otherwise, processing will continue to the next stage, ending with the +actual storage controller. + +.. warning:: + + For the most of the cases it does not matter what non-None value the storage + pipeline returns, but sometimes the returned value is used by the transport + layer and you have to be careful. For example, during queue creation + request, if the storage driver returns ``True``, the transport layer + responds to the client with the ``201`` http response code, if ``False``, + it responds with ``204`` http response code. See: + ``zaqar.transport.wsgi.v2_0.queues.ItemResource#on_put``. + +Zaqar finds stages with their source codes through the Python entry points +mechanism. All Python packages containing stages for Zaqar must register +their stages under ``zaqar.storage.stages`` entry point group during their +install either by ``setup.py`` or by ``setup.cfg``. If the stage is registered, +and the name of the stage's entry point is specified by the user in the one of +``zaqar.conf`` storage layer pipeline options, the stage will be loaded to +the particular storage layer pipeline. Zaqar imports stages as plugins. See +``zaqar.storage.pipeline#_get_storage_pipeline``. + +For additional information about plugins see: `Stevedore - Creating Plugins`_ +and `Stevedore - Loading the Plugins`_. + +Example of external stage (written outside Zaqar package) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is an example of small package with a stage that can process queue-related +requests in Zaqar. The stage does not do anything useful, but is good as +example. + +File tree structure of the package: + +.. code-block:: none + + . + ├── setup.py + └── ubershystages + ├── __init__.py + └── queues + ├── __init__.py + └── lovely.py + + 2 directories, 4 files + +``setup.py``: + +.. code-block:: python + + from setuptools import setup, find_packages + + setup( + name='ubershystages', + version='1.0', + + description='Demonstration package for Zaqar with plugin pipeline stage', + + author='Ubershy', + author_email='ubershy@gmail.com', + + url='', + + classifiers=['Development Status :: 3 - Alpha', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.4', + 'Intended Audience :: Developers', + 'Environment :: Console', + ], + + platforms=['Any'], + + scripts=[], + + packages=find_packages(), + include_package_data=True, + + entry_points={ + 'zaqar.storage.stages': [ + 'ubershy.lovelyplugin = ubershystages.queues.lovely:LovelyStage', + ], + }, + + zip_safe=False, + ) + +``lovely.py``: + +.. code-block:: python + + class LovelyStage(object): + """This stage: + 1. Prints 'Lovely stage is processing request...' on each queue creation or + deletion request. + 2. Prints 'Oh, what a lovely day!' on each creation request of a queue + named 'lovely'. + 3. Prevents deletion of a queue named 'lovely' and prints 'Secretly keeping + lovely queue' on such attempt. + """ + + def __init__(self, *args, **kwargs): + print("Lovely stage is loaded!") + + def create(self, name, metadata=None, project=None): + """Stage's method which processes queue creation request. + + :param name: The queue name + :param project: Project id + """ + + self.printprocessing() + if name == 'lovely': + print("Oh, what a lovely day!") + + def delete(self, name, project=None): + """Stage's method which processes queue deletion request. + + :param name: The queue name + :param project: Project id + :returns: Something non-None, if the queue has a name 'lovely'. It will + stop further processing through the other stages of the pipeline, and + the request will not reach the storage controller driver, preventing + queue deletion from the database. + """ + + self.printprocessing() + if name == 'lovely': + print('Secretly keeping lovely queue') + something = "shhh... it's a bad practice" + return something + + def printprocessing(self): + print('Lovely stage is processing request...') + +To install the package to the system in the root directory of the package run: + +.. code-block:: console + + # pip install -e . + +In ``zaqar.conf`` add ``ubershy.lovelyplugin`` to the ``queue_pipeline`` +option: + +.. code-block:: ini + + [storage] + queue_pipeline = ubershy.lovelyplugin + +Start Zaqar: + +.. code-block:: console + + $ zaqar-server + +If the stage has successfully loaded to Zaqar you will see amongst terminal +output lines the ``Lovely stage is loaded!`` line. Then you can try to perform +queue create and queue delete operations with the queue 'lovely' and see what +will happen in Zaqar's database. + +.. note:: + + You can hold multiple stages in one package, just be sure that all stages + will be registered as entry points. For example, in the ``setup.py`` you + can register additional ``ubershy.nastyplugin`` stage: + + .. code-block:: python + + entry_points={ + 'zaqar.storage.stages': [ + 'ubershy.lovelyplugin = ubershystages.queues.lovely:LovelyStage', + 'ubershy.nastyplugin = ubershystages.messages.nasty:NastyStage', + ], + }, + +.. _`Stevedore - Creating Plugins`: http://docs.openstack.org/developer/stevedore/tutorial/creating_plugins.html +.. _`Stevedore - Loading the Plugins`: http://docs.openstack.org/developer/stevedore/tutorial/loading.html