1.5. First own plugin¶

In this chapter we will create our first plugin, register a command, work with csv files and create a thread.

But before we start, let’s take a look what kind of application and plugins we want to build.

1.5.1. Example Application¶

This tutorial uses one single example application, which gets extend by each chapter.

The overall use case of the example application is the monitoring of changes on comma separated files (CSV files).

CSV files represent a table of information and can be read and saved by office tools like Microsoft Excel, Libreoffice Calc an others. They are often used in automation processes and in a lot of cases they are created and updated by scripts.

We want to be able to get notified, if a monitored CSV file gets changed and we also want to know the changes. In later chapters we will also store these changes in a database, so that we know the history of a CSV file. And we will also add some kind of a web interface to watch files and history in the browser.

1.5.1.1. Sequence diagram¶

The following sequence diagram shows how user, application and plugin interacts with each other and what kind of tasks must be done during which step.

But don’t worry, we will implement these features step by step.

1.5.2. Preparation¶

Before we can start coding, we have to create a place for our plugin.

Create a folder csv_watcher_plugin inside the folder CSV-Manager/csv_manager/plugins/.

Inside the newly created folder create two new files: __init__.py and csv_watcher_plugin.py.

At the end your CSV-Manager should look like this:

CSV-Manager/
├── csv_manager
│   ├── applications
│   │   ├── configuration.py
│   │   ├── csv_manager_app.py
│   │   └── __init__.py
│   ├── patterns
│   │   └── __init__.py
│   ├── plugins
│   │   ├── csv_watcher_plugin  # We added this
│   │   │   ├── csv_watcher_plugin.py  # We added this
│   │   │   └── __init__.py  # We added this
│   │   ├── csv_manager_plugin.py
│   │   └── __init__.py
│   └─ ...
└─ ...

Warning

Be sure your __init__.py is empty and you haven’t copy and pasted an __init__.py file from another location, which has some code in it!

1.5.3. Birth of a plugin¶

Now we can start to write our first plugin. Let’s start with an empty plugin and add needed functions step by step.

Open csv_watcher_plugin.py and add the following:

from groundwork.patterns import GwBasePattern


class CsvWatcherPlugin(GwBasePattern):
    """
    A plugin for monitoring csv files.
    """
    def __init__(self, app, **kwargs):
        self.name = "CsvWatcherPlugin"
        super().__init__(app, **kwargs)

    def activate(self):
        pass

    def deactivate(self):
        pass

Each plugin and pattern must inherit from GwBasePattern. If you inherit from another pattern, GwBasePattern gets automatically loaded, because the other pattern already inherits from it [1+4].

There are 4 rules, each plugin must follow:

A name is available in self.name [9]

The __init__() routine of patterns gets called [10]

An activate() routine exists [12]

A deactivate() routine exists [15]

Rule 1,3 and 4 were checked by GwBasePattern during initialisation. If they are not passed, your plugin can not be used.

A missing call of the __init__() routine of patterns (rule 2) may lead to not correct initialised patterns. For example a database pattern has not initiated its database, because its __init__() routine was not called.

1.5.3.1. Plugin registration¶

Right now, your plugin exists only in a code file and groundwork does not know it.

But we want be able to load our plugins inside a groundwork app by just adding its name into the related configuration parameter.

So we have to tell groundwork or better the used Python environment that there is a new plugin. This can only be done during installation of the CSV-Manager package. And the installation gets its information from our setup.py file.

So open CSV-Manager/setup.py and take a look on the parameter entry_points:

entry_points={
    'console_scripts': ["csv_manager = "
                        "csv_manager.applications.csv_manager_app:start_app"],
    'groundwork.plugin': ["csv_manager_plugin = "
                          "csv_manager.plugins.csv_manager_plugin:"
                          "csv_manager_plugin"],
}

entry_points is a python dictionary, which has 2 keys: console_scripts and groundwork.plugin.

console_scripts are used to register commands for the command line. In this case, it allows us to use csv_manager as command instead of calling the needed python file with python csv_manager/applications/csv_manager_app.

groundwork.plugins is the place where all the magic happens. It is a python list and there we need to add our plugin.

An entry of this list is a normal string, which must follow the syntax <name> = <packages>:<plugin_class>. <name> is not used by groundwork and can be everything.

The needed entry for our plugin is csv_watcher_plugin = csv_manager.plugins.csv_watcher_plugin.csv_watcher_plugin:CsvWatcherPlugin.

Let’s add it to our setup.py:

entry_points={
    'console_scripts': ["csv_manager = "
                        "csv_manager.applications.csv_manager_app:start_app"],
    'groundwork.plugin': ["csv_manager_plugin = "
                          "csv_manager.plugins.csv_manager_plugin:"
                          "csv_manager_plugin",  # Do not forget the "," here!
                          "csv_watcher_plugin = "
                          "csv_manager.plugins.csv_watcher_plugin.csv_watcher_plugin:CsvWatcherPlugin"
                          ],
}

After saving setup.py we have to reinstall the CSV-Manager::

>>> cd CSV-Manager
>>> pip install -e .

And we also need to add our plugin to the application configuration, so that it gets activated. Open CSV-Manager/csv_manager/applications/configurations.py and change the PLUGINS = [...] line to

PLUGINS = ["csv_manager_plugin", "GwPluginsInfo", "CsvWatcherPlugin"]

1.5.3.2. First run¶

Ok, that’s it. Let’s see if our plugin really gets activated:

>>> csv_manager
2017-01-14 13:08:14,221 - INFO  - Application signals initialised
2017-01-14 13:08:14,407 - INFO  - Application commands initialised
2017-01-14 13:08:14,407 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 13:08:14,408 - INFO  - Application documents initialised
2017-01-14 13:08:14,408 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 13:08:14,409 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 13:08:14,409 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  hello_world  Prints hello world
  plugin_list  List all plugins

Line 7 shows us, that the plugin was found and initiated. And in line 8 we see that it also got activated.

1.5.4. Using commands¶

Now we can start to bring some functionality to our plugin.

At first we should make sure that our functions can be called by the user. For this we need to register a command.

To give our plugin access to command registration, we only need to make sure that our plugin class inherits from GwCommandsPattern:

 from groundwork.patterns import GwCommandsPattern


 class CsvWatcherPlugin(GwCommandsPattern):
     """
     A plugin for monitoring csv files.
     """
     def __init__(self, app, **kwargs):
         self.name = "CsvWatcherPlugin"
         super().__init__(app, **kwargs)

     def activate(self):
         pass

     def deactivate(self):
         pass

We have changed line 1 and 3 to use GwCommandsPattern instead of GwBasePattern

The registration of commands should be done inside the activation() routine to make sure that our commands are only available, if our plugin really got activated:

 from groundwork.patterns import GwCommandsPattern


 class CsvWatcherPlugin(GwCommandsPattern):
     """
     A plugin for monitoring csv files.
     """
     def __init__(self, app, **kwargs):
         self.name = "CsvWatcherPlugin"
         super().__init__(app, **kwargs)

     def activate(self):
         self.commands.register("csv_watch",
                                "Monitors csv files",
                                self.csv_watcher_command)

     def csv_watcher_command(self):
         self.log.info("watcher command called")

     def deactivate(self):
         pass

And again a test:

>>> csv_manager
2017-01-14 15:29:32,742 - INFO  - Application signals initialised
2017-01-14 15:29:32,956 - INFO  - Application commands initialised
2017-01-14 15:29:32,957 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 15:29:32,957 - INFO  - Application documents initialised
2017-01-14 15:29:32,958 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 15:29:32,958 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:29:32,959 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  csv_watch    Monitors csv files  # <--- That's our command.
  hello_world  Prints hello world
  plugin_list  List all plugins

>>> csv_manager csv_watch
2017-01-14 15:30:47,952 - INFO  - Application signals initialised
2017-01-14 15:30:48,134 - INFO  - Application commands initialised
2017-01-14 15:30:48,135 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 15:30:48,135 - INFO  - Application documents initialised
2017-01-14 15:30:48,136 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 15:30:48,137 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:30:48,137 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 15:30:48,137 - INFO  - watcher command called  # <--- That's our output

1.5.4.1. Adding a command argument¶

Our command gets called, but we still need the information which file our plugin must monitor. Let’s add an argument to our command:

from click import  Argument
from groundwork.patterns import GwCommandsPattern


class CsvWatcherPlugin(GwCommandsPattern):
    """
    A plugin for monitoring csv files.
    """
    def __init__(self, app, **kwargs):
        self.name = "CsvWatcherPlugin"
        super().__init__(app, **kwargs)

    def activate(self):

        # Argument for our command, which stores the csv file path.
        path_argument = Argument(("csv_file",),
                                 required=True,
                                 type=str)

        self.commands.register("csv_watch",
                               "Monitors csv files",
                               self.csv_watcher_command,
                               params=[path_argument])

    def csv_watcher_command(self, csv_file):
        self.log.info("watcher command called with csv_path: %s" % csv_file)

    def deactivate(self):
        pass

groundwork uses the library click for handling the command line interface. Therefore arguments are defined by using the Argument class from click [1].

In line 16 - 18 we define our argument. It gets a name and is marked as required. We also define the type, so that we can be sure that our function always gets a string.

The self.commands.register() function has a parameters called params, which takes a list of arguments and options [23].

We have also updated our command function to accept the argument as function parameter [25]. The given csv file location will also be used inside our log message [26].

Time for a small test:

>>> csv_manager csv_watch test.csv
2017-01-14 15:51:40,617 - INFO  - Application signals initialised
2017-01-14 15:51:40,820 - INFO  - Application commands initialised
2017-01-14 15:51:40,821 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 15:51:40,821 - INFO  - Application documents initialised
2017-01-14 15:51:40,821 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 15:51:40,822 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 15:51:40,823 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 15:51:40,823 - INFO  - watcher command called with csv_path: test.csv  # <-- It works!

1.5.5. Handling csv files¶

It’s time to start coding the csv handling part of our plugin. At first we need a way to read the content of a csv file. Luckily Python provides a built-in solution for this: The csv module

But before we start, we need a test.csv file. Create one in the CSV-Manager folder and add the following content:

name,city,phone
Daniel,Munich,123-456
Maria,Cologne,111/222
Richard,Paris,0445-4545-4545
Anabel,London,-

We also need the built-in python libraries csv for csv handling and os for checks regarding file existence. So let’s import them by adding the following lines at the beginning of our file csv_watcher_plugin.py:

import os
import csv

Now we can write the part for reading the csv file:

 def csv_watcher_command(self, csv_file):
     self.log.info("watcher command called with csv_path: %s" % csv_file)

     # Check if the given csv_file really exists
     if not os.path.exists(csv_file):
         self.log.error("CSV file %s does not exist" % csv_file)

     with open(csv_file) as csv_file_object:
         reader = csv.DictReader(csv_file_object)
         for row in reader:
             self.log.info(row)

Ok that wasn’t much code and all the magic is done in the lines 8-11. As you can see, we read the csv file and log every single line. Nothing more yet.

And again the test on the command line:

>>> csv_manager csv_watch test.csv
2017-01-14 16:32:35,179 - INFO  - Application signals initialised
2017-01-14 16:32:35,361 - INFO  - Application commands initialised
2017-01-14 16:32:35,362 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 16:32:35,362 - INFO  - Application documents initialised
2017-01-14 16:32:35,363 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 16:32:35,364 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 16:32:35,364 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 16:32:35,364 - INFO  - watcher command called with csv_path: test.csv
2017-01-14 16:32:35,365 - INFO  - {'name': 'Daniel', 'city': 'Munich', 'phone': '123-456'}
2017-01-14 16:32:35,365 - INFO  - {'name': 'Maria', 'city': 'Cologne', 'phone': '111/222'}
2017-01-14 16:32:35,365 - INFO  - {'name': 'Richard', 'city': 'Paris', 'phone': '0445-4545-4545'}
2017-01-14 16:32:35,365 - INFO  - {'name': 'Anabel', 'city': 'London', 'phone': '-'}

As you can see, each row is a dictionary and the first row of our csv-file was selected to hold the needed dictionary key names.

1.5.5.1. Making csv diffs¶

Our application shall monitor csv files for changes. Therefore it must periodically read a file and compare its current content with an old, stored content. Let’s add this to our csv_watcher_command function:

def csv_watcher_command(self, csv_file):
    self.log.info("watcher command called with csv_path: %s" % csv_file)

    # Check if the given csv_file really exists
    if not os.path.exists(csv_file):
        self.log.error("CSV file %s does not exist" % csv_file)

    # Start with an "empty csv file"
    old_content = []

    while True:
        csv_file_object = open(csv_file)
        new_content = list(csv.DictReader(csv_file_object))

        if new_content != old_content:
            self.log.info("Change detected")

            # Check if there are new/changed rows
            for row in new_content:
                if row not in old_content:
                    self.log.info("New row: %s" % row)

            # Check if old rows are missing
            for row in old_content:
                if row not in new_content:
                    self.log.info("Missing row: %s" % row)

            # Store the current csv file content as old content
            old_content = new_content

        csv_file_object.close()

        # Wait 2 seconds
        time.sleep(2)

To allow the easiest way of comparision, the csv content is transformed to a python list, where each list element represents a single row [13]. After that we compare the old and the new list [15].

The code detects changes per row. If one single value has changed, the whole row is detected as “New row” [19-21]. And the old row is detected as “Missing row” [24-26].

Let’s see an output example:

>>> csv_manager csv_watch test.csv
2017-01-14 17:20:17,477 - INFO  - Application signals initialised
2017-01-14 17:20:17,660 - INFO  - Application commands initialised
2017-01-14 17:20:17,660 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 17:20:17,661 - INFO  - Application documents initialised
2017-01-14 17:20:17,661 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 17:20:17,662 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 17:20:17,662 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
2017-01-14 17:20:17,663 - INFO  - watcher command called with csv_path: test.csv
2017-01-14 17:20:17,663 - INFO  - Change detected
2017-01-14 17:20:17,663 - INFO  - New row: {'city': 'Munich', 'name': 'Daniel', 'phone': '123-456'}
2017-01-14 17:20:17,663 - INFO  - New row: {'city': 'Cologne', 'name': 'Maria', 'phone': '111/222'}
2017-01-14 17:20:17,663 - INFO  - New row: {'city': 'Paris', 'name': 'Richard', 'phone': '0445-4545-4545'}
2017-01-14 17:20:17,663 - INFO  - New row: {'city': 'London', 'name': 'Anabel', 'phone': '-'}
2017-01-14 17:20:33,682 - INFO  - Change detected
2017-01-14 17:20:33,682 - INFO  - New row: {'city': 'London', 'name': 'Anabel', 'phone': '777-888888'}
2017-01-14 17:20:33,683 - INFO  - Missing row: {'city': 'London', 'name': 'Anabel', 'phone': '-'}
^C
Aborted!

At the beginning the whole csv file content is detected as change, because we compared its content to an empty list. In groundwork and databases we will fix this behavior by using a database to store old csv content.

Another problem is that we are using an infinite loop to check our csv-file. See lines 11 and 34 in the code example above (while True ...). Therefore we have to hardly stop our application by pressing Ctrl + C. Also other code from plugins is blocked, too. So in a complex application nothing would work anymore except our csv watcher code.

Let’s fix this by using a thread.

1.5.6. Working with threads¶

Threads can be used on a computer to execute something in parallel to the current execution. They are an ideal solution for long running tasks like our csv watcher.

groundwork makes the usage of threads very easy. All we need is the GwThreadPattern and a python function, which shall be executed in the new thread.

To see the whole picture, here is the complete code of our plugin CsvWatcherPlugin:

 import os
 from click import  Argument
 import csv
 import time
 from groundwork.patterns import GwCommandsPattern, GwThreadsPattern


 class CsvWatcherPlugin(GwCommandsPattern, GwThreadsPattern):
     """
     A plugin for monitoring csv files.
     """
     def __init__(self, app, **kwargs):
         self.name = "CsvWatcherPlugin"
         super().__init__(app, **kwargs)
         self.csv_file = None

     def activate(self):

         # Argument for our command, which stores the csv file path.
         path_argument = Argument(("csv_file",),
                                  required=True,
                                  type=str)

         self.commands.register("csv_watch",
                                "Monitors csv files",
                                self.csv_watcher_command,
                                params=[path_argument])

     def csv_watcher_command(self, csv_file):
         # Register thread
         self.csv_thread = plugin.threads.register("csv_thread_%s" % csv_file, self._csv_watcher_thread,
                                                   "Thread for monitoring a csv file in background")
         # Start thread
         csv_thread.run()

     def csv_watcher_thread(self, plugin):
         csv_file = plugin.csv_file
         self.log.info("watcher command called with csv_path: %s" % csv_file)

         # Check if the given csv_file really exists
         if not os.path.exists(csv_file):
             self.log.error("CSV file %s does not exist" % csv_file)

         # Start with an "empty csv file"
         old_content = []

         while True:
             csv_file_object = open(csv_file)
             new_content = list(csv.DictReader(csv_file_object))

             if new_content != old_content:
                 self.log.info("Change detected")

                 # Check if there are new/changed rows
                 for row in new_content:
                     if row not in old_content:
                         self.log.info("New row: %s" % row)

                 # Check if old rows are missing
                 for row in old_content:
                     if row not in new_content:
                         self.log.info("Missing row: %s" % row)

                 # Store the current csv file content as old content
                 old_content = new_content

             csv_file_object.close()

             # Wait 2 seconds
             time.sleep(2)

     def deactivate(self):
         pass

We have created a new function csv_watcher_thread() and moved all code from csv_watcher_command() to it (see lines 29-72).

csv_watcher_command() is now responsible for registering and starting the thread [32-36]. It also stores the received csv_file to the plugin class itself, so that the thread has access to it [39].

The thread function csv_watcher_thread() gets a plugin instance as second parameter when the function is called by the thread-handler. This plugin instance is the instance which has registered the thread. As our plugin class has done this, the thread function has now access to all plugin variables and functions. So also to csv_file [39].

We still have an infinite loop in the thread. Therefore we still must exit our application with Ctrl + c. But compared to the old version, our watcher is not blocking the rest of our application anymore.

1.5.7. Add interval option¶

Currently our application checks the file every 2 seconds. This value is hard coded and that’s not a good idea. We should allow the user to define the interval. So let’s change the code a little bit and add an optional command option.

 from click import Argument, Option

 ...

     def activate(self):

     # Argument for our command, which stores the csv file path.
     path_argument = Argument(("csv_file",),
                              required=True,
                              type=str)

     interval_option = Option(("-i", "--interval"),
                              type=int,
                              default=10,
                              help="Sets the time between two checks in seconds")

     self.commands.register("csv_watch",
                            "Monitors csv files",
                            self.csv_watcher_command,
                            params=[path_argument, interval_option])

     def csv_watcher_command(self, csv_file, interval=10):
         ...

     def csv_watcher_thread(self, plugin):
         ...
         # Wait x seconds
         time.sleep(plugin.csv_interval)

Let’s check, if the new option is mentioned in the help text for the command:

>>> csv_manager csv_watch --help
2017-01-14 18:49:03,390 - INFO  - Application signals initialised
2017-01-14 18:49:03,595 - INFO  - Application commands initialised
2017-01-14 18:49:03,595 - INFO  - Plugins initialised: csv_manager_plugin
2017-01-14 18:49:03,595 - INFO  - Application documents initialised
2017-01-14 18:49:03,596 - INFO  - Plugins initialised: GwPluginsInfo
2017-01-14 18:49:03,596 - INFO  - Application threads initialised
2017-01-14 18:49:03,597 - INFO  - Plugins initialised: CsvWatcherPlugin
2017-01-14 18:49:03,597 - INFO  - Plugins activated: csv_manager_plugin, GwPluginsInfo, CsvWatcherPlugin
Usage: csv_manager csv_watch [OPTIONS] CSV_FILE

  Monitors csv files

Options:
  -i, --interval INTEGER  Sets the time between two checks in seconds
  --help                  Show this message and exit.

That’s it, We have created an awesome plugin, which can be started and configured via a command on the command line.

On the next chapter First own pattern we take a look into patterns and make our csv-watcher code reusable for other plugins.