Copyright @ Lenovo US

    Have you ever wondered what layer-basic is for? and why every charm needs to include it? In this article we will take a look at its code base to decipher this mystery.

    Hooks

    We already know hooks are hardcoded. Juju expects certain hooks and hook sequence is always executed in an order that is dictated by Juju code.

    Sequence of charm hooks

    What do we see in hooks? If you run charm build and examine the dist folder, hooks live in dist/yourcharm/hooks folder:

    .
    ├── config-changed
    ├── hook.template
    ├── install
    ├── leader-elected
    ├── leader-settings-changed
    ├── relations
    │   └── rack-pdu
    │       ├── __init__.py
    │       ├── interface.yaml
    │       ├── provides.py
    │       └── requires.py
    ├── start
    ├── stop
    ├── update-status
    └── upgrade-charm
    

    We know we haven't defined any custom hook code for this charm, so how were they generated? Even more interesting is that all hooks are strangely identical. Take install hook script for example:

    #!/usr/bin/env python3
    
    # Load modules from $JUJU_CHARM_DIR/lib
    import sys
    
    from charms.layer import basic
    
    # This will load and run the appropriate @hook and other decorated
    # handlers from $JUJU_CHARM_DIR/reactive, $JUJU_CHARM_DIR/hooks/reactive,
    # and $JUJU_CHARM_DIR/hooks/relations.
    #
    # See https://jujucharms.com/docs/stable/authors-charm-building
    # for more information on this pattern.
    
    from charms.reactive import main
    sys.path.append('lib')
    basic.bootstrap_charm_deps()
    basic.init_config_states()
    main()
    

    You can diff all of them, and they are all the same! and they are all copies of the hook.template. The from charms.layer import basic actually clearly states that these hooks are depending on layer-basic. There, is why all charms are using it.

    This actually gives us a clue to further investigate how dependent charms are to layer-basic. It turned out not as necessary as we thought.

    charm build

    Source of charm build is here. Looking into the charm Builder class:

    class Builder(object):
        """
        Handle the processing of overrides, implements the policy of BuildConfig
        """
        HOOK_TEMPLATE_FILE = path('hooks/hook.template')
    

    Ah ha, that's where it is expecting hook.template. Following this we have discovered the followings:

    1. Relatioin hooks and storage hooks are dynamically generated. For this to work, at least one layer must provide hook.template. The interesting point is that it doesn't relie on layer-basic anymore. If you charm has a /hooks/hook.template, it will work.

      ```python
      def plan_interfaces(self, layers, output_files, plan):
        ......
        if not meta and layers.get('interfaces'):
            raise BuildError(
                'Includes interfaces but no metadata.yaml to bind them')
        elif self.HOOK_TEMPLATE_FILE not in output_files:
            raise BuildError('At least one layer must provide %s',
                             self.HOOK_TEMPLATE_FILE)
      ```
      
    2. If you didn't define those must-have hooks, eg. install hook, charm build will happily make the dist, but it will fail at run time. What is happening!? It turned out charm binary has an option to make proof, and this will complain if you miss expected hooks (bug #325), but the proof isn't part of the charm build process.

      1. Code expected hooks:

            ```python
            lint.check_hook('install', hooks_path, recommended=True)
            lint.check_hook('start', hooks_path, recommended=True)
            lint.check_hook('stop', hooks_path, recommended=True)
            if os.path.exists(os.path.join(charm_path, 'config.yaml')):
                lint.check_hook('config-changed', hooks_path, recommended=True)
            else:
                lint.check_hook('config-changed', hooks_path)
            ```
        
      2. charm proof will catch the missing hooks:

            ```shell
            fengxia@fengxia-xenial-dev:~/workspace/wss/charms/charm-pdu$ charm proof
            I: metadata name (pdu) must match directory name (charm-pdu) exactly for local deployment.
            W: no copyright file
            W: no README file
            I: relation rack has no hooks
            I: missing recommended hook install
            I: missing recommended hook start
            ```
        

    If charm is executed, install hook will run first, which then call two functions from layer-basic:

    1. basic.bootstrap_charm_deps()
    2. basic.init_config_states()

    Let's take a look them respectively.

    boostrap_charm_deps

    This function is to setup the host Python environment for charms.

    1. charm-pre-install: execute any nested file named charm-pre-install under an exec.d folder. It uses check_call so any script will work. Once the script has been executed without error, a hidden file .{}_{}.done'.format(module_name, submodule_name)) will be created so the same preinstall script will only run ONCE → this is the way to make the execution only once regardless the sequence of hooks. Therefore whoever runs it once will write this file as breadcrumb for others to check.

    2. Install packages in wheelhouse folder. Again, if this has run, a hidden file wheelhouse/.bootstrapped is created so all these packages are installed ONCE (reference: distutils, easy_install).

      In wheelhouse.txt file:

      ```shell
      pip>=7.0.0,<8.2.0
      charmhelpers>=0.4.0,<1.0.0
      charms.reactive>=0.1.0,<2.0.0
      ```
      
    3. Install python-virtualenv if it is included in config.yaml.

    init_config_states

    This is where charms will start using Juju commands (via charmhelpers lib) to set states with Juju controller. I'm copying the codes below since they are fairly self-explanatory. Unlike bootstrap_charm_deps, there is no magic file or flag to prevent this block executed multiple times. This makes sense since each hook can potentially modify charm states, thus run this in each hook is necessary.

    def init_config_states():
        import yaml
        from charmhelpers.core import hookenv
        from charms.reactive import set_state
        from charms.reactive import toggle_state
        config = hookenv.config()
        config_defaults = {}
        config_defs = {}
        config_yaml = os.path.join(hookenv.charm_dir(), 'config.yaml')
        if os.path.exists(config_yaml):
            with open(config_yaml) as fp:
                config_defs = yaml.safe_load(fp).get('options', {})
                config_defaults = {key: value.get('default')
                                   for key, value in config_defs.items()}
        for opt in config_defs.keys():
            if config.changed(opt):
                set_state('config.changed')
                set_state('config.changed.{}'.format(opt))
            toggle_state('config.set.{}'.format(opt), config.get(opt))
            toggle_state('config.default.{}'.format(opt),
                         config.get(opt) == config_defaults[opt])
        hookenv.atexit(clear_config_states)
    
    def clear_config_states():
        from charmhelpers.core import hookenv, unitdata
        from charms.reactive import remove_state
        config = hookenv.config()
        remove_state('config.changed')
        for opt in config.keys():
            remove_state('config.changed.{}'.format(opt))
            remove_state('config.set.{}'.format(opt))
            remove_state('config.default.{}'.format(opt))
        unitdata.kv().flush()
    

    Conclusion

    layer-basic is the foundation of charm building because it provides the entry point to call preinstall scripts, to setup the host Python environment and to initialize charm states. It has many hardcoded lines for using apt-get CLI and expecting an Ubuntu environment. These have been addressed somewhat in python2 charm. Going forward, the code base can use some work to support host other than Ubuntu and Python2 instead of 3.

    — by Feng Xia

    Related:

      2017-10-22
    Juju GUI nginx proxy

    In LXD on localhost we introduced using LXD container to bootstrap a Juju controller. But how to access the Juju GUI? Launching it is easy enough with $ juju gui from juju host; ...

      2017-09-06
    Juju local LXD

    Using Juju's LXD provider is the least-hassle way to start an experience of Juju and its charms. However, if you have done charm development for a while, you know making a one line of code ...

      2017-07-06
    Charm Ansible integration

    Let's face it. Ansible has the mouth (and market) share these days. For our modeling purpose, we are to utilize its procedural strength to carry out actions, which provides an abstraction instead of coding in charm's Python files.

    Design ...