walter-logo

Hi, this is AINOYA(@ainoya) from AP Solution Group. I developed a simple tool named Walter with ATL which helps to execute Build Pipeline in CI and CD.

My motivation of the development: the Build Pipe lines setting depended heavily on the Jenkins plugin.

If we perform CI with Jenkins, we make a set of Build Pipelines connected with several jobs in general. Jenkins has a plugin convenient for forming Build Pipelines, which operate executive control in fail and concurrency control easily.

When we operate the CI with those plugins in real, however, some points almost came out perfectly;

  • We cannot check the whole executive flow of pipelines only on Jenkins.
  • We cannot check activities until when we really kick Jenkins jobs.
  • We find it difficult to migrate settings. We cannot easily export the Build Pipeline settings with main use of the GUI to the other Jenkins one.

    • Although, with Docker and Ansible, we can manage the server environment as we aggregate source code, we have to depend on Jenkins when we build the setting with Jenkins.

    Overview of Walter

    The tool Walter allows you to do the following things;

  • Build Pipeline writing in setting files: easy handling of setting files in the same way to manage code bases such as the ones of the Ansible playbook and Travis-ci (e.g. Travis.yml).
  • Possible to execute Build Pipeline even in a local environment: you can execute Build Pipeline without a CI server.
  • Simple use: you can form Build Pipeline just with simple writing even if you do not understand the complicated GUI and original DSL

    • The Walter activity is simple, and it executes jobs in order along with executive rule which you write in your original settings file. The way of writing is similar to the one in Travis-ci setting file (Travis.yml), but in Walter you can easily write concurrency executive processing.

    Use case

    In Walter, you write the executive order and processing context of Build Pipeline under Pipeline:, and you write the details of each job in each property defined in Stage.

    The Stage where you write the setting on each job has the following properties;

  • Stage_name: name of the stage
  • Stage_type: an writing format of Stage. If you set it as Command, in the Command:property what you write is executed. (Initial stage_type is set as Command if you skip here.)
  • Command: description of system command executing jobs.
  • Run_after: write the Stage you want to execute after the setting Stage finished executing.

    • Some sample setting files (pipeline.yml) of Walter are demonstrated below.

    We can illustrate the steps of the Build Pipeline defined in Pipeline.yml above.　As you see in the illustration, the Stage defined in the top level of Pipeline is executed in order after it waits for each job to finish. On the other hand, the job written in run_after is executed in parallel.

    walter-pipeline-sample

    We give you other examples such as providing an application with building a Docker cantina. Please refer to them.

    Try Walter.

    You can execute Walter by downloading the Walter-built binary files distributed in GitHub. Let’s execute Walter in the mac environment.

    1. Obtain the binary files which is built specifically for Mac.

    2. Obtain the Sample Build Pipeline setting file.

    3. Execute Walter and select the Build Pipeline defined file for execution in –c option. If you do not make any selection, the default value is ./pipeline.yml.

    pipeline-execution

    Was the stage executed in the order you selected in Pipeline? Make sure that the behavior has changed if you changed the Stage writing and executing order. If you find any bugs, please report to issues.

    Microservices with walter

    In some architecture composed of several services/servers, there are some restrictions in the deployed order if there is a relationship depending on each service.

    It often happens in the case that the monolithic service becomes separated and to be operated as several services.　In this case too, when you apply the way to use Walter explained above, you can easily write the Build Pipeline of the architecture composed of several services/servers.　You can also apply to the cases where you deploy the two kinds of services which should be ready first and then you can deploy the rest of the 4 services which depend on the first two.

    walter-msa

    Wait_for writing and parallel executing option makes it possible to provision several servers with Ansible only. I feel personally it is easier to be handled if you execute Ansible with appropriate granularity. I aim to have a rich visibility of the whole composition through appropriately dividing the executing units provisioned by Ansible, joining them by Walter, and getting control of them.

    Current and future development of the schedule

    I wrote the Walter tool so that it solved the inconveniences to the ways of composing the Build Pipeline as-is. I am going to make its quality better mainly of the error processing and the retry mechanism, although the code quality is still a prototype. I will give a feedback through the improvements on production in company, of course. To make it more convenient, I will approach the variety of functionalities as follows;

  • Enhancement of message-passing functionality between jobs: I use Goroutine for implementing the Pipeline, with which I will enhance the execution data relationships between each job.
  • Ways of writings with easy comprehension: I will stop the way of the complicated writing in Yaml. I will make the ways of Build Pipeline writings better and more easily understood (I do not want to make them DSL, but I will trade-off the DSL and the better ways of writing.
  • Enhancement of affinity with ChatOPs:: I will make an enhancement of support to Slack and more delicate notification dynamic link functionality.
  • Dynamic link with Jenkins: I will think about a mechanism to feedback the job progress executing Build Pipeline into Jenkins .

    • Please leave me a comment about how you feel when using the code in GitHub opened to the public.