takahi_i from ATL here.
The build pipeline tool Walter was released almost exactly one year ago. Although it has been slow going since then, development is still ongoing. In this blog entry, I will explain how Walter has been improved since its release. These improvements can be classified into several types. I shall explain each of them in the following section.
- Use of environmental variables
- Support for properties that can be used in stages (directory, only_if)
- Added cleanup pipeline
- Reuse of already-defined stages
- Reuse of stage results
Use of Environmental Variables
Environmental variables can now be used on Water pipelines. Environmental variables are described in the pipeline settings file using $VAR. Environmental variables can be used by the settings during the implementation of Walter commands. For example, the following command outputs the details of the environmental variable GOPATH.
|
1 2 3 4 |
pipeline: - name: print GOPATH command: echo $GOPATH |
With the following settings, safety can be improved by describing information that you do not want to register to a repository, such as the keys required for HipChat communications, as an environmental variable.
|
1 2 3 4 5 6 |
messenger: type: hipchat2 room_id: $HIPCHAT_ROOM_ID token: $HIPCHAT_TWO_TOKEN from: $HIPCHAT_USER_NAME |
Support for Properties that Can Be Used in Stages (directory, only_if)
Walter adds lists of stages to the pipeline blog and describes their processing. Normally, only commands (command) and stage names (name) are specified in Walter’s stages. For example, in the stage added to the pipeline below, the name “show the files” and its command (ls -la) are specified.
|
1 2 3 4 |
pipeline: - name: show the files command: ls -la |
Specifying a property for use by a newly-added stage enables the command implementation conditions and implementation directory to be specified.
only_if
only_if implements the stage when the specified conditions are satisfied. In the following example, the command specified using “command” is implemented only when the value of the global variable WERCKER is set to TRUE.
|
1 2 3 4 |
pipeline: - name: Put SSH public key command: echo $PUBLIC_SSH_KEY > $HOME/.ssh/id_rsa.pub only_if: test "$WERCKER" = "true" |
directory
“directory” specifies the directory that implements the command. For example, the following Is stage displays a list of the /usr/local files.
|
1 2 3 4 5 |
pipeline: - name: show /usr/local files command: ls -la directory: /usr/local |
Added Cleanup Pipeline
The newly-added cleanup pipeline defines the processes. Specifically, cleanup is implemented after the process described in the pipeline has finished. The following pipeline block changes the input.data file to Apache Solr input format before registering the data. The cleanup pipeline operates after the pipeline has been implemented. In the following example, the data created for Solr inputs is deleted.
|
1 2 3 4 5 6 7 8 9 10 |
pipeline: - name: create data command: create_solr_data.rb -i input.data -o output/solr-data.xml - name: register data command: $SOLR_HOME/bin/post -host $SOLR_HOST -port $SOLR_PORT -c car-instance -o output/solr-data.xml cleanup: - name: cleanup xml data command: rm -f output/solr-data.xml |
Reuse of Already-Defined Stages
There have been calls wanting to reuse stages already defined in Walter. Mechanisms to import files that include stage definitions (stage definitions files) are now supported.
An example of a stage definitions file is shown below. The stage definitions file describes the namespace and stages. This was added so that names are not covered when multiple namespace definitions files are called.
The def block is created in the stages, and the stage defined. In the following example, the stage called whomai is defined.
|
1 2 3 4 5 6 7 |
namespace: mystages stages: - def: name: whoami command: whoami |
The stage definition file described above is saved using the name stages.yml. The require block imports the stage definitions file from the pipeline definitions file. The defined stage is called using call. The called stage is described using the namespace::stage name format.
|
1 2 3 4 5 6 |
require: - stages.yml pipeline: - call: mystages::whomai |
Reuse of Stage Results
This function was created in response to calls to be able to reuse the results of previous implementation stages. The results of previously implemented stages can be acquired using three special variables (__OUT, __ERR, __RESULT).
- __OUT: Stores the result (“true” or “false”) output to standard outputs.
- __ERR: Stores the results output to the standard error outputs.
- __RESULT: Stores the command return value.
The variables are used as maps. Assigning a stage name as the variable key acquires the stage results.
For example, in the following example the second stage uses the output results from the first stage. Specifically, the document is checked during the 1st stage, and then in the 2nd stage, whether or not an error was output to the standard error outputs of the 1st stage is checked. If an error was not output during the 2nd stage, a PDF is created.
|
1 2 3 4 5 6 7 |
pipeline: - name: Check document with RedPen command: redpen -c redpen-conf.md document.md -f markdown - name: Convert markdown to PDF command: markdown-pdf -o document.pdf document.md only_if: test -z "`echo __ERR["Check document with RedPen"] | grep ERROR`" |
Other Improvements
- Slack communications are now supported
- Communications with the local HipChat (proprietary) server are now supported
- Communications with Slack are now colored
Future Schedule
Ongoing development of Walter is scheduled for the future. In particular, this winter, Walter is scheduled to support server functions. We also want to enable the stage start conditions to be specifiable. Currently, when the previous stage finished, the next stage ends up starting immediately, but specifying the start conditions will enable settings such as wait until the necessary port is ready to be set.