In the first part we learned the basics of the DSL plugin features with detailed explanation of the building blocks to enable you to create simple pipelines.
The second part described the side effects of dynamic pipeline creation, the role of seed/boostrap job and finally demonstrated and described a simple pipeline project for an application. They were the basic part of DSL scripting and focuses on the pipeline as a whole.
Now we are entering to the intermediate level and focusing on particular problems of the pipeline creation like passing values via environment variables, using credentials to access protected resources, extending the DSL plugin functionality with the configure block to use non-covered plugin functions and so on.
Environment (related) variables
Environment variables are great for customization of our script's behaviour. We could inject stage/environment specific values like database connection strings or dynamically changing values like URL to sandbox/production grade services (artifact repositories for example). In a later part of the series I'll demonstrate a docker based Jenkins instance where I'm using environment variables to customize the standard image for my needs. I separate them two big groups by their role: the pipeline generation and the build related variables.
I put the Jenkins predefined variables to the first place. We could use them to tag our artifacts, get some path info about the workspace and the execution environment. At the link I provided you could get detailed information about them. I strongly recommend to avoid build number as unique variable for anything, because every time when you regenerate your pipeline it will be set to the default. Again and again. These variables are managed by Jenkins and we are only using them. Their scope is changing from build to server depending on the role.
The next big group is the custom environment variables as information holder for our scripts. Here you could communicate your repository url, branch, environment or stage or you could inject some endpoints or database related data. There are more ways to set them:
Last but not least we could use parameters for job run with default values. These parameters has build scope and they can change dynamically from build to build. Interesting fact if you use them for the boostrap/seed job, the scope is changing from build to pipeline.
In normal case you should click on "This build parameterized" checkbox you have multiple options to pass. In this simple example I'm adding a 'branch' parameter to the checkout job with default value, to make sure it could run both manually and automated.
Variables for pipeline generation
These variables are used by the bootstrap/seed script and they are controlling the final look of the pipeline. We could store here repository addresses, important endpoints, server urls etc. Their lifecycle - usually - ends after the pipeline generation and not participating directly in the builds. We could pass them as OS environment variable, system property or Jenkins Server setting.
Controlling the build
The build related variables are more dynamic. We could use them to decide what target environment could be used for releasing, we could switch between branches, database schemas on the fly to run a special build for us or release a specific version of the app to the manual testing environment. Typical usage is the build parameter customization.
Types of environment variables
I put the Jenkins predefined variables to the first place. We could use them to tag our artifacts, get some path info about the workspace and the execution environment. At the link I provided you could get detailed information about them. I strongly recommend to avoid build number as unique variable for anything, because every time when you regenerate your pipeline it will be set to the default. Again and again. These variables are managed by Jenkins and we are only using them. Their scope is changing from build to server depending on the role.
The next big group is the custom environment variables as information holder for our scripts. Here you could communicate your repository url, branch, environment or stage or you could inject some endpoints or database related data. There are more ways to set them:
- OS level with set/export commands (or using ENV entry in the Dockerfile)
- inject with -e Dockerized environment
- Adding -D
= to the Jenkins startup command
Last but not least we could use parameters for job run with default values. These parameters has build scope and they can change dynamically from build to build. Interesting fact if you use them for the boostrap/seed job, the scope is changing from build to pipeline.
In normal case you should click on "This build parameterized" checkbox you have multiple options to pass. In this simple example I'm adding a 'branch' parameter to the checkout job with default value, to make sure it could run both manually and automated.
Usage of environment variables
We have two ways to get their value. The properties passed by -D= during the startup we could read as System.getProperty("key") thanks to the Groovy's strong relation with Java.
Reading normal environment variables in Java way is the System.getenv("VARIABLE"), but the Groovy has a smarter way with normal placeholders. I suggest to read this tutorial to see the difference between the 'string' and "string" where the double quoted string is handled differently. Within the string you could refer to other groovy and environment variables with the simple ${key} format and also true for the """...""" formatted GStrings. Just use double quote and no sophisticated extraction needed in your code.
I hope you noticed in the code example I'm using single quote, because I don't want to interpret the placeholder during the bootstrap/seed job run, just right in the checkout Job, where the placeholder interpreted properly to a value.
Reading normal environment variables in Java way is the System.getenv("VARIABLE"), but the Groovy has a smarter way with normal placeholders. I suggest to read this tutorial to see the difference between the 'string' and "string" where the double quoted string is handled differently. Within the string you could refer to other groovy and environment variables with the simple ${key} format and also true for the """...""" formatted GStrings. Just use double quote and no sophisticated extraction needed in your code.
I hope you noticed in the code example I'm using single quote, because I don't want to interpret the placeholder during the bootstrap/seed job run, just right in the checkout Job, where the placeholder interpreted properly to a value.
Credentials, server IDs
Even in an internal system we could access some resources only via authentication. In Jenkins we could store them on server level globally to the jobs and we can refer to them via a label in our script. Adding credentials to Jenkins is easy via the Credentials menu, but you should make sure you clicked on Advanced button, to avoid automated credential ID generation. This ID should be used as reference in DSL configs. When the credential created previously and can't be changed you can extract the ID easily if you click on the credential details from the URL:
http:///credential-store/domain/_/credential/<credential_ID>
Usually you'll get something like this: 1b36b1e2-aedb-4601-898c-602d3545d947.
Here's an usage example of the newly created credential
http://
Usually you'll get something like this: 1b36b1e2-aedb-4601-898c-602d3545d947.
Here's an usage example of the newly created credential
Server IDs
Another big group of mysterious IDs. Personally I faced with this first time, when I tried to refer to our Artifactory server settings in my script and I scratched my head, how to find it.
Assume we are added an Artifactory to our Jenkins config in the Manage Jenkins/Configure System screen:
There's not ID, no url there. Nothing. After spending some time to find the ID, I had no better idea then go directly into the page's HTML source and find it manually. I looked for the server name and I had luck, because it was there, near to the "org.jfrog.hudson.ArtifactoryServer" item:
name="serverId" type="hidden" value="-1859036962@1437294997077"
It looks weird, but this is the ID you need to refer to the server. Later I found out in the JENKINS_HOME all data is stored in the org.jfrog.hudson.ArtifactoryBuilder.xml file, but this is a different story. Stupid me :)
Let's see an example with the newly found server ID:
During your script development you could face with same challenges to find settings, so I suggest to be brave and don't afraid to get your hands dirty with some HTML hacking or fire a grep -iR '
Adjust unknown plugins with configure block
The DSL plugin evolves quickly, but can't cover all plugins released for Jenkins and in a mature continuous delivery environment we are using wide selections of plugins to simplify our tasks. The XML config tricks are perfectly described on their wiki page and I strongly recommend to spend some time to read, but I'd like to add few thoughts to it.
Limitations
First of all, not all sections can be configured, so you are partially limited. You should scan for the word 'configure on the Job reference page to see where could you change the job's behavior and add some sprinkle of direct xml adjustments, but the hardest part is to figure out what to add and how...Finding blueprint
The xml configuration is an advanced topic and we can't do it without a blueprint that describes the plugin's possibilities. This is not so clearly described on the wiki page, so I'm sharing my method with you. To find the plugin's configuration options you should create a job manually to persist settings with Jenkins into a config.xml file on the server. This config.xml represents almost the whole job (there are some exceptions, when you need more).
So you have the file, how could you read it? There are two ways to dig deeper:
If you have direct access to your Jenkins you should go to the $JENKINS_HOME/jobs/ folder to find the config.xml of the job.
If possible do this experiments on your local machine for safe testing and try to add changes to the Jenkins server via scripts only.
Here's a docker build example:
The implemented and slightly customized DSL script:
So you have the file, how could you read it? There are two ways to dig deeper:
- browser based
- file based
Reading config.xml from the browser
You could find it quite easily, i'll show an example. Assume we have a job called 'demo-job' and we want to see the config.xml. On my localhost the URL to the job configuration is: http://localhost:8080/job/demo-job/configure. So let's replace the 'configure' with 'config.xml' and we have this URL: http://localhost:8080/job/demo-job/config.xml where we could see all the config.xml settings related to our job.
Reading config.xml from the Jenkins workspace area
If possible do this experiments on your local machine for safe testing and try to add changes to the Jenkins server via scripts only.
Here's a docker build example:
The config.xml:
The implemented and slightly customized DSL script:
As you could see the normal language elements can be used within the blocks, so the customization not limited only some static stuff, but complete loops or complicated decision mechanisms can be added to the code. This is a quite powerful option to use new features without DSL plugin coverage.
Modularization
As my scripts started to grow I was disturbed by the size and structure, so I started to break up the pipeline code to reusable components (steps, views, etc.). At the beginning I had no idea how to create job/view/etc closures within those classes, because I wasn't precise enough reading the plugin's wiki. In the last paragraph was the truth, but I missed it! I suggest to read the whole page :)
The point is to pass the "this" variable to your modules, because the seed/bootstrap script is holding the context as DslFactory (!!!) implementation and this is what you need to use DSL plugin closures. I can't repeat enough times how important to understand this mechanism and the passing of the context. Bootstrap/seed IS the context, your classes are USING the context.
The example on the page is a bit outdated, because the v1.30 deprecated the job creation without name. From v1.30 you should specify the job name as initialization:
The point is to pass the "this" variable to your modules, because the seed/bootstrap script is holding the context as DslFactory (!!!) implementation and this is what you need to use DSL plugin closures. I can't repeat enough times how important to understand this mechanism and the passing of the context. Bootstrap/seed IS the context, your classes are USING the context.
The example on the page is a bit outdated, because the v1.30 deprecated the job creation without name. From v1.30 you should specify the job name as initialization:
Refactoring the pipeline
In the previous article I demonstrated a simple pipeline code to checkout, build and deploy an application in a single file with poor reusage possibility. Now I refactored to a more modularized version with the recent learnings to follow DRY principle better. The code submitted to my Github area under integration-bootstrap. This is a simplified version of my framework used at my company and open sourced. I tried to demonstrate the usage of configure block, environment variables, credentials and all things I mentioned in the articles.
I hope it almost bug-free (or not...), but all feedback is welcome. Some TODOs left in the code (branching, etc..), but the purpose was to show a potential starting point for your own scripts.
I hope it almost bug-free (or not...), but all feedback is welcome. Some TODOs left in the code (branching, etc..), but the purpose was to show a potential starting point for your own scripts.
Parts of the framework
The framework built up from reusable parts with highly customizable format using builder pattern. I hope it makes easy for you to build your own pipeline generation code quickly.
Project
Project is the umbrella of your components it could hold global variables and settings (i.e. name prefix to group them together, etc..)
Environment
Environment is a potential customization per developer base. You could separate your pipeline from other's while you are working on your own fork (we are using Forking Workflow with Git).
Component
Component is the smallest unit of delivery. Mainly designed for our microservice architecture to keep them together under one Project, but enable customized flow per microservice. Of course you could create 1-1 connection between the Project and your Component in less granular environment.
Pipeline
Pipeline is aggregating Project-Environment-Component settings with the defined PipelineSteps tree and generates the jobs structure based on these settings.
PipelineSteps
PipelineSteps is the tree structure of your steps.
Steps
Steps are the building blocks of your pipeline (build, deploy, compile, test, etc.) and you can freely extend them with your own solutions.
tools.*
The tools package holds the helper objects to purge deprecated jobs, create views based on regex and so on. All generic stuff is going here.
Summary
Well done, you did it again :) This is the end of part 3 and now you are familiar with the customization options for your scripts. Also you could make it more modular with the DslFactory and there's no chance to get confused by any credential setting challenge from now. In the 4th part I'm showing the library creation for Jenkins DSL Plugin as an extraction of often used configuration blocks into a shared one. But this is not the end of my series, because I have a very long list of TODOs about Git workflows, Docker as build platform, Jenkins configurations and dockerizations and so on. I hope you will stay with me on this journey :)
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.