Radek Sprta (Posts about DevOps)https://radeksprta.eu/categories/cat_devops.atom2024-01-07T02:28:54ZRadek SprtaNikolaLevel Up Your YAML: Tips For Advanced Usagehttps://radeksprta.eu/posts/level-up-your-yaml/2023-12-29T23:19:42+01:002023-12-29T23:19:42+01:00Radek Sprta<div><p>Leveraged by tools such Ansible & Kubernetes, YAML has become a ubiqitous configuration format over the last decade. It owes its high adoption
rate can be attributed to its simple design. It is consise yet readable, thanks to its minimal use of syntactical marks. However, under the
surface it is surprisingly complex. This post aims to introduce some of its features you might not be aware of.</p>
<p><a href="https://radeksprta.eu/posts/level-up-your-yaml/">Read more…</a> (3 min remaining to read)</p></div>Create Terraform Array In a Loophttps://radeksprta.eu/posts/create-terraform-array-in-a-loop/2023-01-09T23:09:13+01:002023-01-09T23:09:13+01:00Radek Sprta<p>The other day, I needed to create a multi-SAN TLS certificate in Terraform. The DNS names ended in a number sequence, so you could easily create them in a loop. Now, Terraform has count & for_each loop to create multiple resources. But what if you want to create a variable in a loop? (I am sure some people are screaming <a href="https://www.pulumi.com/">Pulumi</a> now.)</p>
<p>Luckily, Terraform has <a href="https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource">null_resource</a> available, which is exactly what I needed. It creates a virtual resource with a map of strings. So, you can define it in a loop and then use the result to declare a variable.</p>
<p>Here's example code:</p>
<div class="code"><pre class="code literal-block"><span class="nb">locals</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">dns_sans</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">null_resource.dns_sans</span><span class="p">[</span><span class="err">*</span><span class="p">].</span><span class="nv">triggers.dns_name</span>
<span class="p">}</span>
<span class="kr">resource</span><span class="w"> </span><span class="nc">"null_resource"</span><span class="w"> </span><span class="nv">"dns_sans"</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">count</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">var.replicas</span>
<span class="w"> </span><span class="nb">triggers</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="na">dns_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"service-${count.index}.service-internal"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>This made my life easier. Maybe it will make yours too.</p>Start AWX Job Via APIhttps://radeksprta.eu/posts/start-awx-job-via-api/2022-12-03T22:18:22+01:002022-12-03T22:18:22+01:00Radek Sprta<p>If you are using Ansible, there is high change you are also using <a href="https://github.com/ansible/awx">AWX</a> (or Ansible Tower) to orchestrate your jobs. And you might want trigger AWX jobs externally in some cases, such as from your CI pipeline. Luckily, AWX has an <a href="https://docs.ansible.com/ansible-tower/latest/html/towerapi/api_ref.html">API</a> that allows you to do just that.</p>
<p>To run AWX jobs remotely, you will need to make 3 API calls. One to start the job itself, another one to monitor its progress and lastly a request to print the output. You can see sample code to do that in bash shell below. In order to keep things simple, it uses authentication token, but you could also use OAuth 2.</p>
<div class="code"><pre class="code literal-block">POLLING_SLEEP=30
function<span class="w"> </span>print_job_output()<span class="w"> </span>{
<span class="w"> </span>api_request<span class="w"> </span>GET<span class="w"> </span>"/jobs/<span class="cp">${</span><span class="nb">id</span><span class="cp">}</span>/stdout/?format=txt"
}
response=$(
<span class="w"> </span>curl<span class="w"> </span>-s<span class="w"> </span>-i<span class="w"> </span>-o<span class="w"> </span>-<span class="w"> </span>"<span class="cp">${</span><span class="n">AWX_API</span><span class="cp">}</span>/job_templates/<span class="cp">${</span><span class="n">TEMPLATE_ID</span><span class="cp">}</span>/launch/"<span class="w"> </span>\
<span class="w"> </span>-H<span class="w"> </span>"Authorization:<span class="w"> </span>Bearer<span class="w"> </span><span class="cp">${</span><span class="n">API_TOKEN</span><span class="cp">}</span>"<span class="w"> </span>\
<span class="w"> </span>-H<span class="w"> </span>"Content-Type:<span class="w"> </span>application/json"<span class="w"> </span>\
<span class="w"> </span>-XPOST
)
http_status=$(head<span class="w"> </span>-n<span class="w"> </span>1<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">response</span><span class="cp">}</span>"<span class="w"> </span>|<span class="w"> </span>awk<span class="w"> </span>'{print<span class="w"> </span>$2}')
body=$(grep<span class="w"> </span>'^{'<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">response</span><span class="cp">}</span>")
if<span class="w"> </span>[[<span class="w"> </span><span class="cp">${</span><span class="n">http_status</span><span class="cp">}</span><span class="w"> </span>!=<span class="w"> </span>201<span class="w"> </span>]];<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"AWX<span class="w"> </span>returned<span class="w"> </span>error."
<span class="w"> </span>exit<span class="w"> </span>1
fi
id=$(jq<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">body</span><span class="cp">}</span>"<span class="w"> </span>'.id')
echo<span class="w"> </span>"Monitoring<span class="w"> </span>job<span class="w"> </span>ID:<span class="w"> </span><span class="cp">${</span><span class="nb">id</span><span class="cp">}</span>"
while<span class="w"> </span>true;<span class="w"> </span>do
<span class="w"> </span>echo<span class="w"> </span>"Sleeping<span class="w"> </span>for<span class="w"> </span><span class="cp">${</span><span class="n">POLLING_SLEEP</span><span class="cp">}</span>s"
<span class="w"> </span>sleep<span class="w"> </span><span class="nv">$POLLING_SLEEP</span>
<span class="w"> </span>response="$(api_request<span class="w"> </span>GET<span class="w"> </span>"/jobs/<span class="cp">${</span><span class="nb">id</span><span class="cp">}</span>/"<span class="w"> </span>-i)"
<span class="w"> </span>http_status=$(head<span class="w"> </span>-n<span class="w"> </span>1<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">response</span><span class="cp">}</span>"<span class="w"> </span>|<span class="w"> </span>awk<span class="w"> </span>'{print<span class="w"> </span>$2}')
<span class="w"> </span>body=$(grep<span class="w"> </span>'^{'<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">response</span><span class="cp">}</span>")
<span class="w"> </span>if<span class="w"> </span>[[<span class="w"> </span><span class="cp">${</span><span class="n">http_status</span><span class="cp">}</span><span class="w"> </span>!=<span class="w"> </span>200<span class="w"> </span>]];<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"AWX<span class="w"> </span>returned<span class="w"> </span>error."
<span class="w"> </span>exit<span class="w"> </span>1
<span class="w"> </span>fi
<span class="w"> </span>status="$(jq<span class="w"> </span>-r<span class="w"> </span>'.status'<span class="w"> </span><span class="err"><<<</span>"<span class="cp">${</span><span class="n">body</span><span class="cp">}</span>")"
<span class="w"> </span>if<span class="w"> </span>[[<span class="w"> </span>"<span class="cp">${</span><span class="n">status</span><span class="cp">}</span>"<span class="w"> </span>==<span class="w"> </span>"failed"<span class="w"> </span>]];<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>failed."
<span class="w"> </span>print_job_output
<span class="w"> </span>exit<span class="w"> </span>1
<span class="w"> </span>elif<span class="w"> </span>[[<span class="w"> </span>"<span class="cp">${</span><span class="n">status</span><span class="cp">}</span>"<span class="w"> </span>==<span class="w"> </span>"successful"<span class="w"> </span>]];<span class="w"> </span>then
<span class="w"> </span>echo<span class="w"> </span>"Job<span class="w"> </span>has<span class="w"> </span>finished<span class="w"> </span>successfully."
<span class="w"> </span>print_job_output
<span class="w"> </span>exit<span class="w"> </span>0
<span class="w"> </span>fi
done
</pre></div>Choosing the Right Programming Fonthttps://radeksprta.eu/posts/choosing-the-right-programming-font/2021-08-22T09:25:52+02:002021-07-06T08:44:44+02:00Radek Sprta<div><p>Programmers and other IT professionals often customize their desktops to increase productivity. However, fonts often remains overlooked. Since you spend around 8-10 hours per day staring at a screen, it is an important choice. The right font increases legibility. That in turn helps battling eye fatigue and lowers the risk of typos (which can lead to nasty bugs). This article compares some of the most popular programming fonts.</p>
<h3>Criteria for choosing a font</h3>
<p>First of all, a programming font should be monospace. I don't think this needs much explanation. The main reason is for structures in code to be aligned.</p>
<p>As mentioned above, the font should also be legible. Especially characters that are easily confused, such as lowercase l and uppercase I. To compare the various fonts, I am using a modified <a href="https://github.com/martinus/programming-font-test-pattern">programming font test pattern</a> from <a href="https://github.com/martinus">Martinus</a>. It looks like this:</p>
<div class="code"><pre class="code literal-block">o0O s5S z2Z qg9 !|l1Iij {([|])} .,;: ``''""
a@#* vVuUwW <>;^°=-~ öÖüÜäÄßµ \/\/ -- == __
the quick brown fox jumps over the lazy dog
THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG
0123456789 &-+@ for (int i=0; i<=j; ++i) {}
</pre></div>
<p>Last, but not least, you should consider the license. As more people use some mix of Linux, Windows and MacOS to do their work, I have picked only open-source fonts. You can easily get those on any OS. This disqualifies some popular choices, such as Consolas and Menlo.</p>
<p><a href="https://radeksprta.eu/posts/choosing-the-right-programming-font/">Read more…</a> (5 min remaining to read)</p></div>Elasticsearch Index Lifecycle Management for Fluentdhttps://radeksprta.eu/posts/elasticsearch-index-lifecycle-management-for-fluentd/2021-04-09T19:42:01+02:002021-04-09T19:42:01+02:00Radek Sprta<div><p>External tools, such as <a href="https://github.com/elastic/curator">Curator</a>, used to be a necessity for managing Elasticsearch indexes. This has changed with the introduction of Index Lifecycle Management in (ILM) Elasticsearch 6.6. It has all but eliminated the need for other tools. While has been developed primarily with Logstash in mind, you can also take advantage of it when using Fluentd. It works with both data streams and regular indexes. Becase most people are probably familiar with the latter, this post will explain how to setup ILM for your Fluentd indexes.</p>
<p><a href="https://radeksprta.eu/posts/elasticsearch-index-lifecycle-management-for-fluentd/">Read more…</a> (2 min remaining to read)</p></div>Control Whitespace in Ansible Templateshttps://radeksprta.eu/posts/control-whitespace-in-ansible-templates/2021-01-02T12:13:38+01:002021-01-02T12:13:38+01:00Radek Sprta<div><p>Ansible uses the powerful Jinja templating engine. However, the way it handles whitespace in templates is not ideal. Specifically, the fact that it preserves leading whitespace around Jinja blocks. So, you can either not indent Jinja syntax, making the templates hard to comprehend, or accept broken indentation in the resulting file (not an option with whitespace-sensitive formats such as yaml).</p>
<p>Luckily, there is a third option. Jinja has two configuration options regarding whitespace:</p>
<p><a href="https://radeksprta.eu/posts/control-whitespace-in-ansible-templates/">Read more…</a> (1 min remaining to read)</p></div>Import Scaleway Infrastructure to Terraformhttps://radeksprta.eu/posts/import-scaleway-infrastructure-to-terraform/2020-11-03T11:28:43+02:002020-11-03T11:28:43+02:00Radek Sprta<div><p><a href="https://www.terraform.io">Terraform</a> is one of the leading applications to manage your infrastructure as code. It defines your server instances and accompanying services using a simple declarative language. Moreover, the infrastructure state is kept in a separate file. So, whenever you make a change in your configuration, Terraform compares it to the current state and only performs necessary changes.</p>
<p>Terraform has plugins for all the major IaaS providers, so you should be covered there. However, you are probably not starting from a scratch, but already have some infrastructure running. Personally, I use <a href="https://www.scaleway.com">Scaleway</a> as my cloud provider, so I will show you how to import their resources to Terraform. I will demonstrate the process on a single server instance.</p>
<p><a href="https://radeksprta.eu/posts/import-scaleway-infrastructure-to-terraform/">Read more…</a> (3 min remaining to read)</p></div>Deploy Docker Container from Gitlab CIhttps://radeksprta.eu/posts/deploy-docker-container-from-gitlab-ci/2020-09-12T10:46:27+02:002020-09-12T10:46:27+02:00Radek Sprta<div><p>Containers are all the rage nowadays and for a good reason. They help in unifying development and production environments. They also provide application encapsulation and isolation, among other things. But to get the most out of them, you should build and deploy them automatically. This post will show you how to do it using Gitlab CI and docker-compose.</p>
<p><a href="https://radeksprta.eu/posts/deploy-docker-container-from-gitlab-ci/">Read more…</a> (2 min remaining to read)</p></div>Run Nikola Blog in Dockerhttps://radeksprta.eu/posts/run-nikola-blog-in-docker/2020-08-28T08:27:20+02:002020-08-28T08:27:20+02:00Radek Sprta<div><p>A lot of you might have a blog or a personal website created by static generator. Thanks to their simple requirements (just a webserver, really), they are an ideal starting point for your dockerization journey. In this post, I will explain how to run a <a href="https://getnikola.com">Nikola</a> website in a container. Nikola powers this website and is my static generator of choice. But the steps should be fairly similar for other generators out there.</p>
<h3>Dockerfile</h3>
<p>The dockerfile I am using looks like this:</p>
<div class="code"><pre class="code literal-block"><span class="k">FROM</span><span class="w"> </span><span class="s">python:latest</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="s">builder</span>
<span class="c"># Copy the whole repository into Docker container</span>
<span class="k">COPY</span><span class="w"> </span>.<span class="w"> </span>.<span class="w"> </span>
<span class="c"># Build the blog</span>
<span class="k">RUN</span><span class="w"> </span>pip<span class="w"> </span>install<span class="w"> </span>nikola<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="o">&&</span><span class="w"> </span>run<span class="w"> </span>nikola<span class="w"> </span>build
<span class="k">FROM</span><span class="w"> </span><span class="s">nginx:alpine</span>
<span class="c"># Copy output to the default nginx directory</span>
<span class="k">COPY</span><span class="w"> </span>--from<span class="o">=</span>builder<span class="w"> </span>output<span class="w"> </span>/usr/share/nginx/html
<span class="c"># Copy nginx host configuration</span>
<span class="k">COPY</span><span class="w"> </span>nginx/default.conf<span class="w"> </span>/etc/nginx/conf.d/
</pre></div>
<p><a href="https://radeksprta.eu/posts/run-nikola-blog-in-docker/">Read more…</a> (2 min remaining to read)</p></div>Modify All Items in Ansible Listhttps://radeksprta.eu/posts/modify-all-items-in-ansible-list/2020-08-08T08:58:40+02:002020-08-08T08:58:40+02:00Radek Sprta<p>Ansible lets you easily interpolate list items within values (like <code>interpolated_{{ item }}_value</code>). However, sometimes you need a more powerful transformation. This is where the map filter comes to rescue again. You can use it to perform regular expression replace on each item in a list. As you can see, the syntax is relatively simple:</p>
<div class="code"><pre class="code literal-block"><span class="nb">map</span><span class="p">(</span><span class="s1">'regex_replace'</span><span class="p">,</span> <span class="n">REGEX_PATTERN</span><span class="p">,</span> <span class="n">OUTPUT</span><span class="p">)</span>
</pre></div>
<p>For a concrete example, let us say you want to extract network mask from a list of IP addresses (<code>192.168.0.100/24</code> for example). Assuming this list is stored in the <code>ip_addresses</code> variable, the regex replace would look like this:</p>
<div class="code"><pre class="code literal-block"><span class="p">{{</span> <span class="n">ip_addresses</span> <span class="o">|</span> <span class="nb">map</span><span class="p">(</span><span class="s1">'regex_replace'</span><span class="p">,</span> <span class="s1">'.*/([0-9]{1,2})'</span><span class="p">,</span> <span class="s1">'</span><span class="se">\\</span><span class="s1">1'</span><span class="p">)</span> <span class="o">|</span> <span class="nb">list</span> <span class="p">}}</span>
</pre></div>
<p>Of course, you can easily use it as a part of longer jinja2 pipelines. If you also want to learn how to <a href="https://radeksprta.eu/posts/loop-over-dictionary-attribute-in-ansible/">loop over dictionary attribute</a>, or see other Ansible tips, take a look <a href="https://radeksprta.eu/categories/ansible/">here</a>.</p>