journal on chamila.dev

Stop Using Jump Hosts for EC2 Windows Instances

Sun, 14 Dec 2025 15:31:27 +1300

For the most part of virtualisation and public IaaS Cloud existence, the pattern to access VMs that are in private networks has been to use a jump host, also known as Bastion hosts. This would be another VM, usually with less resources, and even less maintenance, that would be hosted in a public (or directly accessible) network that a user would first access, and then “jump” to the privately hosted VM because the public network or the instance would be whitelisted in the firewall rules to be able to access private resources.

What AWS outtage really means

Tue, 21 Oct 2025 22:50:51 +1300

So it’s been a couple of hours since the last major AWS outtage has been set to RESOLVED. This was a major outtage in the AWS us-east-1 region, an incident that went on for more than 15 hours!

What resulted was a global outtage of several major service providers, including accounting software, messaging software, even banking systems. There seems to be multiple millions or billions of loss estimated as a direct result of this outtage.

How to Securely Design and Build Cloud Applications

Wed, 13 Aug 2025 14:11:38 +1200

In the last post/video, I tried to replicate the stupidity of (almost) intentionally exposing your private user data as a public bucket/database. While it felt great to reinforce my belief that Cloud is relatively safe by itself, to reinforce my ego to know at least I’m not that stupid, I did feel a little incomplete. I broke down what could’ve possibly gone wrong, but I didn’t show how to fix it, or prevent that from happening.

Is it that easy to make stupid mistakes? - Trying to Reproduce the Tea App Situation

Tue, 05 Aug 2025 16:28:16 +1200

Tea app getting hacked is an interesting case.

It’s not technically a hack. No existing security measures were breached, there wasn’t any complicated social engineering, or zero-days or unpatched CVEs getting exploited. What happened is what’s called a Data Extraction. Data that should have been private was public, and naturally, people extracted it, twice!!

AWS Authentication for K8s: IAM Roles for Service Accounts (IRSA)

Mon, 10 Mar 2025 15:59:36 +1300

This is going to be multiple videos for various reasons. First video is below.

In this installment of the series on authenticating into AWS from K8s workloads, I’m going to go into details on another major approach that can be used. This approach is called IAM Roles for Service Accounts (IRSA), and that adds nothing to explaining what this approach is. If you remember from the first video, almost all approaches use IAM Roles that are mapped to Service Accounts somehow (unless you’re embedding IAM user credentials in the Pod definition, for the upteenth time, let me strongly advise you not to do that).

AWS Authentication for K8s: EKS Pod Identities Deep Dive

Sat, 01 Feb 2025 00:04:06 +1300

In the last article, I covered in brief, the solutions for the problem of authenticating into AWS from K8s workloads. This is the first detailed article (and the video) I promised I’ll follow up that with, going into to details on EKS Pod Identities. I’m hoping to make this the most detailed, most in-depth video on the topic, but this article would probably have less details, so I highly recommend watching the video. It also has a full demo on how to create an EKS cluster for Pod Identities.

AWS Authentication for K8s: Introduction

Mon, 27 Jan 2025 00:01:44 +1300

This is an introductory post on the problem of authenticating into AWS from K8s workloads, either from within AWS or from outside of it. I’m going to discuss three major approaches to this problem and compare them against each other. I’ll follow this up with more detailed articles (and videos) on each approach, where I’ll go into details as much as possible.

Why I don't like taking Remote Proctored Exams

Thu, 16 Jan 2025 22:47:18 +1300

I recently renewed my Certified Kubernetes Security Specialty certification, and it reminded me, hard, why I don’t like taking remote proctored exams. This is going to be rant, probably not educative.

All CNCF certification exams are remote proctored. This means that you book the exam online and then take it online at a place of your choosing. Sounds unbelievably simple right? There’s a reason why it’s unbelievable.

Deep Dive in to AWS Resource Control Policies

Sun, 15 Dec 2024 23:35:34 +1300

This article is on one of the new features AWS released recently, called Resource Control Policies (RCPs).

RCPs are a new way to enforce policies on data (resources) rather than principals in AWS. For principals, the way to enforce policies is to use Service Control Policies (SCPs), which if you’re already familiar with AWS would know plenty about. However, RCPs can still be a little confusing.

Becoming a CNCF Kubestronaut

Sun, 17 Nov 2024 23:06:10 +1300

In early July this year, I became a Kubestronaut, the 2nd (or 3rd, I’m not exactly sure) in New Zealand.

The Programme

The CNCF Kubestronaut programme is a developer recognition programme launched by Cloud Native Computing Foundation. The aim of the programme is to appreciate the engineers who put “consistent effort into building skills and education” in Kubernetes (and related CNCF areas).

Completing 11 Professional Certifications in 8 Weeks

Mon, 28 Oct 2024 21:36:19 +1300

I wanted to write these details down for a long time, but I wanted to wait until I received the Golden Jacket. In this article, I’m going to talk about how I planned, prepped for each exam, and some generalised lessons from my experience

From end of May to mid of July, I studied for, and completed 11 professional certification exams. 9 out of those were AWS certifications, and the other two were the recently introduced CNCF foundational exams, Kubernetes and Cloud Native Associate (KCNA) and Kubernetes and Cloud native Secutiy Associate (KCSA). I’m mostly going to talk about the AWS exams in this article.

Receiving my AWS Golden Jacket

Sun, 13 Oct 2024 00:00:19 +1300

So this article and the video is my congratulating myself for getting the coveted AWS Golden Jacket.

Seriously though, it is a really good feeling to finally have the physical jacket with me. Not that I can wear it during the coming spring and summer. To be honest though, not that I’ll be wearing this outside anyways. It’s so shiny that I’ll get pulled over for blinding other drivers.

AWS Backup: Restoring Data and Automated Restore Testing

Mon, 23 Sep 2024 00:26:50 +1300

Intro

I covered establishing a comprehensive cross-account backup strategy in AWS Organizations in the previous articles and videos.

However, as an architect, one of the most frequent patterns I come across in the real world in this space is organisations having an imbalance of backup and recovery strategies. This is when there is a really detailed backup process that includes all the production resources in the backup jobs, with encrypted and replicated copies, but having minimum or even zero effort in testing recoverability of the backed up data and having playbooks for such scenarios.

AWS Backup Cross-Account Backups in AWS Web Console

Sun, 11 Aug 2024 18:03:37 +1300

So my last post on AWS Backup uses a lot of Terraform code that I had written even before the post. When doing a video on the subject, I realised that for a beginner, explaining something like AWS Backup with Terraform presents two fronts to understand at the same time.

Using Multiple AWS CLI Profiles in the Rust AWS SDK

Sat, 01 Jun 2024 05:34:43 +1200

I wrote a blog post on getting started with Rust AWS SDK a few months ago. I also started my youtube channel with a video on the same topic. If you haven’t yet seen it, my channel is sliced array (the name would probably change, but this is good enough for now).

One of the comments asked about how to use different AWS CLI profiles in the Rust SDK, and I did a follow up video on the topic. This post is a supplementary post for that video.

Hit the ground Running with FinOps in AWS

Fri, 31 May 2024 09:48:59 +1200

Sydney Harbour Bridge, about 9 years of build time

Understanding and managing Cloud Expenditure is an increasingly sought after topic with companies as the focus on Cloud Financial Management is increasing. Existing AWS customers with migrated workloads in the Cloud that are mature are looking to optimise their Cloud Costs and realise the dream of low-cost elasticity.

At the same time, new projects that start their journey in the Cloud can also find themselves out of control of their Cloud Expenditure, without being able to control and react to unexpected spending patterns. The main offering the Cloud, elasticity, that enables easy access to infrastructure that would otherwise take a long time to be provisioned, also provide an easy way accidentally end up with a large bill. A slight misconfiguration of the permission policies could allow a new developer to unintentionally rack up recurring Cloud costs.

AWS Multi-factor Authentication in Detail

Wed, 03 Apr 2024 11:21:09 +1300

Multi-factor Authentication (MFA) is an easy way to enhance the security of your AWS cloud architecture with minimal work. It takes two minutes to enable, and minimizes the possibility of an account being compromised by tenfolds. This post and the associated video explores MFA in AWS and how to use it to control access to critical parts of your cloud architecture

Deriving AWS Account ID from Access Key

Mon, 11 Mar 2024 17:04:21 +1300

AWS Account IDs are interesting. They are not directly considered as sensitive, but on the other hand everyone seems to consider them as sensitive.

It’s probably for good reason. You shouldn’t give out anything that isn’t needed to get work done, even if those data are not “sensitive” in particular. Your name isn’t sensitive in particular, but you don’t usually walk around with your name on your tshirt either.

Setting Yubikey MFA for AWS CLI temporary credentials

Thu, 18 Jan 2024 16:30:03 +1300

Walkway harness on a Redwood tree in Redwoods Tree Walk, Rotorua

It is a good practice to use temporary credentials in AWS instead of permanent IAM generated credentials that are long term and potentially leaked. AWS has different services that issue temporary credentials with given permission policies such as IAM Instance Profiles for EC2 instances, Lambda Execution Roles, ECS Task Roles, and IAM Roles Anywhere for those cases that require providing trusted access permissions to a process running outside of AWS.

Deploying Rust on Lambda

Tue, 16 Jan 2024 12:57:07 +1300

Dead (not a professional opinion) Pine tree on the coastal California

Following up from the last post let’s explore how to use Rust in Lambda functions.

AWS has introduced a neat cargo command to make life easier and focus on building the business logic rather than get bogged down in deployment details. This is the cargo lambda command. Installing the command is pretty straightforward on a linux environment.

pip3 install cargo-lambda

One detail to note is that cargo lambda command uses zig toolchain to facilitate cross compiling the Rust code. This is important because Rust in Lambda runs on the OS only runtimes which run on Amazon Linux versions. So, unless you’re developing on a Linux workstation, chances are you’d probably be cross compiling your code. zig toolchain is used because it comes built in with the cargo-zigbuild crate. More information on cross-compiling and known issues can be found on the documentation.

Getting Started with AWS Rust SDK

Mon, 15 Jan 2024 05:26:45 +1300

Some building in Washington DC. It's just pretty, not related to Rust or AWS

AWS re:Invent 2023 announced the GA versions of the Rust and Kotlin SDKs. The SDK was in developer preview for sometime, and this release allows the use of it in production. Following is an introduction to the Rust SDK, its defaults, and a few detailed scenarios that are different from the default cases.

The working code for the following examples are in Github.

Pi-hole Restore CLI Tool

Mon, 19 Jun 2023 00:09:59 +1200

A loading tunnel in Wrights Hill Fortress

I made a thing (this time more niche than the last project).

I’ve been using Pi-hole as a DNS ad-blocker and an overall local DNS solution for about two years now. I lived dangerously for most of this time with only a single old Raspberry Pi 2 with Pi-hole in it as the sole DNS server for the local network.

That changed recently when I added another couple of Docker containers as Pi-hole servers to the network.

Information Security Compliance in AWS Solutions

Sun, 04 Jun 2023 19:30:02 +1200

Information Security Compliance can be a tiring and complicated exercise for all parties involved. Making a solution compliant for the set of security standards and controls is a massive effort, especially if the architecting stage did not think about these requirements in the first place.

It’s even more complex when it comes to proving the compliance level of the solution to internal or external auditors. The language InfoSec and Security Auditors speak is vastly different from the language used by engineers.

Malware Protection on AWS

Sat, 25 Feb 2023 07:00:19 +1300

Point Dorset, Wellington

Malware protection in compute workloads in AWS is a topic that I’ve been looking into recently. After working with security auditors for a few weeks (adding up a few months now), I was essentially forced to evaluate all the possible scenarios for malware protection in AWS for typical workloads.

Typical Malware Protection Requirements

The following key features can be identified as typical malware protection requirements for most solutions.

AWS Backup: Implementing a Simple Cross-Account Backup Strategy

Mon, 20 Feb 2023 00:15:44 +1300

Kothmale Valley view near Ramboda

Following up from the last two₁ blog posts₂ on AWS Backup, this post focuses on a technical implementation of a simple backup strategy implementation for AWS Organizations.

Let’s extend the last two posts on AWS Backup to an example.

In this scenario, we have an AWS Organization with three accounts, management, production (prod), and central backup (backup). There is a DynamoDB table named tracking and an RDS MySQL instance named mydb that contains critical production data in the prod account. Backing up data with AWS Backup needs to be set up for these two databases.

AWS Backup: Cross-Account Backups

Mon, 13 Feb 2023 23:29:05 +1300

Mount Kaukau

Continuing from the previous article on introduction to AWS Backup

Offsite Backups

Let’s talk theory for a bit.

Data protection in a given solution depends on how thoughtful the backup strategy is. While a simple Backup Plan in AWS Backup that periodically takes a backup of the database into the Account-local Vault could technically be considered as a backup strategy, it is not a good one. For an effective and resilient data protection policy, backups should exist in locations other than the solution deployment, with access to them separated by network and access levels.

AWS Backup: Introduction

Wed, 18 Jan 2023 09:14:05 +1300

What else is better than a photo of water being backed up - Birchville Dam

AWS Backup

AWS Backup is a service introduced a few years ago to centrally manage backup options for various AWS services that deal with data persistence. These include RDS, DynamoDB, EBS, EFS, EC2, S3, and Amazon FSx. What was earlier at best by processes and architecture reviews and audits in different services with different strategies can be consolidated into a single service with a single Console. For example, what was earlier RDS specific backup design would be collected into an AWS Backup Plan that aligns the specific RDS instance with the rest of the data persistence in a solution without having to manually compare different approaches for backup with different services.

MLOps for Non-ML Engineers 04 - Unique aspects of an ML Project Execution

Mon, 28 Nov 2022 09:11:34 +1300

Whanganui War Memorial Centre

Continuing comparison between Machine Learning and Software Development, from the previous article in the series.

MLOps for Non-ML Engineers 01 - Introduction

MLOps for Non-ML Engineers 02 - Differences Between ML and SW Dev

MLOps for Non-ML Engineers 03 - More Differences Between ML and SW Dev

MLOps for Non-ML Engineers 04 - Unique aspects of an ML Project Execution

Differences between Software Development projects and Machine Learning projects were discussed in the previous articles. Let’s look at ML Engineering responsibilities and other unique aspects involved in a Machine Learning project.

MLOps for Non-ML Engineers 03 - More Differences Between ML and SW Dev

Fri, 25 Nov 2022 06:37:25 +1300

Evans Bay, Wellington

How is Machine Learning Different?

Continuing comparison between Machine Learning and Software Development, from the previous article in the series.

MLOps for Non-ML Engineers 01 - Introduction

MLOps for Non-ML Engineers 02 - Differences Between ML and SW Dev

MLOps for Non-ML Engineers 03 - More Differences Between ML and SW Dev

MLOps for Non-ML Engineers 04 - Unique aspects of an ML Project Execution

Source Version Control

Software development and version controlling is virtually impossible to separate. Yes, there are still people who treat attaching source code to emails and JIRA tickets as version control, and some developers may refuse to learn git citing “separation of concerns” or whatever. The world is not perfect. But overall, code progresses through the build and deployment lifecycle with version control closely supporting it.

MLOps for Non-ML Engineers 02 - Differences Between ML and SW Dev

Tue, 22 Nov 2022 06:32:18 +1300

Lighthouse, Galle, Sri Lanka

How is Machine Learning Different?

Continuing comparison between Machine Learning and Software Development, from the previous article in the series.

MLOps for Non-ML Engineers 01 - Introduction

MLOps for Non-ML Engineers 02 - Differences Between ML and SW Dev

MLOps for Non-ML Engineers 03 - More Differences Between ML and SW Dev

MLOps for Non-ML Engineers 04 - Unique aspects of an ML Project Execution

Understanding the solution

Various iterations of design and architecture of a software development project usually determines the approach of the solution for a software engineering problem. As the design becomes more verbose, the higher the likelihood that details can change based on new findings. These changes usually have a low cost of switching, and do not impact the end product and the development pipeline significantly. As much as initial understanding of key requirments of the problem is important, approaches like Agile Methodology try to avoid committing to a grand detailed design at the start of the project and instead develops the solution and detailed design in iterations.

MLOps for Non-ML Engineers 01 - Introduction

Sat, 19 Nov 2022 05:49:28 +1300

Bixby Creek Bridge on CA-01 highway, Monterey, California

This is the first part of a series of posts on the topic.

MLOps for Non-ML Engineers 01 - Introduction

MLOps for Non-ML Engineers 02 - Differences Between ML and SW Dev

MLOps for Non-ML Engineers 03 - More Differences Between ML and SW Dev

MLOps for Non-ML Engineers 04 - Unique aspects of an ML Project Execution

Last year I had an opportunity to work with a team of Data Scientists on developing an Enterprise AI/ML practice. As someone who had not interacted at development level with Machine Learning and AI before, I jumped on the chance to learn something new and find a challenge on a new area. The following are a couple of notes (more to myself) of key points I learnt as an engineer with only software development and DevOps experience.

Rotating AWS Secrets Manager Secrets with Lambda

Tue, 01 Nov 2022 07:09:49 +1300

Again, the only photo with me for now that's closest to the theme, a spider weaving a trap near Manigala, Sri Lanka

Secrets Manager secret types

AWS Secret Manager supports a several types of secrets. These are mainly to support effective secret management by integrating them closer with the AWS Services they support. These are mainly various types of database services offered by AWS, which is one of the key uses for using Secrets Manager to manage master credentials.

Automating Snowflake

Fri, 12 Nov 2021 19:32:16 +1300

Only photo with me for now that's closest to the theme

I had an opportunity to work on automating a Snowflake workflow last year. The scenario involved a typical Snowflake based setup for a Data Warehouse where data would be ingested from an Azure Blob Storage or AWS S3 and stored on multiple databases. I’m not going to dive into intricate details, but I’d like to write down some of the approaches I wanted to take during this exercise. And yes, that makes this blog post way overdue.

CI/CD for Apache Nifi 02: Automation with NipyAPI

Sun, 18 Jul 2021 22:20:47 +1200

Wharf on a good day

In the earlier post on setting up Apache Nifi and Nifi Registry, two Nifi and Registry deployments set up and integrated with each other. This post will focus on automating the migrations between the two environments. Just to refresh, the following diagram shows what we are trying to achieve.

Though this shows three environments, we have only taken the dev -> test step into consideration in the current and the previous posts. This is because once you automate the series of steps, it’s just a matter of pointing the script to the next step.

CI/CD for Apache Nifi 01 - Introduction and Set up

Sat, 17 Jul 2021 13:21:12 +1200

Harbour on a misty day

I’m getting back to writing after more than an year of a gap. A lot happened during this period, moved from one city to another, changed employers, but mostly, I think I switched my writing time for some PS4 time. At one point though, you come to accept that your best K/D is going to be 0.2 and you’re too old to get better. In any case, I managed to do a few rounds of bug fixing to my Kibana Prometheus Exporter, and learn a few things in an area I didn’t have that much of experience in, Big Data.

A Standalone Prometheus Exporter for Kibana

Sat, 14 Mar 2020 13:37:33 +1300

I wrote a Prometheus Exporter that converts and exposes some metrics from Kibana API. This was mostly inspired by an already existing Exporter, that needs to be installed in Kibana as a Plugin. While that is a neat feature, managed ELK services like ElasticSearch Service by Elastic (commonly known as ElasticCloud) require another, more standalone approach.

kibana-exporter -kibana.uri http://localhost:5601 -kibana.username elastic -kibana.password password

The Exporter is pretty basic one written in less than 500 lines of collective Go code. The following metrics are exposed to be scraped by Prometheus.

Authentication and Authorization for ElasticSearch: 03 - Multi-Tenancy with KeyCloak and Kibana

Wed, 12 Feb 2020 07:07:35 +1300

Horton Plains at sunrise, Sri Lanka

This is a continuation of the addendum to a series of articles on ELK on K8s.

ElasticSearch on K8s: 01 — Basic Design
ElasticSearch on K8s: 02 — Log Collection with Filebeat
ElasticSearch on K8s: 03 - Log Enrichment with Logstash
ElasticSearch on K8s: 04 - Log Storage and Search with ElasticSearch
ElasticSearch on K8s: 05 - Visualization and Production Readying
ElasticSearch Index Management
Authentication and Authorization for ElasticSearch: 01 - A Blueprint for Multi-tenant SSO
Authentication and Authorization for ElasticSearch: 02 - Basic SSO with Role Assignment
Authentication and Authorization for ElasticSearch: 03 - Multi-Tenancy with KeyCloak and Kibana

The previous post on this series was about enabling SSO between Kibana and KeyCloak. However, it did not address any concerns about multi-tenancy with SSO. This article aims to come up with a design for a multi-tenant SSO solution for Kibana with KeyCloak.

Authentication and Authorization for ElasticSearch: 02 - Basic SSO with Role Assignment

Mon, 10 Feb 2020 20:26:46 +1300

This is a continuation of the addendum to a series of articles on ELK on K8s.

ElasticSearch on K8s: 01 — Basic Design
ElasticSearch on K8s: 02 — Log Collection with Filebeat
ElasticSearch on K8s: 03 - Log Enrichment with Logstash
ElasticSearch on K8s: 04 - Log Storage and Search with ElasticSearch
ElasticSearch on K8s: 05 - Visualization and Production Readying
ElasticSearch Index Management
Authentication and Authorization for ElasticSearch: 01 - A Blueprint for Multi-tenant SSO
Authentication and Authorization for ElasticSearch: 02 - Basic SSO with Role Assignment
Authentication and Authorization for ElasticSearch: 03 - Multi-Tenancy with KeyCloak and Kibana

As discussed in the last article, I will try to lay out the configuration details of a usable SSO based authn/authr design for an ELK deployment. Note that for certain features mentioned in this article to work, some features that are in the paid ElasticSearch tiers have to be available.

Authentication and Authorization for ElasticSearch: 01 - A Blueprint for Multi-tenant SSO

Wed, 27 Nov 2019 19:43:20 +1300

Point Sur, Cali 01

I just completed a mini article series on details of deploying an ELK stack on K8s. Following are the links to the series.

ElasticSearch on K8s: 01 — Basic Design
ElasticSearch on K8s: 02 — Log Collection with Filebeat
ElasticSearch on K8s: 03 - Log Enrichment with Logstash
ElasticSearch on K8s: 04 - Log Storage and Search with ElasticSearch
ElasticSearch on K8s: 05 - Visualization and Production Readying
ElasticSearch Index Management
Authentication and Authorization for ElasticSearch: 01 - A Blueprint for Multi-tenant SSO
Authentication and Authorization for ElasticSearch: 02 - Basic SSO with Role Assignment
Authentication and Authorization for ElasticSearch: 03 - Multi-Tenancy with KeyCloak and Kibana

So far, the articles have been discussing points related to functional requirements of a Log Aggregation stack deployed on K8s. The functionalities involved ELK core functions and how to mix and match K8s features to get the best out of the core ELK features. After getting the stack to receive and store logs, there are parallels that should be addressed. User management is one these critical features that this article will be discussing about.

ElasticSearch Index Management

Tue, 26 Nov 2019 22:03:00 +1300

The comms tower on top of Riverston, Sri Lanka

As the series on ElasticSearch deployment management in K8s is complete, I thought of writing down some of the Index Management tasks that I had to implement in order to reduce the manual work involved in cluster maintenance.

Following is the series of posts on ElasticSearch on K8s.

The following management steps are not mandatory to be implemented in a cluster to be production ready, however having them in place would greatly reduce some of the common headaches involved in an ELK stack management.

ElasticSearch on K8s: 05 - Visualization and Production Readying

Mon, 25 Nov 2019 22:29:57 +1300

Woolshed Hut on Mt. Somers

This is part of a series of short articles on setting up an ELK deployment on K8s.

Visualization

Kibana is a web application which can be used to query data from an ElasticSearch cluster. It does so through issuing queries to the ElasticSearch cluster through the REST API. Additionally it also acts as a widget and dashboard server that can be used to visualize specific queries, as well as a management frontend that can perform several actions like role and user management that are a lot cumbersome to do so through the ElasticSearch REST API. Kibana itself doesn’t have a requirement for persistence as it uses ElasticSearch itself as a persistence layer.

ElasticSearch on K8s: 04 - Log Storage and Search with ElasticSearch

Sat, 23 Nov 2019 16:01:40 +1300

The morning shadow on the lower mountaineous plain beside Adam's Peak, Sri Lanka

This is part of a series of short articles on setting up an ELK deployment on K8s.

Storage and Indexing

Now that the logs are translated into a meaningful set of data, it’s time to store and index them for querying. ElasticSearch is the tool of choice, the main attraction of the stack to be honest. It’s an efficient useful tool to store and index unstructured data. How ElasticSearch’s design (which is something that was done in a time when Docker was not even invented yet) is matched to K8s is an interesting story of itself.

ElasticSearch on K8s: 03 - Log Enrichment with Logstash

Fri, 22 Nov 2019 01:05:37 +1300

A stash of sugar cane ready to be processed into sugar and Arrack

This is part of a series of short articles on setting up an ELK deployment on K8s.

Log Enrichment

Now that the logs are being collected from the required sources, it’s time to start making some sense out of them. This part of the process is called enrichment, and Logstash is the tool of choice in our stack.

Medium to Hugo

Sun, 20 Oct 2019 18:24:16 +1300

Taking a minor break from the “ElasticSearch on K8s” series

tl;dr: I’m moving back to maintaining my own site rather than depending on Medium as a platform for technical blogging, because of various reasons. Also introducing a tool to convert from Medium to Hugo.

A tale of not making up my mind

To start with somewhat of a boast, I’ve been writing blog posts for more than a decade now, on various platforms, SaaS ones to self-hosted solutions. Most of the posts are now nowhere to be seen since I cannot seem to make up my mind about where to keep my writings. I started some serious technical blogging about five years ago. That started with using Github Pages as a hosting service using Octopress as a site generator. After a while, getting exhausted on the themeing and waiting for the next Octopress version, I switched to Medium halfway through, importing all the posts.

ElasticSearch on K8s: 02 — Log Collection with Filebeat

Sat, 21 Sep 2019 05:57:08 +0000

This is part of a series of short articles on setting up an ELK deployment on K8s.

Log Collection

The typical task for a log collection tool is to collect a specified set of logs, from a specified set of locations, and offload them to a specified endpoint. Let’s explore these three aspects in detail.

ElasticSearch on K8s: 01 — Basic Design

Thu, 19 Sep 2019 20:29:53 +0000

A design for a useful ELK deployment on K8s

Log aggregation in a K8s environment is something I have lightly touched upon previously in multiple occasions. However setting up a minimal but a reliable log aggregation stack on top of K8s could quickly become an evolutionary process with each step improving on the previous one (and of course, everyone thinks they can do log aggregation before they actually start to do so). The following is a pattern for ELK I came across while improving such a stack. While this pattern addresses most common requirements for a native K8s log aggregation stack, there could be better, more suitable patterns that you may have to consider when setting up solutions of your own.

Load Balancing and Reverse Proxying for Kubernetes Services

Thu, 28 Feb 2019 17:56:15 +0000

Different load balancing and reverse proxying strategies to use in Production K8s Deployments to expose services to outside traffic

Morning sunlight on Horton Plains National Park

In this post, I’m going to tackle a topic that any K8s novice would start to think about, once they have cleared the basic concepts. **How would one go about exposing the services deployed inside a K8s cluster to outside traffic?**The content and some of the diagrams I’ve used in the post are from an internal tech talk I conducted at WSO2.

Releasing DocBlock v0.1

Fri, 15 Feb 2019 18:17:04 +0000

A tool to automate technical content generation for configuration files

Many months ago, a technical writer colleague of mine complained about how they were struggling to keep up with the frequent releases that the company was doing at the time. There were multiple products in their plate, with each having multiple configuration files (sometimes numbering more than 10). Although the configuration files overlapped within each product, because of the componentized platform the company had built the products upon, each product in theory could have different release versions of the components that used these configuration files. All of these had to be documented into readable (and most importantly usable) technical content.

How to Design a WSO2 Docker Image

Mon, 24 Dec 2018 16:49:16 +0000

What should be your concerns for WSO2 on Docker?

Deploying WSO2 products on Containerized platforms is a well-tested well-resourced activity. There are various resources available to deploy WSO2 products on Docker, Kubernetes, CloudFoundry, AWS ECS, and Apache Mesos, both officially and unofficially. However, designing a Docker image so that optimal non-functional traits like performance, operational efficiency, and security is a separate topic in itself.

Options available to obtain WSO2 products other than downloading the Zip file itself (https://wso2.com/api-management/install/)

Docker, being both a DSL and a utility tool for Container image packaging, can be written in several different ways keeping various operational goals in mind. These approaches vary from an all files and configs pre-baked end to another end where almost everything, except a tiny helper layer, being mounted at Container run-time. However, not all these approaches are suitable for a WSO2 deployment, and figuring out these details can be a daunting task, especially at an earlier phase of the deployment. Therefore, it’s important to have a set of common guidelinesthat can help determine which approach to take in each case. I emphasize the word guidelines because these are only them, not hard rules or must-do’s in every deployment. There is freedom for the user to stray away, as long as the compromises made in doing so are understood.

Publishing WSO2 Logs to Splunk from a Containerized Deployment

Wed, 28 Nov 2018 09:51:30 +0000

Or how to publish to Splunk from any Docker environment

WSO2 products follow a standard structure when it comes to configuration, data, artifacts, and logging. Configuration files are found in <CARBON_HOME>/repository/conf folder, data in <CARBON_HOME>/repository/data, artifacts in <CARBON_HOME>/repository/deployment (or in <CARBON_HOME>/repository/tenants folder if you’re in to multi-tenancy). All the log files are written into <CARBON_HOME>/repository/logs folder.

Log Aggregation

All log events are output as entries to files through Log4J. Because of this, when it’s time to attach WSO2 logging to a log aggregator, it’s a matter of incorporating a tailing file reader agent and directing it towards <CARBON_HOME>/repository/logs folder. For an example, for ELK this could be something like FileBeat.

CI/CD APIs with WSO2 API Manager

Sun, 25 Nov 2018 03:47:20 +0000

How to do continuous integration and continuous delivery of APIs with WSO2 API Manager

WSO2 API Manager, the only Open Source Leader in API Management Solutions in Forrester Wave, packs in a wide range of advanced API Management features that covers a number of end user stories. Through customization introduced to the extension points available throughout the product, WSO2 API Manager can be adopted to almost all API Management scenarios imaginable.

A Primer on Observability for Dynamic Organizations — Part 2

Thu, 25 Oct 2018 06:44:38 +0000

What should an Observability Framework address?

In the previous post, we measured the temperature of the water on what Observability is and why it should be a first class consideration in system design. Let’s explore the possibility of a structured approach for designing observable systems.

Massive industrial process of Sugar production where the mechanics are mostly literally behind 10-inch steel walls

Why should there be a structured approach?

In short, because Observability has to be designed into a system rather than be considered as an on-the-spot hack.

A Primer on Observability for Dynamic Organizations — Part 1

Tue, 21 Aug 2018 04:29:46 +0000

A introduction to the buzzword and the rationale for implementing it

What is Observability?

Before we dive in to the waters, we need to define what observability is. Let’s go for some tweets first.

Monitoring is for operating software/systems
Instrumentation is for writing software
Observability is for understanding systems
— Charity Majors (@mipsytipsy) September 23, 2017

That’s Charity Majors, a well known voice on Observability and someone who has built a business doing it well.

That Tricky Thing Called Automation

Fri, 20 Jul 2018 05:31:06 +0000

Automation of processes is something every serious organization should look into. The other option is to waste your precious human resources on tasks that are repetitive and hardly challenging. That is a depreciation in both man-hours and the employee morale. No one wants to keep on doing the same thing over and over again, especially if the tasks do not require the proper use of the human intellect.

Automation enables the machine driven initiation and management of these tasks and sometimes can even help coordinate the use of manual intervention. Therefore, any organization that doesn’t want to get stuck in the everyday details, automation of processes is a must.

In Other Good News, Microsoft buys GitHub

Tue, 05 Jun 2018 06:00:19 +0000

Or why you should cool down with the anti-Microsoft rhetoric, at least for now

Disclaimer: This is an opinionated article.

What a historic moment! May be only second to the moment Windows incorporated the Linux Subsystem, one-time anti-open source giant buys off may be the biggest open source platform that drove the eco-system, the ethos, and the practices for open source to a new level during past few years. And my Facebook feed (yes, I’m still friends with that guy) is filled with rants on how Microsoft will kill Open Source.

Server Immutability

Thu, 17 May 2018 08:19:19 +0000

Server Immutability is an interesting concept that I managed to come across when I first started playing around with Containers. Interestingly, it’s often used in conjunction with Containerization because of the use of startup file systems (or Images). However Server Immutability is something that goes beyond simple Docker images.

What is Immutability?

Immutability, when it comes to server instances, is not changing a once deployed instance. Any change that should be done, has to be done in a new version of the instance image and the old running instances should be replaced with the instances spawned from the new image.

Basic Process Metrics Collection and Visualization in Linux

Tue, 15 May 2018 06:36:59 +0000

When it comes to deployment observability there are multiple, comprehensive solutions that can handle almost any system of scale you throw at them. I said almost, because there are certain stories that do not require setting up a central monitoring server with satellite agents to collect and enrich data.

For an example, consider a scenario where you’re troubleshooting a certain single process for a particular period of time. You want to collect the approximate CPU usage, approximate memory usage, light weight process (aka thread) count, and the TCP connection count. You want these stats collected during the time period you’re conducting load test against the process.

A little bit of Bashfu for a specific case of log analysis

Tue, 01 May 2018 07:46:05 +0000

This post is not about a solution to a generic issue. This post is about how I found some way to cut and assemble a large log file using a few Bash tools (and subliminally why Unix principle of implementing stuff may be the superior way).

So today, there came a need to sort through a large access log file for a list of request counts for each key. It’s not just a key — request count relationship. To elaborate briefly,

Sample: WSO2 EI Cache Mediator based Token Caching

Sun, 12 Nov 2017 14:30:44 +0000

This post and the sample code are the results of a particular issue I had to tackle recently. Though the sample code is my own, the idea and the approach have many authors, arising from the collective knowledge on the WSO2 Middleware Stack.

The Typical Case for Caching

Token based authentication is not a new paradigm. The basic story is,

Talk to a Identity Management Service and obtain a token based on a kind of authentication
Call a service provider API, providing the token received in step #1
Service provider validates the token and acts on the privileges translated from the token

Subject Alternative Names in SSL Certificates

Mon, 06 Nov 2017 14:26:50 +0000

Or the SSL Cert Extension that saves your back during development

I recently wrote an article on how to generate and upload a self-signed SSL Certificate to AWS Certificate Manager. One of the cases I had missed to address there was the inconsistency of the Load Balancer and development domain names during the early phases of any project. This is a practical issue, fortunately that an extension in the SSL Certificate standard addresses.

Adding a Self-Signed SSL Certificate to AWS ACM

Tue, 17 Oct 2017 18:56:28 +0000

When setting up AWS Load Balancers (Classic Load Balancers or Application Load Balancers), after adding a HTTPS transport, an SSL Certificate should be added so that SSL termination can be done at the Load Balancer. Here, for development purposes, the certificate added can be a self-signed one.

However if you try to upload a self-signed SSL Certificate to IAM or ACM using the AWS Web Console during Load Balancer creation, you will frequently come across an error similar to the following.

Let’s make your Docker Image better than 90% of existing ones

Tue, 05 Sep 2017 18:50:34 +0000

Or why you should always Label your Docker Image.

Imagine this.

You’re working on a new project, an exciting one that’s Container native. You’re almost done, and now at the stage where you want to build the Docker Image that ships your little project out to the public. You build on top of Alpine Linux, and pack your tool in to a nifty 80Mb Image that you tag latest and push to the public Docker Hub.

Docker Image Size Concerns Out of the Window — Squash with Confidence

Tue, 22 Aug 2017 09:58:17 +0000

Docker is finally bringing Squash support

For a while, Docker image size has been an interesting topic for discussion in the Containerization world, especially things like how if you’d observe silent image size increases if you don’t use --no-cache flag while building. The layered structure of a Docker image would spook out a beginner.

Typically, these layers would be handled in such a way that maximum optimization in terms of re-use and storage is achieved. However, it’s often the case that overlaying file systems can introduce unnecessary image sizes, especially when the same file system locations are edited in different layers.

HAProxy Config Syntax Highlighting

Thu, 13 Jul 2017 05:26:55 +0000

A brief post on getting this done on Atom and Vim.

Atom

Just install language-haproxy Atom plugin through either Atom UI or apm .

apm install language-haproxy

When opening an haproxy.cfg file, Atom will detect and highlight accordingly.

Vim

Syntax highlighting in Vim for HAProxy configuration has been available for more than 10 years it seems from VimAwesome. To enable this, just clone the Github repository, copy the .vim file inside syntax/ folder to $HOME/.vim/syntax .

Infrastructure as Code : AWS CloudFormation

Mon, 03 Jul 2017 11:40:50 +0000

This blog post is based on a talk I did on AWS CloudFormation as an introduction.

The case for Infrastructure as Code

Imagine a typical deployment of a software stack. The process might look like the following.

Design solution architecture
Estimate infrastructure specs
Configure compute, network, and database resources manually
Deploy the applications and prepare databases
Test and ship

Pretty simple from the looks of it doesn’t it? The catch is this. This is just half of the solution maintenance. There are several more scenarios involved that can quickly turn this in to a complex fragile piece of engineering.

Ballerina with Container Support

Tue, 21 Feb 2017 06:48:33 +0000

WSO2 unveiled its latest contribution to the world of integration, Ballerina, yesterday during the WSO2Con 2017 in San Fransisco. Ballerina is a general purpose language with a focus on integration and a visual approach to coding. It’s strongly typed, integration friendly, and carries native support for a list of technologies such as support for REST, JSON, XML, Swagger, and “Connectors” that communicate with Facebook, and Twitter etc. This write up will focus on Docker based Containerization of Ballerina programs.

Thinking of Moving Your WSO2 Deployment On to Kubernetes?

Sun, 22 Jan 2017 19:25:15 +0000

Moving from a Virtual Machine based deployment to a Container Clustering environment can be quite a few sleepless nights. Cloud deployment artifacts for WSO2 products would reduce this number by a few, because of their tried and tested nature. These include Dockerfiles, Puppet based configurations, Kubernetes and Mesos deployment artifacts. As easy as these artifacts make life easier during a redesign stage of an existing deployment, let us anyway walk through some points to keep in mind regarding a WSO2 deployment on top of Kubernetes.

How to Upload a Carbon App (CAR File) to a Remote WSO2 Server

Sat, 17 Sep 2016 00:49:13 +0000

Carbon Apps, or CAR Files are a deployable package consisting of WSO2 Product related artifacts. These artifacts can range from Synapse Sequences, Proxy Services, Registry items, to Dashboard Server Gadgets, Dashboards, Siddhi Execution Plans, to Tasks like Purging, NTask etc. For example, to group a set of artifacts to be deployed on WSO2 ESB, a CAR file can be used that has some Synapse Sequences, Proxy Services, APIs, and Endpoints. In essence, it’s a compressed Zip file with a different extension.

Monitoring WSO2 Logs with Elasticsearch, Logstash, and Kibana (or Grafana)

Sat, 10 Sep 2016 21:26:48 +0000

I’ve recently been doing some work involving analytics dashboards and the subject proved to be really interesting. It has a wide range starting from Big Data to UX. Starting at a point where you determine what kind of data is collected and what kind information you need to get out of the dashboard, going to data analysis and data processing (probably on top an engine like Apache Spark), it finalizes at the visualization stage which is really fun.

Extracting memory and thread dumps from a running JRE based JVM

Wed, 07 Sep 2016 22:30:52 +0000

Almost every Java developer knows about jmapand jstack tools that come with the JDK. These provide functionality to extract heap and thread information of a running JVM instance. Easy.

What if there’s a running JVM that has produced a deadlock and you want to take a thread dump while the process is running? You go in and run the following.

jstack pid >> thread_dump.txt

Turns out the system doesn’t know what jstack is. You don’t panic, but you get a tiny sensation at the back of your head saying you’re not leaving early this Friday.

Running WSO2 Products on Kubernetes

Tue, 09 Feb 2016 00:00:00 +0000

Running WSO2 Products on Kubernetes

Please note that the following article has ‘expired’ in terms of accuracy when it comes to the artifacts used and the way things are done. WSO2 has made many improvements on top the configurations mentioned below and how to manipulate those artifacts might have been changed since.

It’s 2016. Kubernetes needs no introduction. Neither does WSO2, so let’s get to the point. Let’s run WSO2 Identity Server on Kubernetes!

BreadPool — a Thread Pool for Python

Thu, 10 Dec 2015 00:00:00 +0000

A thread pool is not a new concept. It’s basically a gang of worker threads to whom a task would be given to be executed. Why thread pools? Because the program wouldn’t be starting threads as it sees fit and somehow reach the maximum thread number soon. Simply said thread pools allows us to limit the number of threads spawned by our program execution. Trust me, you don’t want your code going to town spawning threads. It comes back to bite you in your behind, sooner than you think.

Timing Out of Long Running Methods in Python

Thu, 26 Nov 2015 00:00:00 +0000

Sometimes there are conditions under which a function call could not return in a needed time period and would cause unexpected behavior. For example, a file read could take more time than anticipated and leave the code execution without proper control over what to do when such a situation occurs. This can be worse if the said function call directs to an external library which we can’t control.

Python has a nifty module called signal which exposes UNIX Signal numbers and a way to register callbacks for each signal. Out of the UNIX Signals available, what interests us in this particular situation is the SIGALRM signal which allows us to sort of wind out an OS level alarm clock that would send a signal to the calling process after the set number of seconds. We can make use of this functionality (only in UNIX of course) to set a timeout before a function call with the possibility to hang or take unexpected durations to finish.

Creating a Simple ActiveMQ Master/Slave Setup

Tue, 17 Nov 2015 00:00:00 +0000

ActiveMQ is a high performing message broker, however if clustering is needed, it supports a number of methods. Out of these, the Master/Slave is a pattern where the persistence layer is shared between multiple broker instances. A Single Master broker connects to the persistence, and the rest of the Slave brokers keep waiting to attain the lock on the persistence. If the Master node goes down the lock for the persistence is released and a Slave quickly acquires it, allowing a client to continue operation without any data loss. The clients should connect to the Master/Slave setup, using the failover: transport, or they should implement a manual failover mechanism to automatically connect to the next available broker when the first one goes down.

Support for ActiveMQ Master/Slave Failover in Apache Stratos Cartridge Agent

Mon, 16 Nov 2015 00:00:00 +0000

In Apache Stratos the message broker is a crucial point of operation upon which all components depend on. Recent Stratos releases included fixes to secure the message broker communication. The upcoming 4.1.5 release will contain a missing improvement for the Python Cartridge Agent related to message broker communication.

ActiveMQ supports various types of clustering patterns. Out of these, Master/Slave is a deployment pattern where the message store is replicated or shared between the clustered brokers. This makes it possible for a client to failover from the master to the slave in case the master broker goes down, and continue the communication without any data loss.

Secure Message Broker Communication in Apache Stratos With Apache ActiveMQ

Sun, 11 Oct 2015 00:00:00 +0000

Apache Stratos relies heavily on message broker communication. In fact, message broker communication with message broker topics is the main method of communication between components such as the Cartridge Agent, Cloud Controller and the Autoscaler, as this allows a decoupled architecture for the components.

When it comes to message brokers, authentication is a crucial part of securing the communication channel since if left unsecured, anyone with access to the message broker can subscribe to the topics and listen to the communication between the components. This can expose sensitive data easily and the system would be compromised in no time. The purpose of this article is to explain how to engage Username and Password based authentication with Apache ActiveMQ in Stratos.

Apache Stratos Cartridge Agent: Life Cycle Walkthough

Sun, 22 Mar 2015 00:00:00 +0000

This is a part of a series of articles on Apache Stratos Cartridge Agent. If you feel like you’ve missed the memo, why not start from the first article?

The Cartridge Agent is (usually) the first service that starts in a spawned Cartridge instance. From that point onward, it is responsible for keeping the relevant services running, communicating with Stratos to subscribe and publish to message broker topics, processing received events, artifact distribution and health statistics reporting. In this article we will discuss in detail, in which order these tasks are performed.

Apache Stratos Cartridge Agent: Instance Configuration by Puppet 2

Sat, 21 Mar 2015 00:00:00 +0000

This is a part of a series of articles on Apache Stratos Cartridge Agent. If you feel like you’ve missed the memo, why not start from the first article?

One of the main responsibilities of the Cartridge Agent is to start the services related the Cartridge type. To do this the Cartridge Agent should be configured with proper parameters. As we discussed in the last article, Puppet can be used to install, configure and start the Cartridge Agent. In this article, we will discuss how this is done in detail.

Apache Stratos Cartridge Agent: Day 0

Tue, 17 Mar 2015 00:00:00 +0000

As the first post of a series of comprehensive guide to the Apache Stratos Cartridge Agent, let’s look at the Cartridge Agent contract. Keep tuned in and expect more detailed explanations on the instance and Cartridge Agent configuration, workflow, different Cartridge Agent implementations, their configurations and newly introduced plugin system in the Python implementation.

When the instances are spawned in Apache Stratos, there are a few requirements that the particular instance should fulfill. It should

Apache Stratos Cartridge Agent: Instance Configuration by Puppet 1

Tue, 17 Mar 2015 00:00:00 +0000

This is a part of a series of articles on Apache Stratos Cartridge Agent. If you feel like you’ve missed the memo, why not start from the first article?

When an application is deployed in Apache Stratos, what happens is that for each cartridge in the application, an instance creation call is made to the configured IaaS via the Cloud Controller component. This call contains only the base image ID (in OpenStack this is an image UUID, in Amazon EC2 this is an AMI), the instance creation parameters like the instance flavor, security group etc and the payload of Stratos related information that is targeted to that particular instance. So how does the actual configuration of a spawned Virtual Machine happen? (We will discuss the exciting Kubernetes + Docker work flow separately)

Thrift Communication in Apache Stratos

Fri, 10 Oct 2014 00:00:00 +0000

In Apache Stratos, data publishing over Thrift is done via streaming. It uses WSO2 Carbon’s Data Bridge to serialize data in to a stream and publish to a given IP address and a Port. The use of DataBridge is explained here.

Stream Definition

In order to publish data as a stream, the definition of the particular stream should be defined first. This is achieved using the org.wso2.carbon.databridge.commons.StreamDefinition class. The list of attributes that will be written to the stream is defined using a StreamDefinition object to be assigned to a particular DataPublisher.

PPPOE on VirtualBox

Tue, 05 Aug 2014 00:00:00 +0000

A quick note!

If your internet connection comes as a PPPOE connection and it can’t be selected on the drop down list when “Bridged network” is selected on Network tab of the VM settings in VirtualBox. You can create a host-only network and configure a static ip for eth0 on the guest OS to make it accessible to outside.

A host-only network virtualizes the network interface of the host OS and uses it as a loopback device to enable guest OS to communicate on their network and with the host[1].

Consuming a Service Secured by WSO2 ESB

Sat, 19 Jul 2014 00:00:00 +0000

In the last post I explained the steps needed, although somewhat minimal, to secure an unsecured backend service with WSO2 ESB. In this post I will continue on to the client side of the communication explaining the minimal client needed to communicate with the secure proxy service we created and take a peak at the action going on under the hood.

UsernameToken

Before we dive in to the client side code let’s take a look at the WS-Policy for the UsernameToken security we applied to our service.

Securing a Web Service With WSO2 ESB

Fri, 18 Jul 2014 00:00:00 +0000

WSO2 Enterprise Service Bus is one of the best performing implementations for the Enterprise market. In this article I will briefly go through what it takes to secure an unsecured backend web service using WSO2 ESB as a mediator.

Security in Web Services is covered by the WS-Security standard. There are various policies such as simple username and password authentication and PKI certificates that can be used to secure a Web Service. The policy that will be used is described using the WS-Policy standard.

Wireshark Crashing in Ubuntu

Wed, 09 Jul 2014 00:00:00 +0000

If you have customized your Ubuntu GTK theme, Wireshark would most likely crash as soon as scrollbars come in to play. I experienced this in Ubuntu 14.04 and found the solution to be disabling overlay scrollbars that come as the default.

To do this simply add an environment variable called LIBOVERLAY_SCROLLBAR with value0.

export LIBOVERLAY_SCROLLBAR=0

Originally published at chamilad.github.io on July 9, 2014.

Written on July 9, 2014 by chamila de alwis.

Creating a Web Service Using Apache Axis2

Tue, 01 Jul 2014 00:00:00 +0000

Prerequisites

We need Apache Axis2 running and a suitable container. You could also run Axis2 as a standalone server but for this let’s use Tomcat as a container. So in the following order

Download and install Oracle JDK7
Download and install Tomcat 7 (even 8 would do, but it seems as of now Tomcat 8 isn’t being supported by Eclipse, so let’s go with version 7)
Download and install Axis2 on Tomcat

Oracle JDK7

Download and extract the JDK tarball.