Nos últimos dois meses, várias pessoas me procuraram para saber como organizo meus estudos. Decidi, então, compilar um resumo das minhas técnicas e métodos. Espero que encontrem aqui ferramentas úteis!

Método de estudos

Primeiramente, o método de estudo é algo muito pessoal. As práticas que mencionarei tiveram êxito para mim. A primeira e mais importante tarefa é começar a desenvolver o seu método:

• Com que frequência você deseja estudar?
• Como você vai organizar o conhecimento?
• Que tipo de conteúdo mais te engaja (livros, vídeo aulas, cursos, etc)?
• Quanta estrutura você quer dar aos seus estudos? Você funciona melhor com um planejamento detalhado e prazos, ou prefere algo mais flexível, seguindo os tópicos que mais capturam sua atenção no momento?

Posso falar por mim: estudo todos os dias, leio artigos ou livros sempre que possível. Organizo meu conhecimento em notas no Obsidian, usando uma estrutura que combina o Zettelkasten com o método PARA. Prefiro, de longe, conteúdos escritos e não gosto de fazer grandes planejamentos; estudo o que tenho vontade, quando tenho vontade.

Fontes de informação

Explico que existem duas formas de desenvolvimento: horizontal e vertical.

• O desenvolvimento horizontal foca em criar uma base de conhecimento em um universo, abrangendo vários tópicos. Isso não implica em um estudo superficial, mas em entender como os tópicos se relacionam em um contexto. Dependendo do universo, como em segurança da informação, esse tipo de estudo pode ser um dos mais desafiadores.
• O desenvolvimento vertical, por sua vez, foca em aprofundar um tópico e aprender o máximo possível sobre ele. O objetivo é estudar como se quisesse se tornar um especialista naquele assunto.

Na carreira, é essencial manter ambos os tipos de desenvolvimento ativos. Para isso, recomendo realizar uma curadoria constante de diferentes tipos de fontes de informação:

• Newsletters (horizontal): para mim, são a melhor ferramenta para um profissional de tecnologia hoje. Faço uma seleção contínua das newsletters que assino. Ao receber uma nova publicação, seleciono links relevantes, adiciono-os à minha lista de leitura e os consumo gradualmente.
• Feeds de agregadores (horizontal): sites como Hackersnews e Lobste.rs têm feeds RSS organizados por tema. Funcionam como uma linha do tempo de uma rede social, com muito ruído, mas ocasionalmente você encontra artigos úteis. É crucial saber gerenciar fontes de alto e baixo ruído para que não interfiram na eficiência do seu consumo.
• Blogs/Publicações especializadas (vertical): prefiro separar essas fontes dos livros tradicionais porque geralmente são mais sucintas, têm mais exemplos e exercícios práticos e são mais atualizadas que os livros. Exemplos incluem: https://quii.gitbook.io/learn-go-with-tests e https://www.fuzzingbook.org/.
• Papers (vertical): começar a ler artigos acadêmicos pode ser desafiador, mas é um passo importante para conhecer a vanguarda do desenvolvimento. Quanto antes começar, melhor.
• Livros e cursos (vertical): geralmente os utilizo para tópicos que evoluem mais lentamente, como algoritmos, matemática e criptografia. Nessas áreas, você encontrará publicações excelentes e ainda relevantes.

Com o tempo, ao consumir conteúdos, você acumulará links, artigos, livros, ferramentas: todo tipo de referência. Cuide bem dessa coleção, escolha uma forma de organização, um aplicativo, o que preferir. Essa coleção se tornará seu jardim, de onde extrairá insumos para produzir conhecimento.

No entanto, é fundamental não cair na falácia do colecionador, onde simplesmente acumulamos pilhas de materiais e nos damos por satisfeitos.

Organizando conhecimento

Um dos meus maiores arrependimentos foi não ter começado a manter notas organizadas mais cedo! Minha maior recomendação aqui é a simplicidade. É muito fácil cair na armadilha de otimizar a estrutura das suas notas e ser extremamente produtivo (eu já caí várias vezes), criando um sistema demasiadamente complicado.

Algumas coisas que faço para estruturar o conhecimento são:

• Notas de Literatura: fazer notas detalhadas e com suas próprias palavras custa muito tempo, então eu criei um tipo diferente de nota chamada Nota de Literatura. Nela, deposito informações relevantes da fonte de informação, por vezes copiando diretamente, outras vezes reescrevendo.
  • Assistente AI: embora o objetivo seja produzir notas mais simples e rápidas, a AI tem me ajudado a produzir notas de literatura com qualidade de notas de assunto. Eu uso um prompt que gera uma lista de perguntas sobre os tópicos abordados na fonte de informação. Isso tem tornado minhas notas de literatura muito mais eficientes. O prompt que uso está disponível publicamente: thought_provoking prompt.
• Notas de Assunto (Subject/Evergreen notes): essas notas são o personagem principal. Elas têm uma vida longa, são atualizadas sempre que aprendo algo novo sobre o assunto e são escritas com minhas próprias palavras. A técnica mais importante para mim aqui é escrever como se estivesse explicando para outra pessoa.

Aplicando seu método

Existem ainda desafios únicos durante o andamento dos seus estudos. Uma pergunta recorrente é: eu sei qual assunto quero estudar, sei qual tipo de estudo (horizontal ou vertical) e tenho uma curadoria de fontes - como escolher qual material consumir? Nesse caso, meu método é:

1. Seleciono material relacionado ao tema que já possa ter armazenado;
2. Uso DORKs para pesquisar material no histórico de newsletters e feeds de agregadores.
3. Dedico um tempo navegando entre os diferentes materiais selecionados, até encontrar um (ou alguns) com um bom equilíbrio entre didática e cobertura do assunto.

Isso acaba levando a um começo mais lento, porém, na minha experiência, ele se paga, pois me dá a oportunidade de selecionar materiais de estudo de maior qualidade.

Outro desafio comum é: como tirar dúvidas sobre o assunto? Atualmente, LLMs têm sido de grande ajuda para mim (desde que você saiba como lidar com possíveis alucinações). Buscar por comunidades dedicadas ao assunto no Reddit também costuma ter bons resultados. No entanto, a melhor ferramenta é sua própria coleção de materiais: busque por outros materiais que possam explicar o mesmo conceito a partir de diferentes pontos de vista. Eventualmente, você irá montar um quebra-cabeça, com uma compreensão muito mais abrangente do assunto.

Conclusão

A jornada do aprendizado é profundamente pessoal e varia de indivíduo para indivíduo. As estratégias e métodos que compartilhei refletem minhas experiências e preferências pessoais. É importante encontrar o equilíbrio entre a absorção de informação e a criação de conhecimento, evitando a armadilha de apenas acumular referências.

Lembre-se de que a chave para um estudo eficaz não está apenas em selecionar os materiais certos, mas também em como você organiza e interage com essas informações ao longo do tempo. Espero que este guia sirva como um ponto de partida para você desenvolver um método de estudo que seja tão único e eficaz quanto suas próprias aspirações e necessidades de aprendizado.

Over the past two months, several people have approached me to find out how I organize my studies. Therefore, I decided to compile a summary of my techniques and methods. I hope you find useful tools here!

Study Method

Firstly, the study method is something very personal. The practices I will mention have been successful for me. The first and most important task is to start developing your method:

• How often do you want to study?
• How will you organize knowledge?
• What type of content engages you the most (books, video lessons, courses, etc)?
• How much structure do you want to give your studies? Do you function better with detailed planning and deadlines, or do you prefer something more flexible, following the topics that capture your attention at the moment?

I can speak for myself: I study every day, read articles or books whenever possible. I organize my knowledge in notes in Obsidian, using a structure that combines the Zettelkasten method with the PARA method. I vastly prefer written content and do not like making extensive plans; I study what I want, when I want.

Sources of Information

I explain that there are two forms of development: horizontal and vertical.

• Horizontal development focuses on creating a knowledge base in a universe, covering various topics. This does not imply superficial study but understanding how the topics relate in a context. Depending on the universe, such as in information security, this type of study can be one of the most challenging.
• Vertical development, on the other hand, focuses on deepening a topic and learning as much as possible about it. The goal is to study as if you wanted to become an expert on that subject.

In your career, it is essential to keep both types of development active. For this, I recommend constantly curating different types of information sources:

• Newsletters (horizontal): for me, they are the best tool for a technology professional today. I make a continuous selection of the newsletters I subscribe to. When I receive a new publication, I select relevant links, add them to my reading list, and consume them gradually.
• Feed aggregators (horizontal): sites like Hackersnews and Lobste.rs have RSS feeds organized by theme. They work like a social network timeline, with a lot of noise, but occasionally you find useful articles. It is crucial to know how to manage high and low noise sources so that they do not interfere with the efficiency of your consumption.
• Blogs/Specialized publications (vertical): I prefer to separate these sources from traditional books because they are usually more concise, have more examples and practical exercises, and are more up-to-date than books. Examples include: https://quii.gitbook.io/learn-go-with-tests and https://www.fuzzingbook.org/.
• Papers (vertical): starting to read academic papers can be challenging, but it is an important step to know the cutting edge of development. The sooner you start, the better.
• Books and courses (vertical): I generally use them for topics that evolve more slowly, such as algorithms, mathematics, and cryptography. In these areas, you will find excellent publications that are still relevant.

Over time, as you consume content, you will accumulate links, articles, books, tools: all kinds of references. Take good care of this collection, choose a way to organize it, an app, whatever you prefer. This collection will become your garden, from which you will draw inputs to produce knowledge.

However, it is crucial not to fall into the collector’s fallacy, where we simply accumulate piles of materials and are satisfied with that.

Organizing Knowledge

One of my biggest regrets was not starting to keep organized notes earlier! My biggest recommendation here is simplicity. It’s very easy to fall into the trap of optimizing your note structure and being extremely productive (I’ve fallen several times), creating an overly complicated system.

Some things I do to structure knowledge are:

• Literature Notes: making detailed notes in your own words takes a lot of time, so I created a different type of note called Literature Note. In it, I deposit relevant information from the source of information, sometimes copying directly, other times rewriting.
  • AI Assistant: although the goal is to produce simpler and quicker notes, AI has helped me produce literature notes with the quality of subject notes. I use a prompt that generates a list of questions about the topics covered in the source of information. This has made my literature notes much more efficient. The prompt I use is publicly available: thought_provoking prompt.
• Subject Notes (Subject/Evergreen notes): these notes are the main character. They have a long life, are updated whenever I learn something new about the subject, and are written in my own words. The most important technique for me here is writing as if I were explaining to someone else.

Applying Your Method

There are still unique challenges during the course of your studies. A recurring question is: I know which subject I want to study, I know what type of study (horizontal or vertical) and I have a curation of sources - how to choose which material to consume? In this case, my method is:

1. I select material related to the theme that I may have already stored;
2. I use DORKs to search for material in the newsletters’s archives and feed aggregators.
3. Then, I dedicate some time navigating among the selected materials, until I find one (or a few) with a good balance between didactic approach and subject coverage.

This ends up leading to a slower start, but in my experience, it pays off, as it gives me the opportunity to select higher quality study materials.

Another common challenge is: how to ask questions about the subject? Currently, LLMs have been of great help to me (as long as you know how to deal with possible hallucinations). Searching for dedicated communities on Reddit often yields good results. However, the best tool is your own collection of materials: look for other materials that may explain the same concept from different perspectives. Eventually, you will put together a puzzle, gaining a much more comprehensive understanding of the subject.

Conclusion

The learning journey is deeply personal and varies from individual to individual. The strategies and methods I have shared reflect my experiences and personal preferences. It’s important to find a balance between absorbing information and creating knowledge, avoiding the trap of just accumulating references.

Remember, the key to effective study is not only in selecting the right materials but also in how you organize and interact with these materials over time. I hope this guide serves as a starting point for you to develop a study method that is as unique and effective as your own learning aspirations and needs.

This is my extension to the problems and recommendations presented on the paper Dos and Don’ts of Machine Learning in Computer Security (Quiring, et al, 2022). I encourage you to also read the paper, as it’s excellent and provide a lot of insights about how to better build machine learning models.

Machine learning workflow

Data collection

Pitfalls: Sampling Bias, Label Inaccuracy

As with any other domain, Security is also heavily affected by bad data quality used for training. However, it’s often worse in Security because data acquisition of adversary activity is usually hard and the method’s used to acquire it will bring a bias.

Let’s take for example a honeypot implemented with a vulnerable Apache Server. Even though there are lots of bad actors in the wild, if you have a medium size environment, the volume of data they will produce attacking your honeypot will not come near to the volume of packets in your production network. Also, the TTPs you are going to see will come from threat actors that are used to leverage Apache Server in their kill chain, possibly leaving other adversaries, that may be focusing on other types of vectors, producing different kill chains, out of the dataset.

On top of that, we are usually dealing with lots of examples. So, if we have label inaccuracies, dealing with them is a lot harder. We can’t just apply common techniques like using the mode to fill in, because in the Security domain, the adversary is actively trying to mimic the distribution of the benign cases. Therefore, we have very little space for noise, as it would blur even more the distinction between the outcomes.

Besides the recommendations presented in the article, I would add two more:

• User open and disseminate datasets whenever possible. Dataset sharing is still uncommon on the Security community. We already have some network and host datasets, but there are many more assets nowadays (logs for cloud, kubernetes, CDN, CI/CD, etc), and the technologies on networks and hosts are constantly evolving, posing the need for these datasets to be always updated.
• Use a model design that depends less on adversary data, such as we discussed in the previous article. This will reduce the dependency on the adversary behavior and increase the sources of data available to use.

Model design and implementation

Pitfalls: Data Snooping, Spurious Correlations, Biased Parameter Selection

Developing machine learning models is no easy task. Feature engineering, hyperparameters optimization, data preparation and appropriate splitting for validation (to avoid snooping). These are just some of the challenges when taking on this endeavor.

Using tools like AutoML may help reduce the burden, however it won’t take care of everything. In the end, you still need to understand your data characteristics and how it’s related to your problem.

But, some of these relations and characteristics tend to repeat. Time relations, for example, are extremely common and important in a lot of Security problems. Therefore, I suggest that every Security team doing machine learning on its own to think about how they can extract such aspects into reusable components.

This has the added benefit to scale the impact of the Security members that are more focused on implementing machine learning. Our domain has many different areas and enabling other teams and specialists to more easily and correctly implement models, even as proofs of concept, expands the possibilities of what the enterprise can achieve.

In the end, the best strategy to best tackle all these challenges and be more prepared to handle pitfalls it to avoid jumping to complex and sexy algorithms from the start, such as neural networks.

Using explanation techniques, as the article suggests, can help you catch spurious correlation, but even better is to use simpler and more understandable methods. I have been seen great results with simple probabilistic methods, such as with Histogram Based Outlier Score. Using a simple algorithm, we can immediately see what is driving its decision and catch faster these types of pitfalls.

Performance evaluation

Pitfalls: Inappropriate Baseline, Inappropriate Performance Measures, Base Rate Fallacy

As I mentioned before, most times when building a learning-based system in Security, we end up with imbalanced and noisy datasets. This demands a special care when choosing performance metrics to best reflect our model. Building on the previous suggestion, having common components that use by default metrics more fit to most problems in Security, such as precision, recall, and MCC, would reduce the change of human error.

However, even after computing correct performance metrics, we still need good baselines to compare them against. The article suggest goods options such as using simple methods or automated machine learning. I would add that comparing it with vendor products can also be a good experiments to fully understand the impact of replacing the third party with the model.

Operation

Pitfalls: Lab-Only Evaluation, Inappropriate Threat Model

Although the article identifies Lab-Only Evaluation as a common pitfall in Security model, I think this is one of the topic where in Security we have the most options to address.

First, we could use adversary emulation frameworks and popular attack toolkits to launch real world attacks against a test environment and study the model response. Another option would be to set up a honeypot and evaluate how well the model performed by comparing to a human analyses of the evidences afterward.

This type of experiment would also feed back into our threat model. Given the hype of AI in latest months, more knowledge is beings shared and produced about security best practices and process for machine learning, however we can still consider the topic in its infancy.

One special vector that we are still starting to discuss is supply-chain attacks to models, specially with the popularization of transfer-learning for more complex algorithms such as LLMs.

Splunk showed that more than 80% of HuggingFace’s models use pickle-serialized code, which is vulnerable to arbitrary code execution and code injection, although it’s not possible to say if any of them is malicious.

But, besides low-level vulnerabilities like this, a transfer-learning based model can also inherit the biases (intentionally placed or not) from the original model. That is, if the original model is vulnerable to an adversarial example, there is a high risk that your new model is going to also be.

This opens the possible for two types of attacks:

1. Malicious models: base models shared with adversarial examples trained, such as a large batch of images with a negative label having a red square, making the model learn that any image with a red square should be classified as negative, therefore implanting a bypass. This type of attack would be extremely difficult to detect.
2. Cross model generalization: as Szegedy, et all points in their paper, different models may be susceptible to the same adversarial examples, even when having different hyperparameters. In this case, the models would not even be so different, therefore an adversary could search for adversarial examples against the original model and just use them in the target system, with a high chance of success.

Conclusion

In conclusion, implementing learning-based systems in the security domain presents challenges that require careful consideration. Addressing issues related to data collection, model design and implementation, performance evaluation, and operational considerations is crucial. By implementing these recommendations and reflecting on these problems, organizations can enhance their capabilities in security.

To learn this function, we use a set of example data points composed of inputs (also called features) and outcomes (sometimes called labels). These example data points were sampled from the real world behavior, i.e. from the function $f$, at some time in the past. Our goal is that we can extract knowledge from the past to figure out this behavior on unseen data beforehand. This knowledge extracted from the set of examples is materialized on the $h$ function.

In Security, we can imagine some examples where this would be useful, like trying to learn if an HTTP request contains malicious payload or if some set of bytes is a malware or not. However, supervised learning is less often used in Security than in many other domains.

This happens because the principles of the supervised learning theory conflicts with the nature of Security, limiting its application. But, by understanding these principles, it is also possible to see how to best apply this technique and how it can maybe useful.

Stationary assumption

The most important principle in supervised learning is the stationary assumption. When the data that represents the real world behaviors follows the stationary assumption, it means that predicting the behavior using past example is approximately correctly.

The stationary assumption states that the behavior that is being learned don’t change through time. This has some important consequences:

1. We expect that each data point is independent of each other. This is important, because if there were causal effects between data points, then the features of a data point $x_1$, caused by $x_0$, would vary with a probability that is a combination of the probability distribution and the effect of $x_0$, hence the distribution would not remain the same over time, because $x_0$ and $x_1$ would vary in different ways. For example, in a box with 1 blue ball and 2 red balls, the probability of picking a blue ball when drawing the first one from the box is 33% while a red one would be 66%. However, if the first one is indeed blue, then the probability of drawing a red ball as the second one is 100%. The first data point (blue ball draw) changed the probability of the second data point.
2. We expect that each data point is identically distributed, i.e. each data point should be drawn from the same probability distribution. We could learn the shopping behavior using data ranging from Black Friday to New Years, however the users’ behavior in this time is completed different from the rest of the year, therefore the data used to learn has a different probability distribution than the unseen data on which we are going to make predictions.

Any dataset that follows these two characteristics is said to hold the i.i.d assumption (independent and identically distributed). The importance for our training datasets on supervised learning to be i.i.d is because it connects the past to the future, without it, any inference made on the available data would be invalid.

Understand the i.i.d assumption is specially important for Security Machine Learning because it is one of the areas where causality and behavior shifts are most present. So, exactly how this assumption affects our ability to do Supervised Machine Learning?

Causality

Many threat behaviors have causal nature, and therefore we should have a lot of care when preparing our datasets and choosing our validation methods.

A good example is malware classification, where you could have many samples from various families from different years. Each family generation influences each other, and sometimes they have similar characteristics. A special bad situation that could happen with this is that during splitting of the dataset between training and testing, without taking into account the time relation of the families and samples, then you could end up training the model with future information and testing against past samples. This would produce falsely accurate results that would not generalize in the real-world.

There are ways to deal with this, but it depends on the type and strength of the causal relation between the data. For this case, ensuring that the newest malware samples are used for test should be enough.

Behavior Shift

We could create a model to learn a threat behavior like the profile of a botnet, however once we started responding effectively to it, adversaries would adapt and our model would become useless because the new botnets would have a totally different behavior. This would be the case of a change in the probability distribution from which the features are drawn, leading to the break of our assumption.

Looking to the other side

Although causality may be addressed by good data preparation, preventing a model to be become outdated due to behavioral shift is almost impossible. However, this problem isn’t a new one in Security, such that one best practices is to instead of trying to detect and block malicious action, defenders should define what a legitimate system behavior looks like and block everything else. This can be summarized as: allow lists are more secure than block lists.

We can apply the same philosophy for supervised learning, by modeling profiles of legitimate behavior, which usually are more stable. Then, new data points are classified against these multiple profiles models, finally a meta-classifier is used to choose which one is the best fit or if it’s an outlier. This has the ability to catch any new threat behavior that deviates from the know legitimate profiles.

This combination of multiple models is called ensemble learning, which has shown to improve models performance, like when comparing a Decision Tree model to a Random Forest one.

As a side bonus, since this way of building models does not depend on knowing threat behaviors, it avoids the common problem in mining data for Security that usually produce highly unbalanced datasets. We can train each profile using the true positive data of others profiles as false examples, assuming each profile is mutually exclusive.

The main challenge in this approach is that adversaries may try to mimic legitimate profiles, however like the authors of Notos showed, this can be useful sometimes, as in their cases this would imply in adversaries using a more stable network infrastructure that would be easily defeated by static block lists. Therefore, forcing adversaries to mimic a legitimate profile could also reduce their capabilities.

Conclusion

In conclusion, supervised learning is a powerful technique that allows us to extract knowledge from past data points to predict future behavior. However, in the Security domain, applying it comes with unique challenges due to the presence of causality and behavior shifts. Understanding the stationary assumption and the importance of having an i.i.d dataset is crucial, because it informs us how to best prepare our datasets, choose our validation methods and, most importantly, what are the best behaviors to be modeled using supervised learning.

Luckily, by modeling profiles of legitimate behavior, and catching new threat behavior that deviates from known legitimate profiles, we can build intelligent allow lists.

While there are challenges, with proper preparation and understanding, supervised learning can be a valuable tool in Security.

References

• Notos: Building a Dynamic Reputation System for DNS

A common requirement that I face on multiple projects is to safeguard some API endpoints to administrative access, or to provide a secure way for other applications to consume our service in a controlled and traceable manner.

The usual solution for it is API Keys, a simple and effective authorization control mechanism that we can implement with a few lines of code. However, when doing, so we also need to be aware of threats and possible attacks that we may suffer, specially due to the usual privileges that these keys provides.

Therefore, we are going to analyze common points of concern and design a solution that improve our security posture while keeping it simple.

API Keys threats

There are two main concerns when implementing an API Key authorization scheme: key provisioning and timing attacks. Let’s review each threat before designing solutions to address them.

Key Provisioning

The key storage is directly related to how applications expect these secrets to be provided to them. Environment variables are the most common solution used on modern services since they are widely supported and don’t incur a high reading cost (in contrast to files) allowing for dynamic changes to be easily detected.

However, the way developers usually define the environment variables are through scripts or configuration files, for example using a Kubernetes Secret manifest. This introduces a serious threat of API Keys being committed to git repositories, which in the event of data leakage from the internal VCS management system would expose these credentials.

Note: remember that once committed, even if the keys are deleted from the source files, the information is already on the repository history and is easily searchable with tools like TruffleHog.

Therefore, please do not commit your API Keys to git!

Timing Attacks

Once your application is configured with the available API Keys, you need to verify that the end-user provided key (let’s call this the user key) is correct. Doing so with a naive algorithm, like using == operator, will make the verification end on the first incorrect character, hence reducing the time taken to respond.

A timing attack takes advantage of this scenario by trying to guess the correct characters of a secret based on how long the application took to respond. If the guess is right, the response will take slightly longer than if it’s wrong.

Naturally, since equality checks are orders of magnitude faster than the network roundtrip, this type of attack is extremely difficult to perform because it depends on a statistical analysis of many response samples. By looking at the time distribution produced by two different characters, one can infer that if they are different, inferring that the greater one is the correct value. For an extensive discussion of statistical techniques to help perform this attack see Morgan, Morgan 2015.

Middleware design and implementation

Having these threats in mind, we can design a suitable solution. Let’s start with the most simple API Key middleware implementation possible and iterate from it.

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) func(handler http.Handler) http.Handler {
	apiKeyHeader := cfg.APIKeyHeader // string
	apiKeys := cfg.APIKeys // map[string]string

	reverseKeyIndex := make(map[string]string)
	for name, key := apiKeys {
		reverseKeyIndex[key] = name
	}

	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			apiKey, err := bearerToken(r, apiKeyHeader)
			if err != nil {
				logger.Errorw("request failed API key authentication", "error", err)
				RespondError(w, http.StatusUnauthorized, "invalid API key")
				return
			}

			_, found := reverseKeyIndex[apiKey]
			if !found {
				hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
				if err != nil {
					logger.Errorw("failed to parse remote address", "error", err)
					hostIP = r.RemoteAddr
				}
				logger.Errorw("no matching API key found", "remoteIP", hostIP)

				RespondError(w, http.StatusUnauthorized, "invalid api key")
				return
			}


			next.ServeHTTP(w, r)
		})
	}
}


// bearerToken extracts the content from the header, striping the Bearer prefix
func bearerToken(r *http.Request, header string) (string, error) {
	rawToken := r.Header.Get(header)
	pieces := strings.SplitN(rawToken, " ", 2)

	if len(pieces) < 2 {
		return "", errors.New("token with incorrect bearer format")
	}

	token := strings.TrimSpace(pieces[1])

	return token, nil
}

A middleware is a function that takes an http.Handler and returns an http.Handler. In this code, the function ApiKeyMiddleware is a factory that creates an instance of the middleware with the provided configuration and logger. The config.Config is a struct populated from environment variables and logging.Logger is an interface that can be implemented using any logging library or the standard library. You could pass only the header and map of keys, but for clarity we choose to denote the dependency from this middleware to the configuration.

After extracting the fields that it relies on, the function creates a reverse index of the API Keys, which is originally a map from a key id/name to the key value. Using this reverse index it’s trivial to verify if the user key is valid by doing a map lookup on line 18.

However, this approach expects the API Keys as plaintext values and is susceptible to timing attacks, because its validation algorithm is not constant time.

Using key hashes for validation

To improve the key provisioning workflow, we can use a simple yet effective solution: expect the available keys to be hashes. Using this approach we can now commit our key hashes to our repository because even in the event of a data leak they could not be reversed to their original value.

Let’s use the SHA256 hashing algorithm to encode our keys. For example, if one of them is 123456789 (please, do not use a key like this :D) then its hash will be:

15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225

Now you can add this hash to your deployment script, Kubernetes Secret, etc., and commit it with peace of mind.

Next, we need to handle this new format on our middleware. This is what the code will look like now:

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) func(handler http.Handler) http.Handler {
	apiKeyHeader := cfg.APIKeyHeader // string
	apiKeys := cfg.APIKeys // map[string]string

	reverseKeyIndex := make(map[string]string)
	for name, key := apiKeys {
		reverseKeyIndex[key] = name
	}

	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			apiKey, err := bearerToken(r, apiKeyHeader)
			if err != nil {
				logger.Errorw("request failed API key authentication", "error", err)
				RespondError(w, http.StatusUnauthorized, "invalid API key")
				return
			}

			_, ok := apiKeyIsValid(apiKey, reverseKeyIndex)
			if !ok {
				hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
				if err != nil {
					logger.Errorw("failed to parse remote address", "error", err)
					hostIP = r.RemoteAddr
				}
				logger.Errorw("no matching API key found", "remoteIP", hostIP)

				RespondError(w, http.StatusUnauthorized, "invalid api key")
				return
			}


			next.ServeHTTP(w, r)
		})
	}
}

// apiKeyIsValid checks if the given API key is valid and returns the principal if it is.
func apiKeyIsValid(rawKey string, availableKeys map[string][]byte) (string, bool) {
	hash := sha256.Sum256([]byte(rawKey))
	key := string(hash[:])

	name, found := reverseKeyIndex[apiKey]

	return name, found
}


// bearerToken function omitted..

Here we extracted the logic to validate the key into a function that, before checking the equality of the user key against the available ones, encodes the user key using the same SHA256 algorithm.

This simple step improved a lot our security posture without adding much complexity. Now we can have the benefits of version control, like change history and easy detection when someone changes a key hash.

This approach works well when there are few keys to be managed, and you want to follow a GitOps approach. However, if you need to scale the key management, allow for self-service key requests and automatic rotation, you may want to look for a solution like Hashicorp Vault. Even using an external secret store I still believe this strategy, to rely on key hashes to be valid, because your external secret store can persist both the original key and the hash, and the access policy for the application can have fewer privileges in such a way that it can only read the hashes.

Constant time key verification

Once we have a better strategy to provision our keys, we need to defend ourselves against them being exfiltrated by timing attacks. The solution for this kind of vulnerability is to use an algorithm that takes the same time to produce a result whether the keys are equal or not. This is called a constant time comparison, and the Go Standard Library offers us an implementation in the crypto/subtle package that is perfect to solve most of our problems. Hence, we can update our code to use this package:

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) (func(handler http.Handler) http.Handler, error) {
	apiKeyHeader := cfg.APIKeyHeader
	apiKeys := cfg.APIKeys
	apiKeyMaxLen := cfg.APIKeyMaxLen

	decodedAPIKeys := make(map[string][]byte)
	for name, value := range apiKeys {
		decodedKey, err := hex.DecodeString(value)
		if err != nil {
			return nil, err
		}

		decodedAPIKeys[name] = decodedKey
	}

	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			ctx := r.Context()

			apiKey, err := bearerToken(r, apiKeyHeader)
			if err != nil {
				logger.Errorw("request failed API key authentication", "error", err)
				RespondError(w, http.StatusUnauthorized, "invalid API key")
				return
			}

			if _, ok := apiKeyIsValid(apiKey, decodedAPIKeys); !ok {
				 hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
					if err != nil {
						logger.Errorw("failed to parse remote address", "error", err)
						hostIP = r.RemoteAddr
					}
					logger.Errorw("no matching API key found", "remoteIP", hostIP)

					RespondError(w, http.StatusUnauthorized, "invalid api key")
					return
			}

			next.ServeHTTP(w, r.WithContext(ctx))
		})
	}, nil
}

// apiKeyIsValid checks if the given API key is valid and returns the principal if it is.
func apiKeyIsValid(rawKey string, availableKeys map[string][]byte) (string, bool) {
	hash := sha256.Sum256([]byte(rawKey))
	key := hash[:]

	for name, value := range availableKeys {
		contentEqual := subtle.ConstantTimeCompare(value, key) == 1

		if contentEqual {
			return name, true
		}
	}

	return "", false
}

// bearerToken function omitted...

Now, the function apiKeyIsValid uses subtle.ConstantTimeCompare to verify the user key against each available key. Since subtle.ConstantTimeCompare operates upon byte slices we don’t cast our hash to string anymore and also our reversed index has gone in place of a decoded map.

The decoding is necessary because the string representation of our key hashes are actually a hexadecimal encoding of the binary value. Hence, we cannot just cast the string to byte slice because Go assumes all strings to be UTF-8 encoded.

Note: for an example on how using a cast instead of the correct decoding function, the result of []byte("09") is 110000111001 while hex.DecodeString("09") produces 1001. Check out the live example here.

The major disadvantage of this solution is that now we need to iterate over all available keys before finding out if the key is incorrect. This doesn’t scale well if there are too many keys, however one simple workaround would be to require the client to send an extra header with the key ID/name, e.g. X-App-Key-ID, with which you can find the key in O(1) and then apply the constant time comparison.

However, there is one subtle (pun intended) behavior from subtle.ConstantTimeCompare that we must be aware before deploying our solution to production. When the byte slices have different lengths, the functions returns earlier without performing the bitwise operations. This is natural because it does an XOR between each pair of bits from each slice, and with slices of different sizes, there would be bits from one slice without a matching pair to be combined with. Because of it, an adversary could measure that keys with the wrong length have a smaller response time than keys with the correct length, hence leaking the key length. It would only be a vulnerability if you use a short key that is easily brute-forced, but with a simple 30 character key using the UTF-8 printable characters you would have 30^95 = 2.12089515 × 10^140 possible keys.

Finally, we’ve built a simple, secure and efficient API Key solution that should handle a lot of uses cases without additional infrastructure or complexity. Using a basic understanding of threats and the Golang standard library, we could do a security-oriented design instead of leaving security as an after-though in an iterative way.

Photo by Silas Köhler on Unsplash.

Introduction

Kubernetes won the battle for the cloud-native platform and the characteristic that makes me enjoy the most working with it is its extensibility. By providing an open model through the kube-apiserver, without splitting an internal and external interface, we can interact with the cluster and any other system to integrate both from the same application (Controller) and even use custom resources to describe our unique operations, know as the Operator Pattern.

Although one could use any HTTP Client to interact with the API Server, this is no simple task. There are many resources with different response structures and possible operations if we only consider the core resources on Kubernetes. Hence, Kubernetes itself provides a set of clients for easier integration through the k8s.io/client-go project.

The most used client provided by this project is the k8s.io/client-go/kubernets.ClientSet, which is a typed client. What that means is that this interface provides exclusive methods for each resource on Kubernetes (think of Pods, Deployments, Services, everything!) and operation (Create, Get, List, Watch, Update, Patch and Delete). It is obvious why you should, whenever possible, prefer to use this client.

However, there are situations where this can be limiting. It is when k8s.io/client-go/dynamic.Interface, the dynamic client, will enter the game. This client has a twofold purpose:

First, it allows working with custom resources while avoiding strong dependencies. If you want to build some automation or workflow using another Operator as the building block, like ExternalDNS, CertManager, or Prometheus (Operator) usually you would need to add these projects as dependencies to use their Go types and register them on your client instance. This obviously introduces a lot of burdens as you now need to manage their ever-evolving versions and try to keep the version you have installed on the cluster matching the version on your go.mod.

Secondly, you can work with multiple or unknown resources. When your operator implements a generic logic that can interact with any common Kubernetes resource (from RBAC to Pods) and even custom resources, the dynamic client may be your only solution. A few examples are the garbage collection controller relies heavily on it and if you would want to add support for an arbitrary custom resource on a project like KubeWatch.

Therefore, let’s dive into this resourceful (pun intended) component of the k8s.io/client-go project and see how we can leverage it.

Basic operations with the dynamic client

The code below assumes to be running inside a Kubernetes cluster.

Many operations with the dynamic client is similar to the typed client, like creating a new instance can be done by providing the config to its constructor:

func newClient() (dynamic.Interface, error) {
	config, err := rest.InClusterConfig()
	if err != nil {
		return nil, err
	}

	dynClient, err := dynamic.NewForConfig(config)
	if err != nil {
		return nil, err
	}

	return dynClient, nil
}

Since the dynamic client has no knowledge about the resource you want to consume, it does not provide helper methods like CoreV1().Pod . Instead, you need to first provide a schema.GroupVersionResource, which is a Golang type that provides the necessary information to construct an HTTP request to the cluster API Server.

For example, if you want a function to list all MongoDB resources from the MongoDB Community Operator:

var monboDBResource = schema.GroupVersionResource{Group: "mongodbcommunity.mongodb.com", Version: "v1", Resource: "mongodbcommunity"}

func ListMongoDB(ctx context.Context, client dynamic.Interface, namespace string) ([]unstructured.Unstructured, error)  {
	// GET /apis/mongodbcommunity.mongodb.com/v1/namespaces/{namespace}/mongodbcommunity/
	list, err := client.Resource(monboDBResource).Namespace(namespace).List(ctx, metav1.ListOptions{})
	if err != nil {
		return nil, err
	}

	return list.Items, nil
}

Note that if you are dealing with a namespaced resource then .Namespace(namespace) is obligatory, even if you will use an empty string to list on all namespaces.

In this snippet, we can see the main companion of the dynamic client: unstructured.Unstructured. This is a special type that encapsulates an arbitrary JSON while also complying with standard Kubernetes interfaces like runtime.Object , but most importantly it provides a set of helpers on the unstructured package to manipulate this data.

Expanding our example, if we would scale a MongoDB by an proportion we could do so like:

// ScaleMongoDB changes the number of members by the given proportion,
// which should be 0 =< proportion < 1.
func ScaleMongoDB(ctx context.Context, client dynamic.Interface, name string, namespace string, proportion uint) error {
	if proportion > 1 {
		return fmt.Errorf("proportion should be between 0 =< proportion < 1")
	}

	mongoDBClient := client.Resource(monboDBResource).Namespace(namespace)
	mdb, err := mongoDBClient.Get(ctx, name, metav1.GetOptions{})
	if err != nil {
		return err
	}

	members, found, err := unstructured.NestedInt64(mdb.UnstructuredContent(), "spec", "members")
	if err != nil {
		return err
	}

	if !found {
		return fmt.Errorf("members field not found on MongoDB spec")
	}

	scaled := int(members) * (1 + int(proportion))

	patch := []interface{}{
		map[string]interface{}{
			"op":    "replace",
			"path":  "/spec/members",
			"value": scaled,
		},
	}

	payload, err := json.Marshal(patch)
	if err != nil {
		return err
	}

	_, err = mongoDBClient.Patch(ctx, name, types.JSONPatchType, payload, metav1.PatchOptions{})
	if err != nil {
		return err
	}

	return nil
}

Here we leverage unstructured.NestedInt64 to access only the field that we are interested in, keeping our coupling to the MongoDB CRD to a minimum while also being able to manipulate the resource data with type safety.

The unstructured package has lots of helpers like this, not only for reading but also for writing to any field on the resource.

Performing all the usual operations on Kubernetes (get, list, watch, create, patch, and delete) follow the same approach: provide the scheme.GroupVersionResource and handle the unstructured.Unstructured result.

Controller with a dynamic client

More advanced but frequent use of a Kubernetes client is to build a controller that reacts to changes on the actual cluster state to bring it to the desired state.

Usually, we leverage an Informer, a component provided by k8s.io/client-go, that runs a handler when changes are detected, created from a typed client. Luckily the dynamic package also provides an Informer component that we can use.

For example, if we want to capture when a MongoDB is deleted to clean the associated PersistentVolumeClaims:

package main

import (
	"fmt"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"k8s.io/apimachinery/pkg/util/wait"
	"k8s.io/client-go/dynamic"
	"k8s.io/client-go/dynamic/dynamicinformer"
	"k8s.io/client-go/tools/cache"
	"k8s.io/client-go/util/workqueue"
	"time"
)

const maxRetries = 3

var monboDBResource = schema.GroupVersionResource{Group: "mongodbcommunity.mongodb.com", Version: "v1", Resource: "mongodbcommunity"}

type MongoDBController struct {
	informer cache.SharedIndexInformer
	stopper  chan struct{}
	queue    workqueue.RateLimitingInterface
}

func NewMongoDBController(client dynamic.Interface) (*MongoDBController, error) {
	dynInformer := dynamicinformer.NewDynamicSharedInformerFactory(client, 0)
	informer := dynInformer.ForResource(monboDBResource).Informer()
	stopper := make(chan struct{})

	queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
	informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		DeleteFunc: func(obj interface{}) {
			key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
			if err == nil {
				queue.Add(key)
			}
		},
	})

	return &MongoDBController{
		informer: informer,
		queue: queue,
		stopper: stopper,
	}, nil
}

func (m *MongoDBController) Stop() {
	close(m.stopper)
}

func (m *MongoDBController) Run() {
	defer utilruntime.HandleCrash()

	defer m.queue.ShutDown()

	go m.informer.Run(m.stopper)

	// wait for the caches to synchronize before starting the worker
	if !cache.WaitForCacheSync(m.stopper, m.informer.HasSynced) {
		utilruntime.HandleError(fmt.Errorf("timed out waiting for caches to sync"))
		return
	}

	// runWorker will loop until some problem happens. The wait.Until will then restart the worker after one second
	wait.Until(m.runWorker, time.Second, m.stopper)
}

func (m *MongoDBController) runWorker() {
	for {
		key, quit := m.queue.Get()
		if quit {
			return
		}

		err := m.processItem(key.(string))

		if err == nil {
			m.queue.Forget(key)
		} else if m.queue.NumRequeues(key) < maxRetries {
			m.queue.AddRateLimited(key)
		} else {
			m.queue.Forget(key)
			utilruntime.HandleError(err)
		}

		m.queue.Done(key)
	}
}

func (m *MongoDBController) processItem(mongodb string) error {
	// Clean up PVCs
	return nil
}

Most of this code is standard, like the work queue, informer event handlers, and item processing, of a controller using the typed client.

Hence, leveraging the decoupling provided by the dynamic client really comes with a low overhead in terms of complexity.

Testing with the dynamic client

If we were to scale the use of the dynamic client it is paramount that it is as easy to test as the typed client.

As in the controller case, the dynamic package provides an equivalent fake client that allows for stubbing objects and asserting actions performed using it.

package main

import (
	"context"
	"k8s.io/apimachinery/pkg/runtime"
	dynamicfake "k8s.io/client-go/dynamic/fake"
)

func TestDynamicClient(t *testing.T) {
	// Setup an Object as mock on the client
	// Write it like its YAML manifest
	mdb := &unstructured.Unstructured{}
	mdb.SetUnstructuredContent(map[string]interface{}{
		"apiVersion": "mongodbcommunity.mongodb.com/v1",
		"kind": "MongoDBCommunity",
		"metadata": map[string]interface{} {
			"name":      "mongodb-test",
			"namespace": "default",
		},
		"spec": map[string]interface{}{
			"members": 3,
		},
	})

	dynamicClient := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), mdb)

	// Run any logic that depend on the dynamic client
  NotifyMongoDBs(context.Background(), dynamicClient)

	AssertActions(t,  dynamicClient.Actions(), []ExpectedAction{
		{
			Verb: "list",
			Namespace: "default",
			Resource: "mongodbcommunity",
		},
	})

}

Using the unestructured.Unestructured type we can create stub Kubernetes objects using the same syntax as in YAML, but with maps.

After performing the tested logic we can use dynamicClient.Actions() to see all operations that were performed by our code. However, manually asserting these actions on every test often lead to unreadable code and brittle assertions.

Hence, I often use a special assertion function AssertActions that verify if every expected action can be found in the performed actions. An important note is that this function does not perform an exact list match, i.e. if a delete operation was performed using the client the test would not break, the only condition for the AssertActions to fail is if the list operation provided on the expected list isn’t found. One could change the asserting function or make a sibling function that validates only if the expected actions were performed.

Although the current implementation is verbose, this function has the benefit of working both with the dynamic and the typed client.

type ExpectedAction struct {
	Verb string
	Name string
	Namespace string
	Resource string

	// Patch action
	PatchType types.PatchType
	PatchPayload []map[string]interface{}
}

func AssertActions(t *testing.T, got []kubetesting.Action, expected []ExpectedAction) {
	if len(expected) > len(got) {
		t.Fatalf("executed actions too short, expected %d, got %d", len(expected), len(got))
		return
	}

	for i, expectedAction := range expected {
		if !AssertExpectedAction(got, expectedAction) {
			t.Fatalf("action %d does not match any of the got actions", i)
		}
	}
}

func AssertExpectedAction(got []kubetesting.Action, expectedAction ExpectedAction) bool {
	for _, gotAction := range got {
		switch expectedAction.Verb {
		case "get":
			getAction, ok := gotAction.(kubetesting.GetAction)
			if !ok {
				continue
			}

			if getAction.GetName() != expectedAction.Name {
				continue
			}

			if !validateNamespaceAndResource(getAction, expectedAction) {
				continue
			}

			return true
		case "list":
			listAction, ok := gotAction.(kubetesting.ListAction)
			if !ok {
				continue
			}

			if !validateNamespaceAndResource(listAction, expectedAction) {
				continue
			}

			return true
		case "watch":
			watchAction, ok := gotAction.(kubetesting.WatchAction)
			if !ok {
				continue
			}

			if !validateNamespaceAndResource(watchAction, expectedAction) {
				continue
			}

			return true
		case "create":
			createAction, ok := gotAction.(kubetesting.CreateAction)
			if !ok {
				continue
			}

			if !validateNamespaceAndResource(createAction, expectedAction) {
				continue
			}

			return true
		case "update":
			updateAction, ok := gotAction.(kubetesting.UpdateAction)
			if !ok {
				continue
			}

			if !validateNamespaceAndResource(updateAction, expectedAction) {
				continue
			}

			return true
		case "delete":
			deleteAction, ok := gotAction.(kubetesting.DeleteAction)
			if !ok {
				continue
			}

			if deleteAction.GetName() != expectedAction.Name {
				continue
			}

			if !validateNamespaceAndResource(deleteAction, expectedAction) {
				continue
			}

			return true
		case "patch":
			patchAction, ok := gotAction.(kubetesting.PatchAction)
			if !ok {
				continue
			}

			if patchAction.GetName() != expectedAction.Name {
				continue
			}

			if !validateNamespaceAndResource(patchAction, expectedAction) {
				continue
			}

			if patchAction.GetPatchType() != expectedAction.PatchType {
				continue
			}

			patchBytes, err := json.Marshal(expectedAction.PatchPayload)
			if err != nil {
				continue
			}

			if !bytes.Equal(patchAction.GetPatch(), patchBytes) {
				continue
			}

			return true
		}
	}

	return false
}

func validateNamespaceAndResource(action kubetesting.Action, expectedAction ExpectedAction) bool {
	return action.GetNamespace() == expectedAction.Namespace && action.GetResource().Resource == expectedAction.Resource
}

This asserting function allows for more conditions to be added, like verifying list/watch restrictions and create/update bodies.

Conclusion

The Kubernetes ecosystem is rich and every now and then we stumble upon this kind of treasure. I strongly recommend reading through the documentation not only of the k8s.io/client-go project but also from the sigs.k8s.io/controller-runtime project and the Kubernetes Reference API documentation.

Caching is one of the most popular tools used to scale systems and anyone looking to maintain high throughput, resilient and cost-effective products should understand how to use it because it is financially impractical to apply only compute resources in order to meet the access demands.

Knowing the basics about it and what parameters you should be looking when choosing your solution is rarely addressed and hence is the purpose of this article.

Don’t rush tour decision

If one would search for caching it will find a plethora of tutorials teaching how to set up your solution. They make it look so easy to use a cache in your application that one may do it without thinking twice. Be careful, every choice comes with costs and tradeoffs, caching is no different.

Usually, we can find bad cache designs when we stumble on the most important metrics for it: hit ratio and miss ratio. As a brief overview, we can define these metrics as follow:

Hit ratio: when the cache has a key and can provide the value for the system to use, we call it a hit. The metric is simply the number of hits / number of lookups.

Miss ratio: when the cache hasn’t the key and the value must be computed for the system to use, we call it a miss. The metric is simply the number of misses / number of lookups.

The number of lookups is simply the total quantity of cache accesses, which is lookups = hits + misses.

You can know that the cache is not being efficient if it has a low hit ratio and, therefore, a high miss ratio. What low or high means will depend on your problem. Discovering your baseline metrics can only be achieved through our first guideline.

Guidelines

Monitor your cache

You should set up a way to capture the hit and miss ratio of your cache solution. How you will do it is highly dependable on the chosen implementation, but most of it should have an easy way of extracting these statistics and if not, consider looking for others.

The most important about having monitoring is exactly that you will be able to experiment with different algorithms and tradeoffs presented in the below guidelines and strive to improve your ratios. Hence, formulate a hypothesis and let your data drive your solution.

Tradeoffs: Performance and Resilience vs Consistency

Caching is one of the simplest and more powerful ideas in computation. Understanding and applying its basic cases is easy but can become extremely hard sooner than you imagine. Therefore the classical phrase, “there is only two difficult things in computer science: cache invalidation and naming things”. But, why cache invalidation is so hard? Because it usually is critical and has many moving parts.

The moving parts come from the fact that caches improve application performance because they bring the data closer, which also means we now have gone off the rails with the most important principle for Consistency: have one source of truth. This has another effect which is resilience, since now if our main source of information goes off, our application can survive a little longer with its cached values.

The critical segment arises usually when you need to update the data on your cache. In order to give your user meaningful information, you need to understand the access patterns to choose the right eviction algorithm and parameters which will balance performance and correctness, choosing the wrong one will probably damage your product. Besides that, there is the case when you need to force clean your cache and the distributed nature of it can cause a lot of pain in the process of invalidating each cache node.

Therefore, adding a cache to your solution is a trade-off between Performance+Resilience versus Consistency and so the first question you should ask yourself is “Can my system live with potentially old and invalid data?” If your answer is Yes, then you can continue here, otherwise, caching will do you more harm than good.

Keyspace

So you decided that you really want a cache. The first decision you will have to make is about your keyspace, i.e. what you will use as a cache key to index your costly computed values?

It is important because you need to analyze your key cardinality, which means how many distinct values your key can have. For example, a boolean key has a cardinality of 2 (true or false) whereas a customer id can have thousands of possible values. This is really important to understand because a low cardinality key would limit the amount of data your cache could store but your hit ratio would be really high. On the other side, if a key has an extremely high cardinality (tending to uniqueness, never repeating itself) your cache could grow exponentially and you may end up with a low hit ratio, demanding much computation and providing little performance improvements.

Hence you want to choose a high cardinality key, avoiding never-repeating ones, but not to small, avoiding limited ones. One way many uses to achieve this balance is to use complex keys (like a map or list), mixing a medium cardinality key like customer id with a low cardinality one like state names.

One scenario that you should be careful is with memoization. For those coming from OO lands, it is a technique to cache the values computed by a function. It uses the function’s arguments as keys and caches the result. But, since a function may change over time and you may use a solution that wraps the function in a place far from the local here it is implemented, there is a great risk that some feature or refactoring adds unique arguments (like a timestamp) or reduce the arguments to low cardinality ones (a small enum and a boolean). You should implement memoization near to the implementation and/or use cache solutions that allow you to choose the keys from the argument list, which is the best solution since one could not accidentally chance that.

Cache algorithms and strategies

Next, you need to understand your access pattern in order to choose your cache algorithm and possible strategies. Each one will have trade-offs, you can combine some of them and you should experiment because in this area data will be better to guide you.

• FIFO (First in First out): this is the simplest algorithm where the cache works like a queue and evict the first block to enter, independent of how many times it was used. Through time, you will have the most used keys remaining in the cache, since even if a block was evicted, since it is highly used, duplicated blocks of this key will be presented at the queue. This strategy is really simple to implement and has low overhead but also has an inefficient usage of memory in comparison to other algorithms.
• LRU (Least Recently Used): probably the most used cache algorithm, it tracks when some block was used and evict the one with fewer accesses. Hence it keeps the most used keys in the cache but with better memory usage. As a tradeoff, it has a more complex and costly implementation since it has to add and track age bits in the cache blocks.
• LFU (Least Frequently Used): imagine that you are using an LRU cache and you have 100 accesses in the last second. There were 80 hits in the key A, 19 hits in key B and 1 hit in key C. If your cache is full, in the next miss that needs to load a new key, your cache would not evict C, because it was the most recent one accessed. This can be really bad since we are probably removing a more usage key (A or B) in favor of C. That is the problem that the LFU algorithm addresses since it keep the most frequent usage keys, it would evict the key C from our example because it has a small access frequency. This is a really interesting model and probably is better suited to most use cases, but as a pattern, it also has more overhead than LRU because how it has to keep track of how many times a block was accessed in relation to how many accesses happened to the cache.
• TTL (Time to Live): one really common strategy is time to live, an algorithm that evicts blocks that are older than a certain pre-defined timespan. It is used for more volatile data and usually with two cases: for low cardinality keys that can’t grow the memory footprint to the point where one block would be evicted or in combination with other cache strategies (like the ones mentioned above) to provide more refresh opportunities.
• Stale data: we say that a block is stale when it passes its expiration time and should be evicted or refreshed. The stale data strategy is an augmentation of TTL that instead of eliminating the block from the cache once the time to live expires, it runs a refresh function (in case of memoization it reruns the function) that will compute a new value for the key. During this time, the stale (old) data is served for those that access the cache. It can also be the case where if the refresh function fails, it simply maintains the block in the cache, that may be evicted by some algorithm, but is not eliminated by its expiration. This can be a really powerful strategy for increased resilience if your system support living with a possibly long-living stale data.

Besides these there is also more modern algorithms like Windowed TinyLFU (used by Caffeine), LIRS and ARC. Note that various discussions about the cache algorithm will reference the theoretical Bélady’s algorithm, so it is good to have a look at it.

Local vs Distributed

One last question you might need to answer is if you are going to use a local or distributed cache. This will be the most important question in financial terms, so look close to your needs.

Local: it means that you will maintain the data in the memory of an application instance. This is the most simple setup, but it can impose a burden in the allocated RAM, maybe forcing you to upgrade to a bigger instance, which can be pretty expensive. It also can have suboptimal performance-wise, since the worst-case scenario for a cache with a size limit of 1000 keys is to have just it stored across all instances, that is, the same data is replicated in all local caches. This intersection diminishes the total performance gain across your system.

Distributed: in this case one would use a solution like Redis or Memcache, running separately from the application, and being shared by all instances. This solution optimizes the resource usage (the RAM is allocated exclusively for the cache), allow for bigger cache entries and improves data density (there is, no more replicated entries), but even then it can be less performant than the local solution since it demands a network roundtrip to access the database. Besides that, the cost to maintain new instances and the skill needed to operate these should not be overlooked.

All these points should be taken into consideration when choosing the design for your cache in your system.

Conclusion

Therefore, before looking into tutorials and getting started articles about how to set up your cache, think about and discuss your problem and how you could configure the cache to deliver the most value to your system. You can use these and other guidelines as starting points for the debates and focus on guiding your decisions based on the data measured from your implementation.

I hope you found it useful and you can find more of my thoughts at caioferreira.dev

Introduction

Being passionate by functional programming I am often asked about how and where one can learn more about this style. Hence, I decided to compile some resources on the subject to suggest a path to this beautiful world.

I split the content into three levels: Beginner, Intermediate and Advanced. Since this is thought of as a starter to master path, I gave preference to contents in Javascript, which is a well-known language that allows for a great extent of functional patterns.

Some purists probably will hate me for this choice.

In this path I tried to add resources on fundamental functional concepts and tools, then an introduction to some functional patterns like Combinators, Category Theory, Railroad Programming and finally a material to deepening into complex subjects.

After all, I also added some tips on dedicated functional languages that I found easy to play with or that I simply love to work with.

The Yellow Bricks Road

Beginner

Functional Programming Principles - Article about basic principles of the functional paradigm, like pure functions and immutability

Functional programming in Javascript - Highly recommended. Video series from the renowned Youtube channel FunFunFunction about functional tools and concepts in Javascript. The explanation is deep and clear about the subjects.

Eloquent Javascript: Functional Programming - The Eloquent Javascript’s chapter about functional programming. It is a hands on to building fundamental tools by hand, understanding its applications and usages.

Becoming Functional - An excellent book about the functional paradigm as a whole, presenting topics from functional principles like pure functions to advanced techniques like pattern matching. Unfortunately, the example is in Java.

Intermediate

Functional Design Patterns - An amazing talk about how to combine different pieces of pure functions into more complex applications.

Javascript Combinators - a talk from one of the greatest names in Functional Javascript, Reginald “Raganwald” Braithwaite, where he explores the concept of composition and introduces the Combinators pattern, a powerful functional tool inspired by Lambda Calculus.

Professor Frisby Mostly Adequate Guide to Functional Programming - An excellent resource on more complex functional patterns using Category Theory. This approach is less common on Front End and probably we can get most of the benefits of the functional paradigm without the complexity of an entire branch of mathematics.

Advanced

Category Theory for Programmers - The most recognized and advanced resource on Category Theory for development ends.

To Grok a Mockingbird & Why Y? Deriving the Y Combinator in JavaScript - two incredible articles from Reginald Braithwaite about the application of the Combinators pattern.

Going full functional

If you enjoyed your travel and want to keep going into this path, I recommend that you play with some fully functional programming language. I suggest Elm, ReasonML, and Clojure.

The first two (Elm and ReasonML) are functional languages that compile to JavaScript and are used to build front end applications. A special note to ReasonML which has great integration with React, being a language developed by Facebook.

The last one is Clojure, a JVM language, dynamically typed and from the LISP family. It is used primarily on the back-end and is well suitable for describing complex domain since it’s syntax is very flexible and can be customized in a way to better describe your problem space.

You can find more about these languages here:

• Elm: Documentation and Code Example
• ReasonML: Documentation and Code Example
• Clojure: Best Book for new Clojurists and Code Example

Summary

The Lambda World is a vast and beautiful one, you will find many ways of thinking and solving problems. That is the main reason why I love FP, it not only brings new tools to your workbench but also constantly shifts your mental models. In this segment I could not leave the most impactful talk on my perspective on the work and profession on software engineering: Simple Made Easy by Rich Hickey, the creator of Clojure.

I hope these tips help you and fair travel!

In this post I will show you what is the debt that we collect with during the software lifecycle, what are its causes and how to pay it back.

Introduction

Often we are faced with a dilemma in software development: implement the best solution for the feature or delivery it quickly but assuming some workarounds and code smells? Whichever side you choose, will be a cost.

This cost is even greater if you work on legacy projects or high changing environments. I am a software engineer @ B2W, the biggest e-commerce on Latin America, and because of its long history, we deal with +6 years legacy applications, in which dozens of developers worked through this time. Sum up the e-commerce environment with its high competitiveness that drives constant changes and new features then you will have a millionaire decision at your hands.

Hence, understating how to manage this cost can be a key asset to achieve a product that has both high quality and a short time to market.

The decision cost

To understand how to manage the decision cost, we must first learn what cost is. Every decision we make in a software project, not just about best solution vs quick & dirty, incurs in a time cost. Sometimes we cannot take this time and need to target production rather earlier than later. In this situation, we exchange bad code for time. Apply this behavior through time and we get a large debt.

Therefore, Technical Debt is a metaphor in software development, based on the financial debt, about the accumulation of low-quality code in a project over time.

Extending the metaphor, we can see financial debt as a resource uptake from some institution, therefore, in the software scope, the resource we borrow is time instead of money and the institution that provides us with this is our code base rather than a bank. Thus, in technical debt calculation, the main amount is the time cost to refactor the codebase to a clean design and the interest represents the extra time that will be taken in the future if the team has to work on the messy code.

Even more debt

Although the best solution vs quick & dirty decision is an excellent example to introduce technical debt, it isn’t its only source. As Fowler (2009) points out, this case is just a Prudent and Deliberate debt, where the team is aware of the cost it is incurring and know how to pay it back. But we can have four types of debt in total.

Each debt type has a source associated with it:

• Prudent & Deliberate - the team know how to build good software and has the opportunity to deliver more product value if they speed up. They analyze the tradeoff and judge that cost/earn ratio is enough to justify the bad code, otherwise delaying the solution. Then, they plan how to refactor the debt and implement it as early as possible.
• Reckless & Deliberate - the team has the skill to build a well-designed solution but do not has the tools or support to do so. This happens when a team is purposely taking debt but without a plan to repay it or to do so in a code that probably will never change. This scenario is more common on unmotivated teams, legacy projects and due to management pressure for fast delivery instead of a constant and quality delivery.
• Reckless & Inadvertent - the team doesn’t know good design practices and incur on costs due to lack of training, experience, and leadership.
• Prudent & Inadvertent - it is a common case that after we delivery a feature or project we realize that the best design approach would be other than what we did. This situation is unpredictable and can happen to any team due to lack of domain knowledge, obscure requirements, and technology limitations.

Of course, this exploration does not exhaust the possible causes for technical debt but includes the most frequent ones.

Living on high debt

Since the debt causes can be so many, projects can experience an ever-increasing technical debt, to the point where adding new functionality, fix current bugs and operate the application becomes impossible.

This type of scenario can be catastrophic, because while the team lives with this application they will lose agility, will observe an increasing bug count, have loss of motivation, increased stress, long production problems (long living bugs), customer complains and possible single points of failure in case only a few people know how to develop and operate the project.

When an application reaches this point we say that it went into technical bankruptcy.

However, using the right strategies we can make the highlighted curve steeper, leading the actual curve nearest to the idealized curve.

Paying back

In order to avoid that applications achieve high debts or to handle projects already with high debt, we need to build a repayment plan. This can have any sort of effort in order to refactor and pay the time due, but the industry set some strategies that have proven effective.

1. Technical Backlog: each task should have a brief description, the reason that the change is important for the project and which part of the code base must change. Like any other task, we should estimate the effort needed to build a good and clean solution. When estimating the cost, the team should add an interest cost that is directly proportional to the probability of this code to change in the future and hence prioritizing the most costly task. A precise estimation is really difficult but a rudimentary approximation is enough to guide the decisions.
   • With this approach the technical debt becomes visible to all stakeholders and the decision of doing such a task can be made upon effort and future impact.
   • The task cost is easily traceable.
   • Don’t mix technical and feature tasks.
2. Refactoring costs included on the feature estimation: explain the costs is necessary since we follow the principle that no feature should be implemented above bad code and therefore this code should be refactored first (Boy Scout Rule). This is a fundamental principle once bad code already incur in interests and adding a new feature using it will increase this interest exponentially, to the point where the cost to work on this code will be so high that will be impossible to make further changes.

Besides that, there are two other critic strategies that can be used depending on how high debt the project is in.

1. Buffer Tasks: the task has a portion of the team’s sprint. It can be used to unplanned refactorings, unforeseen problems, discoveries, etc. To avoid the task to be wasted on non-relevant work members of the team should always propose how they pretend to use the time and discuss with the rest of the team.
2. Clean-up Releases: periodically or in critic scenarios, a team can make releases only with technical refactorings. This strategy is only useful if there is already a refactoring list to be made. Furthermore, the business team should give support as this will probably delay new features. The teams should consider using this when a greater effort is necessary, like in big architectural changes, infrastructural changes, build & deploy, etc.

Conclusion

Therefore, Technical Debt can be key tool from the product perspective. It allows business people to improve time to market and customer satisfaction with the cost of bad code.

However high debt can bring to huge problems to the project, hence we must strive to be Prudent and Deliberated about our debt, always taking it with a repayment plan together, using all kinds of strategies to keep the debt under control.

To achieve this all stakeholders must be aware of the debt’s nature and existence and bring the development team together in the decision.

References

• https://www.infoq.com/presentations/debt-aware-culture
• https://martinfowler.com/bliki/TechnicalDebtQuadrant.html
• https://www.infoq.com/minibooks/emag-technical-debt?utm_source=minibooks_about_TechnicalDebt&utm_medium=link&utm_campaign=TechnicalDebt
• https://books.google.com.br/books?id=pKVFDwAAQBAJ&lpg=PA18&ots=Gc9vC8vf80&dq=Software Quality Assurance Pressman 2014&hl=pt-BR&pg=PA18#v=onepage&q=Software Quality Assurance Pressman 2014&f=false

Photo by Annie Spratt on Unsplash

Today we will try to solve the ambiguity in the concept of state, highlighting the differences in the two main notions about it. The post describes the nature of each one, some use cases and how they fit in the object and functional paradigms.

Introduction

On the last couple of months, I dove into the topic of State Machines and how we can design UI’s with this concept in order to provide better semantic and predictability to our application. When reading and talking about it often I have to stop and clarify which of the two ideas about state I am referring to:

• the idea of a collection of data at a point in time
• the idea of the representation of an entity modeled as a state machine.

For the sake of comprehension, we will use state to refer to the first and State to the last.

Requirements

We will use Typescript for our yummy examples so some familiarity with it would be good.

The state as a travel bag

The first notion we became comfortable with when learning about state in software development is the entity “travel bag”. Basically, we see a state as a collection of data in a specific point in time. Throughout the application lifecycle, this data is manipulated and altered in order to reflect the business process. For example:

class Pizza {
  private dough: Dough; // it's an enum that could be traditional or thin
  private ingredients: Array<Ingredient>;

  // entity controls
  private isBeingnPrepared: boolean;
  private isBaking: boolean;
  private baked: boolean;

  constructor() {
    this.isBeingnPrepared = true;
    this.isBaking = false;
    this.baked = false;
  }

  // getters and setters

  public async bakePizza(): void {
    const oven = new OvenService();

    try {
      this.isBeingnPrepared = false;
      this.isBaking = true;

      await oven.bake(this);

      this.baked = true;
    } catch (error) {
      throw error;
    }
  }
}

From this point on, the pizza state, and hence the application state, has changed, because its data was updated. However, two booleans can be arranged in four different ways, and some of them are invalid states. In the object-oriented paradigm we would avoid this by encapsulating such data in an object and modeling its operations only through methods that guarantee atomic and consistent changes.

Until this use case, our model seems to be fine. But, the time comes to implement the next step in the pizzeria flow, the delivery.

class Pizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // entity controls
  private isBeingnPrepared: boolean;
  private isBaking: boolean;
  private baked: boolean;
  private isBeingDelivered: boolean;
  private hasBeenDelivered: boolean;

  constructor() {
    this.isBeingnPrepared = true;
    this.isBaking = false;
    this.baked = false;
    this.isBeingDelivered = false;
    this.hasBeenDelivered = false;
  }

  // getters and setters

  // bake behavior

  public async deliveryPizza() {
    if (!this.baked) {
      throw new PizzaNotBakedException();
    }

    const deliveryService = new DevelieryService();

    try {
      this.isBeingnPrepared = false;
      this.isBeingDelivered = true;

      await deliveryService.send(this);
    } catch (error) {
      throw error;
    }
  }

  public notifyDelivery(wasSuccessful) {
    if (wasSuccessful) {
      this.hasBeenDelivered = true;
    }
  }
}

What raises a flag in this code is the use of a guard condition at the start of the delivery function that checks if the pizza is baked. If not, it throws an exception. This seems really simple, and if this were the only condition, it would be fine. But, a pizza could already be left for delivery, as such, we don’t want to try to send it again. So, we add another guard condition to our function:

class Pizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // entity controls
  private isBeingPrepared: boolean;
  private isBaking: boolean;
  private baked: boolean;
  private isBeingDelivered: boolean;
  private hasBeenDelivered: boolean;

  // constructor

  // getters and setters

  // bake behavior

  public async deliveryPizza() {
    if (!this.baked) {
      throw new PizzaNotBakedException();
    }

    if (this.isBeingDelivered) {
      throw new PizzaAlreadyLeftException();
    }

    const deliveryService = new DevelieryService();

    try {
      this.isBeingPrepared = false;
      this.isBeingDelivered = true;

      await deliveryService.send(this, this.notifyDelivery);
    } catch (error) {
      throw error;
    }
  }

  // notify delivery behavior
}

If we elaborate all the scenarios which a pizza can be, this kind of implementation with lots of branches and conditions expressed by if/else statements grows exponentially. It increases our code cyclomatic complexity and diminishes maintainability as such code is more fragile, harder to read and understand.

It gets worse when this kind of conditional start to spread across the code, as in the bake function, which needs to be updated in order to not try to bake it again.

class Pizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // entity controls
  private isBeingPrepared: boolean;
  private isBaking: boolean;
  private baked: boolean;
  private isBeingDelivered: boolean;
  private hasBeenDelivered: boolean;

  // constructor

  // getters and setters

  public async bakePizza(): void {
    if (this.baked) {
      throw new PizzaAlreadyBakedException();
    }

    const oven = new OvenService();

    try {
      this.isBeingPrepared = false;
      this.isBaking = true;

      await oven.bake(this);

      this.baked = true;
    } catch (error) {
      throw error;
    }
  }

  // delivery behavior

  // notify delivery behavior
}

Although this kind of design serves several proposes, in special on more simple or data-centric scenarios, in fast evolution and process-centric domains it evolves on a mess of code execution paths and unsynchronized conditionals through different functions.

The state as an entity travel bag has a use and it is to carry the associated information to the model. Try to control the behavior of this entity through the same concept ends up overloading it with responsibility and creating a silent trap for our design.

The problem faced here is that the application architecture allows for invalid behavior through invalid states, and when it does eventually some use case will expose the bugs created by this freedom. Besides that, this approach takes the system invariants, in this case, the Pizza cooking flow, and scatter then inside many implementation points instead of enforcing them in the design.

Side note: if you are versed in Algebraic Data Types you can see this as a Product Type with cardinality which tends to infinity.

Representational State

Once we have the problem of control the entity information and behavior being done by the same construct, the state, our response could not be more simple: let’s break these responsibilities.

Therefore, we need a new pattern to handle our entity’s behavior.

But, the alternative pattern we propose when designing your application is not at all new. It is the State Pattern, describes in many ancient books about OO. And this books will tell you the same, that the State Pattern seeks to delegate an entity behavior to a specific implementation which is the current State and at the end of the method calculate the entity’s next State, which will now represent the entity, replacing its behaviors implementation on the fly. After all, this pattern is a translation of a state machine to the idiom of the nouns. An alternative implementation for our Pizza example can be as below:

interface IPizza {
  bakePizza();
  deliveryPizza();
  notifyDelivery(wasSuccessful: boolean);
}

type Pizza = PreparingPizza | BakedPizza | DeliveringPizza | DeliveredPizza;

class PreparingPizza implements IPizza {
  private dough: Dough; // it is an enum that could be traditional or thin
  private ingredients: Array<Ingredient>;

  constructor(dough: Dough, ingredients: Array<Ingredient>) {
    this.dough = dough;
    this.ingredients = ingredients;
  }

  // setters

  getDough() {
    return this.dough;
  }

  getIngredients() {
    return this.ingredients;
  }

  public async bakePizza(): Promise<BakedPizza> {
    const oven = new OvenService();

    try {
      await oven.bake(this);

      return new BakedPizza(this);
    } catch (error) {
      throw error;
    }
  }

  public async deliveryPizza() {
    throw new PizzaNotReadyForDelivery();
  }

  public notifyDelivery(wasSuccessful) {
    throw new PizzaNotReadyForDelivery();
  }
}

class BakedPizza implements IPizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // constructor
  constructor(pizza: PreparingPizza) {
    this.dough = pizza.getDough();
    this.ingredients = pizza.getIngredients();
  }

  // getters and setters

  public async bakePizza(): Promise<BakedPizza> {
    throw new PizzaAlreadyBakedException();
  }

  public async deliveryPizza(): Promise<DeliveringPizza> {
    const deliveryService = new DevelieryService();

    try {
      await deliveryService.send(this);

      return new DeliveringPizza(this);
    } catch (error) {
      throw error;
    }
  }

  public notifyDelivery(wasSuccessful) {
    throw new PizzaNotLeftForDeliveryYey();
  }
}

class DeliveringPizza implements IPizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // constructor

  // getters and setters

  public async bakePizza(): Promise<BakedPizza> {
    throw new PizzaAlreadyBakedException();
  }

  public async deliveryPizza(): Promise<DeliveringPizza> {
    throw new PizzaAlreadyLeftForDeliveryException();
  }

  public notifyDelivery(wasSuccessful) {
    if (wasSuccessful) {
      return new DeliveredPizza(this);
    }
  }
}

class DeliveredPizza implements IPizza {
  private dough: Dough;
  private ingredients: Array<Ingredient>;

  // constructor

  // getters and setters

  public async bakePizza(): Promise<BakedPizza> {
    throw new PizzaAlreadyBakedException();
  }

  public async deliveryPizza(): Promise<DeliveringPizza> {
    throw new PizzaAlreadyLeftForDeliveryException();
  }

  public notifyDelivery(wasSuccessful) {
    throw new PizzaAlreadyDeliveredException();
  }
}

With this implementation, we enforce the domain invariants with our type system through the interface and the Pizza union type. With it we gain less cyclomatic complexity since we don’t have so many branches in our code and, by design, we don’t allow for invalid States to happen. Besides that, each State carries an internal data, its travel bag. As such, these patterns are not excluding, but rather composable.

In the trend on the front-end what we are usually seeing is more a functional paradigm approach to the state machines. The entity, represented as a state machine, is now just a different data structure for each State that can be interpreted by the pure functions that implement the domain behaviors. These functions than can internally delegate its call to others functions specialized in each State. This separation of the state machine implementation of the behavior is natural as it follows the idiom for functional architectures.

What remains in both cases are the nature of the State as an entity’s representation. It works on its behalf and delimits the possible behaviors it can expose.

For example, a Pizza could never be in Baked and Delivered States at the same time. Now, it isn’t an implementation that guarantees that it is the design itself. Such abstractions, that models the domain, the heart of our product, couldn’t depend on implementation details to be valid, they must depend on the abstractions itself.

Side note: if you are versed in Algebraic Data Types you can see this as a Union Type with finite cardinality in the order of less then a dozen.

Evolving the abstraction

One could implement a State oriented design by using a simple enum, a proper state machine implementation or a more advanced concept, a statechart.

It is true that many domains can be modeled using the two first approaches to code a State, but sometimes we are faced with a high complexity scenario where this abstraction implementation would not scale with the development of the application.

For that reason that in 1987 David Harel proposed a new technique that expanded the grounds of the state machine definition, introducing tools like state hierarchy, parallelism, clustering, history, etc. He called it statecharts and it is a formalism that helps us scale the development of a State design, be implementing it thoroughly or just taking some tools.

I highly recommend reading more about statecharts as it can shift your mindset about how to approach problems.

Summary

Now we can differentiate state from State and avoid accidental complexity by using the right construct to model our domain. Its worth nothing if I don’t say that there is no silver bullet and these are tools to deliver a job. We have been experimenting with this design style on my team and it has been helpful since our scenario is really complex and fast pacing.

If you have any questions or want to discuss these and other topics more in deep please comment or you can reach me at Twitter, @caiorcferreira.

Thanks for reading!

References

• State Design Pattern
• State Driven Development for User Interfaces
• A Statechart implementation on JS
• Statecharts: A Visual Formalism For Complex Systems - the original article
• Pure UI Control
• Constructing User Interfaces with Statecharts