Starting a new project efficiently — technical side

Before we start

Please notice that the text below is related to How to start a new software project – a business perspective, which describes — from a non-technical perspective — how to prepare, start, and develop a new project from the very beginning with the maximum chances to succeed. Nevertheless, it’s not obligatory to read it up front.

Introduction

Quite often a programmer is happy in 2 cases: when starting a green field project (where one still believes that they could do it better than anyone else in the past — a common fallacy — or when it’s simply an exhilarating undertaking and a learning experience) and when joining a well-maintained project where changes can be made efficiently and automated, and automated tests don’t light up red based on a d6 dice roll (d20 on a good day).

A careful reader might have noticed that I explicitly didn’t mention the situation where a programmer’s break, usually reserved for tea or ping-pong, is spent fixing automated tests checking database write operations after a UI button’s color was changed from blue to maroon. Such cases often happen in poorly maintained software and are classified as scenarios where things are not at their best.

In the article below, I described a set of useful techniques, patterns and rules, which will help to maintain order, a sense of security for the changes being introduced, separate the business logic from the framework, thereby, indirectly miss out on the ping-pong session.

In the penultimate paragraph, I included a link to a free repository in which I presented a sample way to apply the practices described below.

Isolated business logic

Software is most often created to solve business problems stemming from the real world. The investor paying for the software puts interest in ensuring the product makes money or meets their expectations, without too many bugs or unexpected situations on production.

Professional developers (so called craftsman developers), and sometimes experienced investors, know that these requirements are very difficult to meet without coherent and transparent code covered by non-fluctuating automated tests.

Given these needs, it is valuable to have the business logic (in other words, a problem to solve or the core logic) separated from the infrastructure code or data storage.

In simpler projects, it’s usually enough to have a layered structure applied to the whole application, in which a specific layer does exactly one thing, but it does it properly. In this case, the most important layer is the business layer (sometimes called the use cases layer), since it contains the business logic. In simpler terms, it’s done to avoid putting the important logic in the views (the infrastructure layer) to extract it into separate methods and then in turn invoke them with specific parameters. With this approach, the actual business logic is separated, contains only the required dependencies and can be invoked in a few places without a need to repeat the code.

In more complex projects, the previous solution is not enough. Domain-Driven Design, an approach focused on solving problems within a given domain, comes to the rescue. Thanks to it, the highest effort can be put into solving the most important task from a business perspective. Moreover, it allows you to postpone aspects involving clearly technical problems, such as implementing API endpoints in a particular framework or choosing a specific database.

A more detailed description of this technique goes beyond the scope of this article, but I encourage you to explore the following reading:

• Learn Domain Driven Design
• Domain Driven Design Tackling Complexity in the Heart of Software

Architecture patterns

Choosing the right structure or architectural solution has a significant impact on software development, progress speed, amount of bugs, and sometimes, in extreme cases, unfortunately, on the failure of the project.

An architecture pattern means the fundamental structural organization of a software system and/or its modules; in other words, a high-level strategy that handles the large-scale components, global properties and mechanisms of the system. 

There is no doubt that it is crucial to carefully and consciously choose the right pattern, best suited for each individual problem.

The book Patterns of Enterprise Application Architecture provides a description of architectural patterns which I recommend considering before choosing the right architecture. It will surely clearly outline the advantages and disadvantages of each solution.

Repository pattern

As applications grow in complexity, managing data access across different parts of the application becomes challenging. The Repository pattern addresses this issue by abstracting the data layer, providing a clear separation between the business logic and the data access logic. This allows developers to work with a consistent interface for accessing data, regardless of the underlying data source (e.g., databases, web services, or in-memory collections).

By encapsulating data access in repositories, applications can more easily manage data operations, ensuring that data access logic is centralized and consistent. This separation also greatly enhances the ability to unit test business logic by mocking up the data repositories, thus not depending on the actual data source. Additionally, if the application needs to change its data source, this can be done with minimal impact on the business logic layer, since the repository interface remains consistent.

CQRS pattern

Common applications operate on more and more data. It’s wise to ensure separate concepts for reading and writing operations e.g. when it is difficult to query from repositories all the data that users need to view and read operations differs from write operations.

CQRS pattern provides this capability. With this approach, independent execution of read and write is added to the project, so in case of potential performance issues these actions can be adjusted separately.

Unit of Work pattern

In complex applications, managing multiple data transactions can become cumbersome and error-prone, especially when dealing with multiple repositories or data sources. The Unit of Work pattern helps solve this problem by keeping track of all data changes during a business transaction and coordinating the write out of these changes as a single operation. This means that if a business process involves multiple steps or manipulations of data, the Unit of Work ensures that these changes are either all committed or all rolled back, maintaining data integrity.

The Unit of Work pattern works hand in hand with the Repository pattern to provide a comprehensive strategy for data access and transaction management. It acts as a buffer that contains all the actions to be performed on the database and executes them as a single transaction. This not only simplifies transaction management but also ensures data consistency and integrity throughout the entire operation. By leveraging the Unit of Work pattern, developers gain better control over transactions, reducing the chances of partial updates or inconsistent data states, which are critical to maintaining application reliability and robustness.

Circuit breaker pattern

It’s important to provide the best possible end user experience; what matters most is a fast reaction (users are most satisfied when any feedback is returned within 200 ms) and the absence of any unexpected crashes.

During developing the MVP, there is a high chance that something has been missed, not forced or wrongly assumed, e.g. outside service has stopped working. This does not absolve the obligation to secure the software against such situations. The code must be properly prepared for them.

The Circuit breaker pattern is handy in this solution. By using it, we ensure that potential failures, even in independent parts, will not cause catastrophic effects in our system, and the user will be promptly informed about the problem.

Microservices

I advise against introducing microservices at the start of a project without a clear reason. Such an approach brings with it communication solutions that are not necessarily easy, may turn out to be slower than regular function invokes or may be poorly partitioned.

I know it’s a trendy topic lately, but complexity and communication issues rarely pay off outside of mature projects. At the initial stage, it’s not always clear what to portion out as an individual service, so it’s better to wait with this.

Design patterns

It may well be that the programming task or problem you are currently facing has already been solved. A design pattern is well tested and proven to work solutions to recurring problems in software construction. A collection of such design patterns can be found in a legendary book Design Patterns by Gang of 4 or RefactoringGuru website that contains descriptions of exactly these patterns.

SOLID principles

It’s important not to lose control over attributes, the parameters passed to them or their dependencies. Otherwise, the code situation will get worse and worse, leading to unnecessary complexity and unclear code relations.

Quite a popular option to avoid this are SOLID principles.

S	O	L	I	D
Single responsibility principle	Open-closed principle	Liskov substitution principle	Interface segregation principle	Dependency inversion principle
Prevents confusion in the codebase by ensuring that a class or module has one, and only one, reason to change	Enables the extension of system behavior without modifying existing code, thereby reducing the risk of bugs in existing functionality	Ensures that subclasses can replace their parent classes without affecting the correctness of the program, promoting reliability	Avoids forcing clients to depend on interfaces they do not use, which simplifies system interactions and dependencies	Decouples high-level modules from low-level modules by introducing abstractions, improving the system’s maintainability and flexibility

I will not describe these rules thoroughly, since it has already been done quite well in many other articles; nevertheless, I recommend two sources. The first is the book Clean Code, in which the principles were presented. The second is a website with a shortened description of the rules.

Automated tests

Quick and reliable automated testing of new functionalities is not an option in current professional standards, it’s a must-have.

Although it costs developer time, it’s a huge productivity boost for developing new features later. The main reason for this is the confidence that a bug has not been introduced to the existing functionality, which brings satisfaction and stability to the end user’s experience. 

Besides, a developer working on modifications or a new feature can quickly and independently check the impact of the modifications on the system. This results in the developer’s job satisfaction and faster feedback for them.

Just be sure to test code behavior, not internals, so that your tests will not need to change each time you’re refactoring a couple of things inside your module.

Code style

It’s important to ensure the aesthetic formatting of the code so that everyone on the team can read it comfortably (after all, code is read more often than written), and also that the code itself appears coherent and aesthetically pleasing.

Regardless of the programming language used, there are automatic formatters available that are easy to configure and even simpler to use. As an example, I can recommend an article prepared by my colleagues for Python.

Documentation

Documenting the architecture and code of a software project is crucial for ensuring that all team members can understand and contribute to the project effectively as well as help to hand over the project.

One highly recommended approach for achieving clear and comprehensive documentation is the use of the C4 model. The C4 model focuses on the software architecture, providing a framework for describing the system at different levels of abstraction. This includes the Context level (system relationships with users and other systems), Containers (the high-level technology choices), Components (how containers are divided into components), and Code (details of the components’ implementation). By documenting a project through these four levels, teams can achieve a better understanding of the system’s architecture, making it easier to maintain and extend. Additionally, the C4 model aids in identifying potential issues and improvements at each level of abstraction. Adopting the C4 model for documentation encourages a structured approach, ensuring that architectural decisions are clear, rationalized, and accessible to everyone on the team, facilitating better communication and collaboration.

In addition to the architectural documentation, it’s crucial to include detailed information about the backend entry points. Utilizing tools such as Swagger (now known as OpenAPI) for this purpose can significantly enhance the documentation process. Swagger allows for the easy creation, visualization, and maintenance of API documentation, providing interactive and user-friendly interfaces. By documenting backend endpoints with Swagger, developers ensure that the API is well-understood and easily accessible, promoting better integration and usage across teams and potentially external consumers. Integrating Swagger documentation into your project not only aids in development and testing but also serves as a valuable reference for API consumers, enhancing the overall quality and usability of your software system.

README.md

Easy-to-follow procedures for setting up a development environment, quickly finding development credentials or knowing your issues all come in handy when working within a team.

A proper place to insert such information is README.md. 

README.md is a text file that provides an overview of a software project, including instructions, information on how to install, use, and contribute to the project, typically found at the root of a code repository.

For this purpose, I always define clear sections describing things such as:

• what the code in the repository does
• required tools to run the project
• environment variables used in the project
• how to set up the development environment
• a link to or reference to the API documentation
• a link to the architecture documentation
• how to run automated tests
• how to generate new database migration

I also encourage you to properly format the command snippets presented in README.md so as to simplify their execution or copying (how to do this is described here).

Architecture Decision Log (ADL)

ADL most commonly consists of a short MD files called ADR, which contain a particular decision made across the project, e.g. why a specific database was chosen and not another one.

It’s quite a short document, but helps memorize particular decisions and reasons behind them.

It becomes very valuable over time.

Containerization

Developing software often requires so much more than writing code: multiple languages, frameworks, and discontinuous interfaces between tools. It’s quite convenient to configure one and share it with others.

This is exactly what Docker does. It simplifies and accelerates workflow.

Moreover, every solid cloud platform provides support for Docker, which means it can be also used to configure and run production environments.

Code pipelines

In the context of software development and deployment, incorporating a well-defined code pipeline that runs whenever changes are introduced to the codebase or other criteria are met is crucial for automating and streamlining the processes of building, testing, and deploying applications. Continuous Integration/Continuous Deployment (CI/CD) practices, often realized through pipelines, provide a structured and automated pathway through which code changes are systematically prepared for production environments.

This allows it to increase efficiently and allows Non-DevOps authorized units to properly release a new staging or production version.

More about this topic can be found in this Makimo’s article.

Sample code template

I’ve created a Python template repository following most of the practices and rules described.

The repository presents a simple example of DDD modeling in which effort has been put into solving an example business task. The business logic is independent of the framework, so it can be freely modified and further developed with any redundant dependencies.

Commands and queries are separated (CQRS) to demonstrate a potential solution for slow reads performance or complex queried data.

Database sessions are controlled via the manager, so potential connection issues are handled automatically.

The whole project is dockerized, so it can be launched on any computer with docker installed with a single command.

It contains automated tests that check functionalities at the unit, integration and end to end level.

What is also important, the template is under the MIT license, so feel free to use it without any financial obligation.

Link to the template: https://github.com/mglowinski93/LargeApplicationTemplate.

Conclusion

Starting a new project efficiently from a technical perspective requires considerability and well-thought-out decisions to avoid redundant frustration, workarounds, and hacks.

Unfortunately, there is no universal guide with a simple path for every software system. None of the patterns and techniques described in this article are silver bullets and should be followed blindly. It’s worth considering applying the principles and patterns described from the very beginning to the extent in which they provide tangible benefits. With the right balance, it will pay dividends both to the development teams and stakeholders.

In some situations, the quality of the code and its architecture don’t matter, for example, when the program will be used only once. However, most systems require changes, and continuous development of new functionalities, clear code and well-chosen solutions definitely facilitate this.