TomaszLos.io

At a certain point (3, maybe four years ago) I realized, that I already have some achievements (meaning the products of my work) in different fields, such as technology or music, and I always saw only more things coming up in the future. It led me to the idea that I should somehow catalog it and keep it growing as this might come useful when approaching new challenges, and who knows how this could end up. I wanted a kind of database for keeping track of my work and progress. I did not seriously consider making it available to the public. When I spent more time working with corporations with teams, I started taking it under consideration. When the time has come, I was required to share my knowledge beyond my usual technical work. I figured – why would I repeat myself? Then, my idea for the knowledge base evolved. I thought such functionality could be easily integrated with my personal website (that I intended to do for some time, as well), where people I am yet to develop could get not only technical knowledge but a broader idea about who do they work with, possibly leading to better understanding in our professional relations. Plus, if they enjoy some of my non-work-related content, I am happy to bring them some joy or inspiration. I have worked with multiple partners since this idea came to my mind, but I never had time to prove the concept finally. Till now…

The time has come, and one of my partners that I have been spending most of my business time on (plus a massive amount of overtime) for quite some time, decided to end our partnership. I understood – this is the moment, and I decided to disappear from the market for a relatively short period, providing my services to a minimal number of partners over this period. As I already had the concept, there were only a few final things to consider before starting the project for good. I knew that I wanted to have a personal page / electronic business card with a universal knowledge base, but how could it work? I figured that conventional forms of knowledge bases (such as wikis) would not do much good for my purpose and would not fit the convention of a personal website, so I knew I needed to focus more on multimedia and capturing the input and the output in my everyday operation, in the most captivating way possible.  Eventually, the knowledge base part should be "switched on" in the form of a blog, as the demand for knowledge sharing arises again, or when I have some important thoughts, I would like to persist in the form of text or multimedia. However, since then, I realized one more thing – at this point, I did not have a strong need for hosting a public knowledge base. I could only go with a personal page, but this and previous experiences taught me that if I do not approach this project at its full complexity now, I will probably never get to do it in the future. Therefore, I was looking for some robust solution to fit with my highly dynamic needs, easy to maintain, and expand. I knew that this solution should be entirely mine, so I was looking at work from scratch. Without knowing the final target, I started to see this project as a big unknown without assumptions regarding strong goals. Still, more importantly – as an experiment, that cannot disappoint me.

As the solution should be robust and scalable, I was looking at microservice architecture. On the other hand, I wanted to keep a common, more monolithic code base for most of the functionality. It seemed easier to maintain a relatively small application with a powerful back-end. With the use of containers and proper NGINX setup, such monolithic implementation can be very well microserviced. I like to use docker-compose in local development, which also saves me some resources by running one container locally for most of the back-end, instead of multiple (I decided to go with separate containers for serving images, music, videos, data, etc. on production). This configuration looks good "on paper" and proves itself. At least, I am happy with it, considering the stack I chose to implement the solution.

I decided to go with the document database MongoDB, considering the experimental character of the project and assuming the models will grow in quantity and change around already existing models containing data. Document databases give more flexibility around model changes than relational databases, where ODMs such as Mongoose for NodeJS give the well-known convenience of ORMs for relational databases. At this point, I use 27 models, and at least a few more will be used soon for the new functionality I am yet to implement. 

It was the fun part, as I wanted to process and store multimedia entirely independently from outside services, and I wanted to do it most effectively and efficiently. I needed a service for handling uploads, another one for processing uploaded files, and another (at least one) for serving processed files. The first and the last one can be simply implemented as REST endpoints, but processing takes much more time and resources, so tasks should be queued rather than hot-executed. Such service should communicate over the queue, so I implemented one (very simple, on the database) and backed it up with Redis Pub/Sub to control media processor operation and monitor tasks progress. 

Due to the experimental character of the project, some statistical system was a must-be. Usually, the obvious choice would be Google Analytics, but I also aimed at an entirely custom-made solution (because I can). I have GA integrated, but it is more for validation (eventually fail-over) purposes than actual statistics. I try to compare the results from my implementation with those from GA. When designing the solution, I made a few assumptions, which may cause certain differences, but this is a separate post material. My system (as most such systems) uses a simple cookie, does not profile a user (I do not care for my users' profile or identity – I care for their privacy), and monitors different kinds of events for quite accurate results. I am looking to watch behaviors regarding my content, not users themselves. Please keep in mind that I do not share any data I analyze (I do not monetize your data, nor my website at all).

Apart from all the technical aspects of my solution, another essential thing was content. I took a few days off before getting to writing texts for the page because I wanted my message to be as clear as it gets. I needed to approach this challenge with a clear mind. Converting a piece of code to a human-readable message would probably result in an inconveniently long text. Here I faced quite the opposite problem. The problem was to capture complex matters in a relatively simple form.