We decided to build the initial Harness Wealth system as a simple Django monolith deployed on Heroku. This to many may seem like a surprising decision. From appointment systems to financial analytics to supporting a team of concierges and business developers, the capabilities required by our system seem much more demanding than that of the early stack. However, building a monolith in Heroku was a deliberate business decision in the early days. We were laser focused on building a product that clients would love as quickly as possible.
Initial Business Needs
As the engineering team tasked with developing this system from the ground up, we needed to balance the initial needs of the business to launch with the future needs of the business as we scaled and as system would mature.
Harness Wealth is dedicated to helping clients effectively unlock financial opportunity to achieve their best financial future through a holistic and personalized approach, driven by technology. We believe the solution centers on an experience that captures a client’s personal balance sheet and goals and helps them identify opportunities, avoid missteps, and ultimately find the best financial, tax, and legal advisors for their needs. To fulfill on that, our objective was to build a system that would enable us to experiment and test the best approach to our business need, and allows us to shape our thinking around the product as we learn from users. Additionally, our system needed to support the operations side of the business which required a user interface and CRM for our Concierge team that could integrate with the application data store. As such, the core competencies of our system were to:
- Iterate and delivery quickly to respond to learnings and introduce new features to delight
- Collect data about our application to paint a picture of our users
- Allow discoverability of this data to our product and marketing team to make decisions
- Allow customer management abilities for non-engineering teams
Initial Technical Stack
In a resource constrained environment, we took the decision to outsource as many parts of our system to frameworks and platforms-as-a-service in order to solely focus on agile feature development. We decided on a three-tier application on the onset — a client layer, a service layer, and a database layer. Other considerations around chosen solutions revolve around four additional criteria:
- Talent and community — how easy would it be to scale up the team and navigate unknowns?
- Flexibility — how far away is the inflection point of where the chosen solution is unsuitable for business needs?
- Difficulty — how big are the knowledge and operating overheads for developing features
The Application Layer — Django and Python
We also wanted to take a conservative approach with our application layer — have low or no language diversity and a monolithic service layer. This is a good solution for small teams because it makes scaling easier with the same toolkits and it is simpler to manage a single application, instead of multiple.
Besides managing a web application, Harness Wealth has data processing needs including financial analytics and various forms of data ingestion. We decided to go with Python for two reasons:
- It moved passed R to become the most popular data science language in recent years, as described here, and data science is a critical component of our business.
- It is beneficial for us from a talent acquisition perspective due to a growing number of engineers being trained in this language as described here.
The best web framework choice for our needs was Django because it provided a couple of key features that solve most of our data management and application development challenges:
- Fast API development is made possible by Django Rest Framework through declarative contracts
- A powerful admin tool that allows our Concierge team to management relationships by designing the admin as a CRM around their needs and developing admin features for the product team to export collected data on our business
- An incredibly robust ecosystem to help us enable functionality already created by the community
The Client Layer — React
React is the popular front-end solution.
Part of the reason we went for this choice initially was because our founding engineering team has a React background.
At this juncture, we have chosen not to implement redux skills given the small size of our application and the fact that the state on our pages is self-contained.
The Data Layer — Postgres
In the same conservative vein, we decided to select Postgres as our application data store largely due to the fact that it has stood the test of time, having been around for two decades, and Django also integrates well with many of our Postgres features. Some nice Postgres features that Harness Wealth has taken advantage of include the JsonB fields for schema-less data and range types.
The Infrastructure — Heroku
In the early days of Harness Wealth, our priorities were to launch our three-tiered application onto the public internet as quickly as possible. Before we brought in new clients, we wanted the product team to work asynchronously with engineering in order to test and evolve the product quickly. Thus we needed a testing environment for feature acceptance testing before deployment. We also had demands to demo our product to advisors in a more stable environment to acquire the supply side of our business. In order to meet these demands right out of the gate with little resources, we outsourced our infrastructure and development operations to Heroku, where they managed our networking, continuous deployment and integration pipelines, and monitoring and logging of our application. This allowed engineering resources to focus solely on development of the product and start growing our business.
Effective Management of Stack Limitations
We adopted the initial stack aware of and fully understanding that there are limitations on the stack that we built. However, we were comfortable and deliberate with that decision. The limitations of our stack are:
Heroku has limited network flexibility
The core problem with Heroku involving networking is the lack of I.P. whitelisting and elastic I.P. that makes it difficult for us to control ingress access to our applications on Heroku and services outside of Heroku. We made an early decision to make security top of mind for the engineering team, and was focused to having access control on sensitive application data such as logs, code revisions, and internal sites on a networking level. Without an elastic I.P. it was impossible to restrict ingress traffic from our Heroku apps. In addition, without I.P. whitelisting we had to rely on access control for API requests the application level, which was not ideal for performance reasons, and had no network control access to the Django admin.
The Monolith
Monoliths make a lot of sense. And it was was feasible, they are a great way to scale the business. Monoliths make on-call moments easy to diagnose, prevent identity drift within service teams, and make operations simpler. However, as we start to scale out our engineering team — the appeals of a microservices architecture become more fitting to the challenges that come with growth. There are three primary challenges that come with a Monolith as the business needs grow and the team starts scaling.
- Poor resource utilization: Because all of our functionality is centered around a single code base, our resource footprint is necessarily big relative to worker functionality. In our context, we use Django as a web worker and as an async worker. The async worker might just be concerned with utilizing the Django ORM for flushing processed data into a data store but mounts the entire codebase. We also have a single service level (agreement, objective, and indicators) for all APIs, which is inappropriate depending on the service.
- Difficulty segmenting the organization into teams: Take a look at Conway’s law — “Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization’s communication structure.” As we start to grow our engineering team, engineering resources can be deployed more effectively by focusing on developing their efforts on services and defining their own SLAs, SLOs, and SLIs around service boundaries. Since everything is in one project, this segmentation practice is rather limited to just engineering teams focused on developing their team’s designated functionality.
- Security and access control over data: Since the entire engineering team is operating with a single project, resource access is shared amongst everyone. Since might be undesirable as a team focused on PII information may want to limit access to members outside.
Addressing the Stack Limitations
In order to work around some of these issues, we made some initial investments in our architecture and project structure.
Stateful services into cloud solutions from day one
We deployed all of our state services, solutions that have an element persistent storage, to our cloud solution on day one. Although this caused challenges around ingress, we were able to start pumping data into resources without the need for migration later down the road.
Thoughtful architecture of the Django monolith and React project
We took the approach of creating decoupled and “service” groups around our Django project. We approached development as if we were developing a microservice and designed our service boundaries and dependencies around the Django apps. This allows us to have strict contracts and audit trail of service dependencies and will reduce overhead for breaking the monolith into services. Further decoupling takes place with the use of Django Signals which allows you to access functionality across applications and reduce module dependencies.
We took the same approach with the React project. We segmented our project using React Router to prime our project for a more micro frontend structure. Spotify is a great example of a company that designed their project in this fashion.
Operational competency by segmentation of tier 1 and tier 2
We also did not take full-stack functionality of Django and segmented the service layer and browser application into two separate projects. This allowed the initial construct of the team to focus on their respective domains, and not get bogged down with the cross-tier overhead. We felt that this decision was important because it helped us reduce cross project sprawl of the browser project. This also started the seed for developing a service oriented culture, where teams start to flex their muscles on thinking about service dependencies and contracts during development and deployment.
Conclusion
As engineers, sometimes we must not resort to tools used in the past so that we can instead fit where we are as a business. However, being mindful of those previous insights will help to navigate the growth of our system. As the scope of the business needs started to grow and required building a client service team, on-boarding new partners, and demoing to stakeholders, balancing the decisions that meet business needs and nurturing areas of growth for the future became one of our more creative and challenging hurdles.
