The setup mentioned in the previous posts though took several months of learning and experimentation. It ended up being bulk and complicated. It was not something that could be maintained by myself or a small team that I could eventually assemble.
I ended up here by searching for the best language or tools for building a web app rather than starting with the question of what was the most important value that the product delivers and what was required to do the same. This leads me to question the whole need for nodeJS itself.
Getting a Flask
I thought since the frontend was already ready, all I needed was was to switch the backend.
No Async was Great: Python has synchronous execution. This was very easy to reason with when compared to Nodejs.
A Single Backend: There is no need to communicate now between two servers just in the backend hence child process/no messaging framework/rest calls / two simultaneous database connections
The overhead did come down significantly. It also came with some quirks
Library Support: Having used NPM a lot it was very easy to decide which libraries are currently good. Github stars also help here. On the contrary Pypi looks very simple especially I did miss the number of weekly downloads from NPM. (You can argue it not a very useful metric however for a newbie it helps)
Mongoose is not equal to PyMongo: There are several things that I had done in mongoose that was not working in the same way with PyMongo. One such case is that after every insert I needed the entire new document, not just the ID. I used this document as an activity update to other connected users. Things like these forced me to change things in the frontend also
JSON to Dataframe: Even in the previous setup converting the MongoDB nested documents to a pandas dataframe for aggregation was messy. However since Nodejs did not have a similar data structure, it was the only way to go. But with python alone, some of these data munging that takes place in the business logic layer can be done easily in pandas. Again the whole use of storing directly in JSON was questionable as SQL seemed to be more simple to reason.
After doing more research comparing the databases especially with SQL and NoSQL types it seemed like Postgres was the best way forward. Some of the notable points were
- Real Working Applications: Well known ERP like Odoo and ERPNext use SQL database. To be precise almost all of the transaction-based applications use some kind of SQL database.
- Wide Community Support: SQL has been used for a pretty long time and also has a much wider community and support. It is also widely used by the Business Intelligence community
- JSON support: Postgres also has JSON support. This meant for some of the data for which I am unable to clearly define a schema say customer details can be stored as a JSON column type. This data can also be queried and can be later converted to columns (vice-versa).
- Full-Text Search: MongoDB offers a full-text search (beta) only on its ATLAS proprietary version. This meant that I needed to integrate with another application such as Elastic Search or Sonic for the full-text support. This adds another one dependency to be maintained. Postgres however has native full-text search support.
- Managed Databases: Managed MongoDB is offered only by the MongoDB company directly. Whereas for Managed Postgres is available with all major cloud infrastructure providers. This frees one from any vendor lockin.
- Columns are better: Thinking in columns was much easier than deciding whether to have a nested data to a separate collection. This also definitely helps in having consistency in the data structure.
- Schema is Better: Setting the type of data (string, integer, etc) in the database directly helps to prevent any ambiguity in the future. All the schema changes can be retrieved easily providing uniformity between different data model versions.
- Opensource: Not all MongoDB features are available in the opensource version like Postgres.
Server Side Templating
Django all the way
All the above changes have completed removed the MERN stack without a trace. Further explorations in python web frameworks lead me to Django.
Most of the articles mention that Flask is a microframework whereas Django is a batteries-included framework. This means that with Flask you choose to install the ORM, form rendering library, authentication, admin framework, etc of your choice.
In Django, all these come bundled with the base library itself. It has been successfully used from small to large projects alike. Further, it is often touted as the best framework for rapid prototyping. Its a fairly opinionated framework and strictly defines how things should be done. This relieves oneself from breaking head over such decisions. Things will work perfectly till you have a special requirement that cannot be satisfied with the existing methodology. Taking a look at the documentation felt kind of intimidating at first hence I took a much slower approach by sticking a good Django book I came across. I started practicing the exercises step by step until I understood how things fit together much better.
And I should say I started making progress faster. Different pieces of the puzzle seem to fit perfectly.
- Being based on python there was no need for separate integration for analysis and data munging. Since most web apps, today have some kind of analytical/ml requirement python as a choice seemed obvious.
- Django offers a very good admin interface out of the box to interact with the data. Thus building a separate interface for the CRUD functionality can be postponed.
- It integrates well with Postgres. It even comes with an SQLite DB inbuilt which is a very quick way to get started. Postgres can be integrated into production easily.
- Good templating engine with Django native and Jinja2 helps in breaking the HTML into smaller reusable modules.
- Django forms come with model validation and CSRF protection
- Offers great class-based inbuilt views to quickly render lists, detail pages, form cruds, etc
- Good support with cloud infrastructure with prebuilt images to easy deployment with just a click
- Has also native support for Memcached for caching read queries and storing session-based data.
- Django Channels also help directly with WebSocket connections for realtime goodness.
- Excellent support for bootstrap4 from the third party plugins like crispy forms, jquery datatables, etc allows you to get excellent stying and interactive with ease.
- VSCode interactive python helps you to do data munging directly in py files rather than working with Jupyter notebook and then converting it to py files or copying and pasting. Django extensions library provides the shell plus commands which lets you work with Django data models within an interactive notebook environment.
After juggling with different stacks I should admit that the current setup was much easy and gets you progress fast.
Previous Post :
Next Post :