Disclaimer: I work on Firebase but I'm always speaking for myself on Hacker News.
This looks really cool! Honestly I think the Firebase comparison may be throwing some people off here because this is a SQL-based system, which means there's a huge base of existing tools/techniques/knowledge to build from.
I like any tool which makes it easier to build an app. It's 2020 and we still start every app like this:
* Pick a database
* Spin up a server
* Connect the DB to the server
* Create a REST API for the server so the client can talk to the DB
* Somehow make that secure enough
* Write a bunch of CRUD code on the client
What a waste of time! Glad to see people like Supabase taking on this problem as well.
Maybe I am an old fart, but what’s wrong with the steps you mention? What would your ideal sequence of steps be?
If the answer is “just call an api to handle your data” sure that works for POCs/Small apps. But I’m a bit hesitant to put all my businesses data inside a proprietary data store that I don’t control and which isn’t collocated with my app (Ie every data store request makes a round trip to the google server hosting the firebase app... maybe not a huge deal if the app itself is in google cloud?)
Disclaimer: I’ve only used firebase tangentially, haven’t built an app from scratch with it.
It's not bad doing it once. Furthermore I'd suggest you do it truly from scratch once just for the learning experience (no framework [Go makes this easy] or from TCP socket [synchronous with Python is easy]). However it's ridiculous going through the same sequence for every single app you build over a career. Nothing changes, it's all just boilerplate.
I don't want to ever write this sort of boilerplate code again, I want it generated from a DB schema or something similar [0]. There are better things to spend time on.
To me there's a sliding scale between productivity, where you use a heavy framework like Django and Rails to do everything for you, and control, where you write boilerplate to stitch all your favorite single-purpose libraries together using your preferred patterns.
They each have their purposes. Django will get you to market fast with all the features you need, and keep you there for a long time. But it forces (through its library structure) and encourages (through its common patterns) ridiculously tight coupling.
I work on a Django Monolith now that runs an org needing to grow beyond it. We need something not quite offered by a Django library, or we need to move something with different scaling needs out to another service - it's all miserably difficult, because they followed all the Django recommended best practices The framework controls you, you don't control it.
Now we're back to writing boilerplate to enforce a semblance of clean architecture onto it. It's kind boring sometimes, but once a domain gets refactored out of the Django way, our ability to deliver features quickly and safely in that domain goes up 10x.
The "Fat Models" recommendation is one of the most destructive in my opinion: https://django-best-practices.readthedocs.io/en/latest/appli..., along with Django Rest Framework "Model Serializers". A JSON serializer that talks directly to the database is just madness.
So just don't use fat models. The only sensible way to use Django is to put all the business logic in service methods, not in models/managers or serializers/forms.
If all your business logic is in models then of course your app is going to be completely unmaintainable and it's going to take developers weeks to do things that should normally take a couple hours.
There is definitely a real problem in the Django community where lots of people have recommended architecting apps in bad ways, so then you get developers who want to implement the app the "standard" way that Two Scoops or whatever recommends. But Django itself is still a great tool, you just need to be willing call out your teammates if they're unable to think for themselves.
I was just acquired into a team that enthusiastically recommended that book. Are there any alternative references I could look at or point to as alternatives? I've used a good bit of Flask but don't have much experience with Django.
The book is actually worth reading, there are just some things that I strongly disagree with. The reason I'm writing my own guide is because there isn't anything else there that I like.
I’m writing a Django style guide since all the existing ones are bad. If you send me an email then I’ll send you a draft, so that you have something to show your coworkers.
Absolutely. There's a big curve to understanding Django, Rails, .NET well enough to be able to prototype a real application. There's an even bigger curve to doing that in a maintainable way.
I think it's good to get familiar with a variety of ways of building applications over a career so you can pull from the best of them to, again, be able to focus on _business problems_ you have and can solve. To me that includes a sustainable development model and system architecture.
TL;DR Django models are the database, which makes them the wrong choice for presenting a service-layer interface to the persistence. They are inherently unable to hide, encapsulate, and protect implementation details from consumer that don't care or shouldn't be allowed to access.
The Django model is a representation of the database state. It's an infrastructure-layer object. It's is _very_ tightly coupled to the database.
Your business needs should not be so coupled to the database! While it is very helpful for an RDB to accurately model your data, a database is not an application. They have different jobs.
(The TL;DR of the following paragraph is "encapsulation and interfaces")
Your business logic belongs in the "service layer" or "use case layer". The service layer presents a consistent interface to the rest of the application - whether that is a Kafka producer, the HTTP API views, another service, whatever. Your service layer has sensible, human-understandable methods like "register user" "associate device with user", whatever. These methods are going to contain business logic that often needs to be applied _before_ a database model ever exists, or apply a bunch of business logic after existing models are retrieved in order to present a nice, usable, uncluttered return value. Your service layer hides ugly or unnecessary details of the database state from the rest of the application. Consumers shouldn't care about these details, they shouldn't rely on them (so you can fix or change without breaking the interface) , and they very probably should not be presented direct access to edit whatever they want.
If you do not do this and instead choose the fat models method all of the following will happen:
1. You will repeatedly write that business logic everywhere
where you use the models. You'll write it in your serializers, your API views, your queue consumers/producers, etc. You'll never write it the same way twice and you damn sure won't test it everywhere.
2. You'll get tired of writing the same thing and you will add properties or methods on the model. This is the Fat Model! This might be appropriate for convenience property or two that calculates something or decides a flag from the state of the model, but that's it. As soon as you start reaching across domains and implementing something like "register device for user" on the user model, or the device model, you are just reinventing a service layer in a crappy way that will eventually make your model definition 4000 lines long (not even remotely an exaggeration).
3. Every corner of your application will be updating the database - via the model - however it wants. They will rely on it! Whole features will be built on it! Now when it's time to deprecate that database field or implement a new approach, too bad. 20 different parts of your app are built on the assumption that any arbitrary database update allowed by the model is valid and a-ok.
Preferred approach:
1. Each domain gets a service layer, which contains business logic, but also presents an nice reliable interface to anything else that might consume that domain. This interface includes raising business logic errors that mean something related to our business logic. It does not expose "Django.models.DoesNotExist" or "MultipleObjectsReturned". It returns an error that tells the service consumer what went wrong or what they did wrong.
2. The service layer is the only thing that accesses or sees the Django models aka the database state. It completely hides the Django models for its domain from the rest of the application. It returns dataclasses or attrs, or whatever you want to use. The models are no longer running rampant all over the application getting updated and saved willy nilly. The service layer controls what the consumers in the rest of the application can know and do.
You will write more boilerplate. It will be boring. You will write more tests. It will be boring. But it will be reliable and modular and easier to reason about, and you can deliver features and changes faster and with much less fear of breakage.
Your business logic will live one place, completely decoupled, and it can be tested alone with everything else mocked.
How your consumers (like API views)turn service responses and errors into external (like HTTP) responses and errors, lives in one place, completely decoupled, and can be tested alone with everything else mocked.
Your models will not need to be tested because they are just a Django model. They don't do anything that's not already 100% tested and promised by the Django framework.
We started moving off "fat models" at my job and onto DDD (service methods, entities, etc.), and I have to say after a year I'm not a fan. Here are my beefs:
1. If you're not using models, it's a lot of work to stay fast.
If you've got a Customer instance, and you want to get customer.orders, you've got a problem if it's not lazy. If it's a queryset, you get laziness for free, if it isn't you have to build it yourself. God help you if you have anything even remotely complicated. You also need trapdoors everywhere if you want to use any Django feature like auth, or Django libraries.
2. You have to build auth/auth yourself
Django provides really nice auth middleware and methods (user_passes_test).
3. Service methods only do things something else should be doing.
You might be doing deserialization, auth/auth checks, database interactions, etc. All of that stuff belongs at a different layer (preferably abstracted away like @user_passes_test or serializers).
4. The model exposed by Django and DRF is actually pretty good, and you'll probably reimplement it (not as well)
The core request lifecycle is:
request -> auth -> deserialize -> auth -> db (or other persistence stuff) -> business stuff -> db (or other persistence stuff) -> serialize -> response
We've reimplemented all of those layers, and since we built multiple domains we reimplemented some of them multiple times. It probably would've been better to just admit "get_queryset" and the like are good ideas.
5. Entities are a poor substitute for regular objects and interfaces.
We've mostly ended up wrapping our existing models in entities, but just not implementing most of the properties/fields/attributes/methods. But again, we have to trapdoor a lot, we have trouble with laziness and relationships in general, and we have a lot of duplicate code in our different domains.
6. We have way too many unit tests.
Changing very small things requires changing between 5-10 tests, each of which use mocks and are around a dozen lines at least. Coupled with the level of duplication, this has really slowed us down. They also take _forever_ to run.
FWIW I think you're right about jamming too much into models; I think that works at a small scale but really breaks down quickly. I think at this point, my preferences are:
1. Ideally, your business logic should be an entirely separate package. It shouldn't know about HTML, JSON, SQL, transactions, etc. This means all that stuff (serialization, persistence) is handled in a different layer. Interfaces are your friend here, i.e. you may be passing around something backed by models, but it implements an interface your business logic package defines.
2. The API of your business logic package are the interfaces you expose and document. The API of your application is your REST/GraphQL/whatever API--that you also document.
3. Models should be solely database-specific. If you're not dealing with the database and joins and whatever, it doesn't go in models and it doesn't go in managers.
4. Don't make a custom user model [1].
5. Serialization, auth, and persistence should be a declarative and DRY as possible. That means class-level configuration and decorators.
6. Bias strongly against unit tests, and rely more strongly on integration tests. Also consider using them during development/debugging, and removing them when you're done.
Does that seem reasonable to you? I spend a lot of time thinking about this stuff, and I would like my life to be less about it (haha) so, any insight you can give would be super appreciated.
I think we're agreeing on the majority of this. We have not chucked DRF or Django auth or anything. We've just created service layers to take the business logic out of the API views, API serializers, and DB models.
Each action looks like
1. Request arrives into the app, auth happens using DRF on the API view. This is all using Django & DRF built-ins.
2. In the API view: request data gets serialized using DRF serializers, but no calculated fields or model serializers or other BS. JSON -> dict only. The dict does not have models in it, only IDs: profile_id, reservation_id, whatever. Letting the "model Serializers" turn a JSON location ID into a Location model is how you get 10 database queries before you've done _anything_. At this point we don't care if the location_id is valid. We are just serializing.
3. Still in the API view: Dict dump from the serializer gets shoved into whatever format you're going to send to the service layer. For us this is often an attrs/dataclass. If we're calling the "Reservations Service" method "create reservation", we pass in location_id, start time, end time, and the User model. The User model in this case is breaking our policy of not passing models through the service boundary, but it's the one exception for the entire code base, because it's too useful not to take getting it for free from DRF's user auth. We would be basically throwing it away then re-calling for it in the service layer which is dumb.
4. Call the Reservations Service layer. The service layer is going to do n things to try to create the reservation. If it needs to insert related records, like in a transaction, cool. Its job is to provide a sane interface for creating a Reservation, and whatever related side effects, not to only ever touch the Reservation model/table and nothing else. The base of our Domain is Reservation, creating a ReservationReceipt and a ReservationPayment are entirely within scope. Use the Payment model directly to do this if there's zero extra logic to encapsulate, or create a Payment service if you have a ton of Payment-creation logic you need to extract/hide from the Reservation service. You can still manage it all in a transaction if you want. The point is that the caller (the API layer) doesn't see this. It only sees that it's calling the Reservation Service.
5. The Reservation service will either return a dataclass/attrs objects representing a successful Reservation created, or raise a nice business error like ReservationLocationNotFound (remember when you passed in a bad location id to the API, but we didn't want to check it in the API layer?)
6. API View takes the service response & serializes it back, or takes the business error and decides which HTTP error it should be.
Got it, yeah that makes sense. At a previous job, we invested pretty heavily in model serializers, but yeah they’re bonkers slow. Thanks for weighing in, really nice to talk about this stuff with someone with a lot similar experience.
Who does the same app over and over, and if then why not just copy it? With new tech there are a lot of churn, who knows Google will announce the shutdown of the db service tommorow, and next week you need to rewrite the app in the latest version of the framework. Meanwhile old boring tech will still work just fine 20 years from now.
I can't copy code I wrote at the previous employer. It's best to learn and stick to a framework, or write something yourself once you can steal from. I'm doing the latter with code generation.
Modern programmers don't get paid to write down the code per say, you both write code as well as do the engineering part. Or maybe if you are using a framework you are just typing code? I dunno. But if you have already solved a problem, it will take significantly less time to re-write, and you could probably make it better too as you know the weakness of the old implementation.
Until you forget it again. I'd like to never worry about the boilerplate class of problems again (CRUD, auth, auditing, filtering, pagination, etc.). That's what I'm working toward.
Rails is probably one of the best ways to not waste time. I knew they had some of this capability but I wasn't sure if they generated the whole controller, fully hooked up to the models, too.
The only thing that turns me off about Rails and .NET is that it's a big learning curve _because_ the projects are so feature complete. Since I haven't worked professionally in either of them, I haven't been able to get myself to learn either well enough to do rapid prototyping.
Agreed, this is a real problem. I recently did a test. I have about a year of rails experience but it was so long ago that I forgot most things and had to look them up. I also wrote quite a bit of node/express but it was in the node 0.10 days so I had a lot of catch-up to do.
I started a new app and I got a basic Users and Sessions model with endpoints running (I like this test because dealing with passwords and API tokens requires writing some code instead of generating it all, but also leverages libraries).
These numbers aren't useful by themselves since you don't know the details of the work I did, but they may be useful side by side:
Express/TypeScript: Took me about 24 to 30 hours to get it all implemented. I started with no ORM (node-postgres) then tried out Knex, then settled on Slonik.js. Because of that I had to re-write some stuff a few times. However, in a bare-bones "framework" this is part of the penalty.
Rails: Took about 10 hours (mostly reading documentation and Michael Hartl's book).
I think I prefer the Node approach simply because I know exactly what is going on under that hood. That said however, if I were a second developer coming into the code base I'd prefer the Rails approach because I'd have to learn "the framework" anyway and a widespread standardized one like Rails would be preferable to learn IMHO.
In conclusion: My numbers are highly individualized and don't tell the whole story of course. In related news, I actually threw both of them away and went back to Elixir/Phoenix, my third love. I'm quite happy there at the moment, and I don't anticipate moving again.
I think your comment about Rails being more accessible since other programmers would probably know it seems to be true of any framework but (anecdotally) I question whether the hypothesis works in real life. Any fairly complex application built using frameworks seems similar at first but then there’s all kinds of custom hacks and non-idiomatic code that needs to be explained anyway. I guess at least most programmers will be familiar with the shape of the code but how import is that?
I suspect that frameworks are ideal for coding boot camps where one doesn’t need to understand the details of what frameworks do as much to be productive. And for boot amp grads perhaps a familiar framework makes the code more accessible? IDK, I’m speculating.
You make some excellent points. With small apps I find I can drop into a rails codebase and be productive in minutes, but with any complex application there's enough hacks that it's almost never true. I think the framework/rails approach doesn't really do much for complex apps. Just my anecdata as well.
If you would like to explore more TypeScript ORM Libraries TypeORM comes to mind, and recently Zapatos made it into the frontpage and although it's new it looks like a nice simple but powerful alternative, much more close to raw SQL than TypeORM's clearly abstract layer/magic sause.
just my 2 cents
Also about more general web frameworks like rails and not ORM's I personally use NextJS with TypeScript, like Ant.Design for it's react components library, use Jest for testing + Eslint + Prettier.
That makes the most of my IDE, it's like having a second pair of eyes behind you in Real Time telling you every typo between Types, Linting, and other VSCode extension goodies...
Thank you. I have been eyeing NextJS quite seriously. I'll be needing to build a non-trivial frontend application pretty soon and I've been really wondering if I should got the NextJS route.
No, sadly I have not, I can't vouch for it's production readiness for large-scale projects by telling myself first hand accounts of it, but I think you should checkout their latest blogposts/releases (Instant Reload, Dynamic Routing, etc)
I would just start with nice defaults like TypeScript, etc, which might help maintain a sane codebase in a large project.
But nextjs is really flexible, I find myself using it as my frontend -hammer-, maybe you should take my opinion with a grain of salt, but I've felt more productive coding with it than others!
I've used Firebase for a couple MVPs. It is very fast to develop on. It makes it easy to get a web or mobile app up in a few hours once you know what you're doing. It is definitely possible to be that fast using other technologies, but the learning curve is higher.
When you're just experimenting, trying to find market fit and get something to stick, Firebase is a decent solution. I haven't tried anything sizeable on it though. Eventually, I'm sure we would've migrated if we were successful enough.
I’m very junior, so take what I say with a grain of salt. But when I was determining if I should use Firebase or not, my biggest hesitation was simply that if Firebase ended today, I have to completely transition my application’s architecture.
Though that is unrealistic, even thinking about it was enough to make me want to be in full control of my application’s various layers.
You are bringing a senior consideration to the table; they can and do change their pricing at any time for any reason. Also, every single day you rely on them is a day that your architecture grows on that volatile cornerstone. Would look at https://parseplatform.org/ or https://postgrest.org/ as alternatives; turnkey, hosted solutions are available for each.
This is not a junior consideration and it's not unrealistic that firebase ends in say, the next 60 days which in terms of porting your entire application is essentially today. You're right to weigh this as a concern in evaluating approaches and everyone should, regardless of experience.
All PaaS providers work hard to couple you to their platform not (necessarily) for nefarious reasons, but because that's how abstractions work. You need to be continuously vigilant and aware of when and how you are tied, do a cost/benefit analysis and have risk mitigations for things like your original thought, so good for you.
I expect that our business logic would eventually have outgrown rules and cloud functions.
We used algolia to do full text and geo search. It worked, but I expected eventually it'd be really painful to reindex.
Same sort of thing with analytics. Firebase analytics is pretty powerful, but eventually we would want all our data mirrored somewhere we could use regular BI tools.
Then there is cost. We would have to weigh the cost of rewriting against the GCP bill, but at some point I expect it'd make sense.
Take this all with a grain of salt. We didn't get far enough to test any of these assumptions.
You can leverage Postgres full-text search, and GeoJSON/PostGIS support. Also for replacing algolia ElasticSearch comes to mind, that should work for most of those use cases.
For analytics and in the SaaS part you have both mixpanel and segment, and also some nice privacy-aware alternatives to Google analytics like fathom and some other indie startup.
Recently some open source segment alternative came along in HN...
MS PowerBI connects with lots of stuff, although it's paid
ELK is open source
Also metabase is a nice open source tool for BI.
What I was trying to say is that I was anticipating having to write and maintain a bunch of glue between firebase and these tools and maybe eventually outgrowing this glue.
A benefit of using a traditional SQL database is much of this stuff exists already off the shelf.
Maybe Firebase is there or is getting there. It's almost two years since I built something on it.
What's more, in the days of IaaS you can just as easily spin up a new server in some web interface, copy-paste a setup script you wrote once and after getting a coffee you come back to a working database server that does 90% of what you need. How much automation we use is independent of how much of this automation we decide to outsource.
EDIT: But hey, maybe that's just me not understanding the 2020 mentality of "Take my data and make it work" well enough
If pouchdb was better maintained I feel like it would be much more popular. IBM should really sponsor it considering they even list it as a cloudant client
> Honestly I think the Firebase comparison may be throwing some people off here because this is a SQL-based system
You're correct - we're building on top of Postgres so it's not a perfect comparison. And we're a bit early to even make it a fair claim.
I'm glad the crowd here has been so receptive. We were planning of launching in September, but it must have been picked up somewhere on the internet (thanks @habosa)
Just a counter datapoint, the Firebase comparison is what got me to click on the link and check it out. I'm a huge Firestore fan... here's what would get me to try something new:
- Better user permissions, ability to actually see my users and user permissions in a UI
- Better ability to choose conflict resolution logic
- Tied to that, ability to get "patches" from the server... right now if a change happens in my Firestore document, the server sends the entire document to the client. I'd love to be able to get a patch with just the changed data (and of course send patches to the server too, which it looks like you would support).
- Join queries to save me from needing to do multiple client -> server trips. If it's SQL I'm guessing you would support this.
Very interesting project! I'll be following your progress.
Just that if two people make changes at the same time on different clients, I don't have much control over merging those changes together with Firebase (that I know of). I think it's just a "last change wins" type system.
This looks awesome. Realtime and easy access to Postgres. However, I'm not sure where the Firebase comparison comes in. You can't really even say it is a Realtime DB or Firestore comp since those are no-sql.
You might want to take off the Firebase comp until ready otherwise it distracts from the cool offering.
If you are adding Auth the Firebase way, remember to add an UI for JWT claims. Right now Firebase does not explicitly expose the list of claims UI through the dashboard so everytime you change permissions you have to programmatically iterate through everything if you want a holistic overview.
Will you provide or are looking into passwordless logins? I currently deal with a lot of support tickets with most of them being for reseting of passwords.
We are still planning our auth system, but this passwordless logins is something that is technically simpler than third-party logins, so I imagine we will. Feel free to create a github issue if you want to track our progress on this one.
I can't help but mention that I rather like the alternative approach of keeping steps 1-3, and using Phoenix LiveView (or C#'s Blazor?) for the rest. Not saying it's better or worse, but I quite like it.
I read "skipping steps 1-3" instead of "keeping steps 1-3", my bad.
Do you have good resources for learning Elixir with OTP and Phoenix ? I also believe LiveView can deliver the 80% of functionalities for 20% of the price, so I'm very interested in learning it
I can also recommend Programming Phoenix. I've also heard good things about the LiveView courses on the Pragmatic Studio.
"Programming Elixir" and "Designing Elixir Systems with OTP" are also good, the former more so if you get started. From what I remember "Programming Phoenix" was enough to get started though, and these two books are better at getting deeper knowledge of Elixir/OTP.
This looks really cool! Honestly I think the Firebase comparison may be throwing some people off here because this is a SQL-based system, which means there's a huge base of existing tools/techniques/knowledge to build from.
I like any tool which makes it easier to build an app. It's 2020 and we still start every app like this:
What a waste of time! Glad to see people like Supabase taking on this problem as well.