Materialize is a streaming database with a SQL API. However, despite the fact that Materialize uses SQL idioms and can process data from databases, it actually has very little in common with “databases” as most people think of them.
Link
Airflow is one of my favorite tools that I frequently use to setup and manage data science pipelines. The Airflow UI gives us a clear picture of the DAGS and its current status. I may be wrong here but from my experience, I have seen that Airflow on a single machine is not scalable. Thus, to scale Airflow, we can use Kubernetes.
Link
When there is a long running task, there are usually below 2 requirements:
As a user, I want to know the progress of the task As a user, I want to get the output of the task if it is finished We will use the out of the box features Threading and Cache in Python and Django respectively to achieve this
Link
Throughout the last 10 months, while working on PyTorch Lightning, the team and I have been exposed to many styles of structuring PyTorch code and we have identified a few key places where we see people inadvertently introducing bottlenecks.
Link
Stop writing boilerplate code, struggling with authentication and managing infrastructure. Start connecting APIs with code-level control when you need it — and no code when you don’t.
