Recently I read an article titled Train sklearn 100x faster, which is about an open-source Python module named sk-dist. The module implements a "distributed scikit-learn" by extending it’s built-in parallelisation of meta-estimator, such as,
ensemble.BaggingClassifier, etc., using spark.
It was 1AM in the morning. Wise-men and women have told me not to stay up late and use computers. However, I have too sedentary life to sleep early, I am too bored with netflix and chill, and I am too sober to dream about the next big thing since tiktok. So, I did the next best thing…
If you do not have the time to read the full article, consider reading the 30 seconds version.
If you have Machine Learning (ML) pipelines in production, you have to worry about backward compatibility of changes made to the pipeline. It may be tempting to increase test coverage, but a high test coverage cannot guarantee that your recent changes have not broken the pipeline or generated low quality results. To do that, you need to develop end-to-end tests that can be executed as part of the continuous integration pipelines. Developing such a test requires sampling the dataset that powers the…
If you do not have time, here is the 30-second version:
Engineering Manager: AI, Analytics and Data @H&M. Opening little boxes, one at a time