Posts on DataThrillz

I Built Agentic Search Four Ways. Here’s What Actually Matters.

Tue, 17 Mar 2026 20:49:40 +0000

While traditional RAG relies on static vector lookups that often lose global context, agentic search transforms retrieval into a dynamic, reasoning-driven process. By utilizing hierarchical structures like RAPTOR, Knowledge Graph RAG and autonomous sub-agents, these systems can navigate complex, multi-hop queries that typically overwhelm standard semantic search. This shift from one-shot retrieval to iterative loops allows for parallelized processing and self-correction, ultimately providing the precision and structural awareness required for professional-grade document analysis.

The Evolution of Agentic Search: From Naive RAG to Reasoning-Driven Retrieval

Sat, 14 Mar 2026 20:18:15 +0000

As Large Language Models (LLMs) transition from simple chatbots to autonomous agents, the methods we use to feed them data must evolve. While Retrieval-Augmented Generation (RAG) remains the industry standard for grounding models in external data, its “vanilla” implementation—converting text chunks into vectors for semantic lookup—often falters when faced with interconnected documents, technical jargon, or multi-hop queries. For Machine Learning Engineers (MLEs) and Product Managers (PMs), understanding the shift toward Agentic Search is critical. This approach moves away from static lookups toward dynamic, iterative, and hierarchical strategies that mirror how a human expert navigates a complex knowledge base.

Understanding How Claude Code Works

Fri, 06 Mar 2026 14:35:47 +0000

Inside Claude Code: How Sub-Agents and Parallel Execution Define Next-Gen Coding Agents

Introduction: The Evolution of Coding Agents

Coding agents represent a fundamental shift in how developers interact with their codebases. Unlike traditional autocomplete tools or simple code generation models, modern coding agents operate autonomously across multiple files, maintain context over extended sessions, and can break down complex tasks into manageable subtasks. These systems leverage Large Language Models (LLMs) in sophisticated agentic loops where the model can call tools, observe results, and iteratively work toward task completion.

Unveiling the Future of Code Generative AI

Wed, 12 Apr 2023 11:29:09 +0000

Picture this: generating web or phone apps is no longer a daunting task - you can simply describe your desired functionality in plain English and watch as lines of high-quality code are generated before your eyes. The ability to understand, learn and create code using cutting-edge code Generative AI (GenAI) tools has far-reaching implications, such as dramatically reducing time and effort required for software development, allowing developers to spend more time on the more creative aspects of coding. Instead of manually researching how to use various libraries, developers can manage multiple AI bots that perform coding tasks for them using powerful Large Language Models (LLMs) built on state-of-the-art deep learning techniques and trained on vast datasets. With the ability to convert human language into optimized, high quality code with astonishing accuracy and speed, the future of coding looks incredibly bright and filled with amazing innovation.

A Machine Learning Engineer’s Top 5 Predictions for the Future of Generative AI

Wed, 25 Jan 2023 12:30:00 +0000

What is GenAI?

Generative AI (GenAI) empowers end-users to generate content, such as images and text, quickly and easily. Entrepreneurs are taking advantage of this technology to create a growing number of startups that utilize GenAI models for various aspects of content creation. In the coming year, we can expect to see a proliferation of new products that build on GenAI models like titans GPT-3 and Stable Diffusion. The GenAI renaissance is just beginning and the recent boom in niche end-user applications for this technology is just the tip of the iceberg. These models will serve as the foundation for many future applications ushering in a new GenAI-economy replete with add-ons to existing software and entirely new offerings for end-users. With GenAI, the possibilities for content creation are endless and entrepreneurs are poised to capitalize on this powerful technology to revolutionize the way we create and consume media.

Considerations for building a rules engine in Python

Wed, 29 Sep 2021 08:47:08 +0000

I recently looked into how to implement a deterministic rule-based model on batches of data in Python and was surprised by the complexity of potential solutions I found. I want to implement a set of rules that when not obeyed will trigger an alert. It is basically a framework for applying a glorified set if-else/switch statements on different variables. Sounds simple, right? But not necessarily, depending on the customer’s needs. For instance, the solution becomes tricky if chaining these rules is needed which may create unpredictable system states. Let’s start by defining what a rule engine is and then discuss the potential rabbit holes that developers can fall into before delving in solution ideas. For several machine learning (ML) problems, we often need to first establish a baseline for making predictions. Without a baseline, how do we measure performance improvement or quantify when we need a new model? As a starting point to tackling data science problems, we can use heuristics or simplistic rules to establish this baseline. This allows us to directly embed domain knowledge into the solution, without spending lots of time training models. Usually, when tackling a data science problem, we get as much labelled data as possible and throw it into our favorite ML model for fitting, where the model’s parameters and hyperparameters are learned from the training data. However, there isn’t a formalized, general-purpose way of hard-coding rules from the domain expert into a machine learning model. When we take on a new data science challenge along with our business partners, we need to first establish baseline performance using a non-ML model such as a simple rule-based system. A rule engine is a set of production rules, which each has a criteria and an action. The rules are basically if-then statements that can be evaluated in any order. Rule Engines can be used as alternatives to or in tandem with more complex ML models (which is pretty common). For the purpose of this article, I will focus on the implementation of the rule engine, but people often use the term to mean a system that helps users (usually non-coders) build and evaluate rules. Other names for rule engines include expert systems, domain language systems or business rule policy.

The next coding frontier- comparing about Julia, Go & Rust with Python

Sat, 10 Jul 2021 19:05:33 +0000

Currently, Python is the dominant programming language of data science and machine learning and is popular for more general scripting. It’s pretty awesome compared to its predecessors like C/C++, FORTRAN due to its ease of use, flexibility and readability. Python also has an active and robust library culture after over 30 years of existence. However, Python has some weaknesses that newer languages like Julia, Go and Rust readily address.

Pythons Challenge Areas

Integrating Both Python & R into Data Science Workflows

Wed, 23 Jun 2021 18:28:01 +0000

These days, I highly prefer coding in Python as compared to other languages that I previously used like Matlab or R. However, I have always wondered when data science teams should use one programming language over another for certain tasks. If all team members know R and Python equally well and need to train a machine learning model, which language should they use? How could they use both Python and R without redundancies? We will discuss how to best leverage both R and Python for building data science workflows. Firstly, it really helps to know the strengths and weaknesses of Python and R. Python has overtaken R in popularity for machine learning, but R is pretty awesome at visualizing data as plots and/or dashboards. Deciding whether to exclusively use Python or R on a data science project is a big hairy topic and the answer depends on a number of factors, but this article provides lots of guidance to help data scientists make an informed decision.

Top 10 mistakes to avoid when using Hive/Impala on Hadoop

Wed, 23 Jun 2021 17:04:26 +0000

I recently took a deep dive into Hadoop for a project where I needed to automate the population of tables using JSONs and CSVs. Inevitably, I made some mistakes along the way and would like to share the lessons learned. By sharing them, I hope to save you some time! Here are 10 mistakes to avoid making when using Hive and/or Impala:

You must invalidate metadata in Impala, if you are working with tables in Hive & Impala. Hive and Impala work from the same data i.e. tables in Hadoop Distributed File System (HDFS), metadata in the Metastore. Impala caches the metadata of tables such that updates like drops or changing the structure of a table doesn’t get picked up automatically, so you must execute the invalidate metadata command. You can call invalidate metadata table; or invalidate metadata; to update for all tables if you have appropriate privileges. This command doesn’t work in Hive so only use Impala or call an Impala query from bash. Although Hive and Impala live on the same Hadoop cluster, the metadata for tables created using Hive does not get automatically updated on the Impala side.
Create views for external products to use instead of tables as best practice. Views can simplify complex logic or joins over multiple tables. They show the underlying logic of the query so are great for supporting legacy code. Views are also more secure as they permit the surfacing of selected columns and hiding of others from the user.
Must flatten arrays and structs before creating views for dashboards. Some dashboards only accept primitive data types in tables e.g. string and int and do not accept complex or user defined ones like arrays of structs.
To flatten arrays/structs in a column-wide manner, use Impala as it lends itself more easily to this. You could use Hive with concat_ws or posexplode depending on the nature of your data i.e. how nested and type. Impala uses the dot notation to easily access nested data types like array.
When flattening complex data types in Impala, use left outer joins to ensure that such columns aren’t removed while flattening them. I spent way too much time debugging why whole rows were removed from my table because of empty arrays ([]) in a single column. 🤦🏾‍♀️
Use partitions only when the table will be searched regularly by this field. Don’t overdo it! Partitions are directories on HDFS. Use buckets when you have a finite number of categories like states in the US.
All tables with complex data types must be stored as parquet to successfully extract or flatten in Impala. It can be tricky to create parquet tables in Impala so do it in Hive first and don’t forget to invalidate in order to actually be able see the change to the table.
Always check the structure of your JSONs before feeding them into HDFS. I got the following error: “Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)”. This obscure error meant that the JSON must start with {. I used an API that didn’t return all responses as strict JSONs, so I had to reformat all responses and check that all were wrapped in curly braces.
Dynamic partitioning is a godsend when creating external tables based on JSONs or CSVs. This avoids the need for manual parsing of the data file while reading files into Hive. It will locate the field that you wish to partition based on the schema specified. Magic!
Add metadata fields like timeLastModified or scriptname to your tables. This enables your data files and their content to be tracked or logged after it populates a Hive table. You should also consider adding comments to tables describing source queries and fields in the table.

Sources: https://www.simplilearn.com/working-with-hive-and-impala-tutorial

Software Engineering as a Data Scientist

Sat, 20 Feb 2021 17:37:51 +0000

Many of us in Data Science come from math, biology, chemistry or engineering or other non-Computer Science backgrounds, which may mean that we don’t have much experience writing and maintaining large code bases. Recently, I found myself getting frustrated with the structure of some of my code and searching for a better way to structure my code base. Let’s explore some ways that we can improve our Software Engineering skills as Data Scientists!

Using Decorators in Python

Sat, 20 Feb 2021 17:07:51 +0000

In Python, decorators allow Data Scientists to extend and modify callables, such as functions, methods and classes, without explicitly changing the callable. Using decorators can improve the readability of your code as well code flexibility and modularity. In this article, we’ll discuss why we would use decorators, how to implement decorators and give a few examples.

Use Cases for Decorators in Python

Decorators are powerful because they enable “wrapping” of functions with a master function. Some common use cases for decorators in Python include:

Binder & Repl.it

Mon, 15 Feb 2021 17:18:46 +0000

I recently discovered two great tools for easily creating interactive coding environments without installing a thing. These tools facilitate sharing of code in multiple languages and are wonderful resources for demonstrating programming concepts when teaching a course.

Binder

The first tool is called Binder, is open-source and was released in 2017. It is awesome because it allows data scientists to share their work in Python , R or Julia in a reproducible manner. Binder can be configured for Python (Anaconda or pip environment) and R (using RStudio and/or Shiny). Multiple user interfaces can be specified. For example, for Python we can use a terminal or Jupyter notebook in the repo.

Creating Projects from Cookiecutter Templates

Mon, 25 Jan 2021 18:09:45 +0000

Ever want to generate a new repo based on a predefined template? Now you can using Cookiecutter! I will show you how to easily spin up a fresh Cookiecutter repo for your latest data science project in Python.

Cookiecutter is an awesome command-line tool and Python package that creates projects (aka populates repo folders) based on cookiecutters (or project templates). What does this mean? Goodbye to manually copying and pasting old project repos. Now, you can automatically clone one of the thousands of cookiecutters or you can create your own.

Best 2021 Resources for Learning about AI/ML

Wed, 20 Jan 2021 19:10:59 +0000

For upskilling on AI/ML, I prefer taking a top-down approach i.e. starting with high level concepts then proceeding to more foundational topics (read: delve more into the theory) . I liked taking the breadth-first approach (rather than a depth-first approach) to initially understand AI/ML. Once I had a solid foundation, I easily pivoted to learning a specific topic, like masked regional CNNs, for building expertise through real-world experience.

I took multiple courses and read several authors at the same time. You want to find the right sources of information for you and there are so many options out there, so explore. Sometimes I need to read or watch something a few times from different folks before it just clicks. This ultimately accelerated my learning of concepts. Completion of courses should not be the ultimate goal as some courses cover trivial topics and others go too deep into other topics.

PyCon 2020

Thu, 14 Jan 2021 18:49:56 +0000

Hey Folks! I finally got around to watching a bunch of the talks and found several of the talks useful for improving my Python coding skills in general and/or in the context of doing Data Science. Here are some interesting talks from PyCon 2020:

Beautiful Python Refactoring video. The talk was simple but powerful in demonstrating the process of refactoring in Python.
Docker and Python: making them play nicely and securely for Data Science and ML. Talk discussed why use Docker in general and for machine learning, and how to manage security and performance while using Docker. It’s a great introduction to Docker - motivating its use case as a Data Scientist. Some tips that she shared include getting standard project templates from cookie cutter data science. Another was to use repo2docker to build a docker based on a local or remote repo. It was also very engaging.
Small Big Data: Using Numpy and Pandas When Your Data Doesn’t Fit In Memory. This talk expanded my knowledge of Numpy and Pandas for handling small big data, which happens more than you think.

Pro-tip: Watch at ~1.5 speed to save time!

Writing Awesome READMEs

Wed, 13 Jan 2021 17:08:23 +0000

This article will explore possible items to include your Git repo README file. We will discuss several items that we can include in a README and when to include them.

What is a README?

A README file is a text file that explains a project to new user. It helps users quickly understand where to find information on the project such as brief description of your project, how to install/fork it and/or how to use your code. READMEs live in the top level directory of your repo in GitHub, Bitbucket or GitLab.

Using Classes in Python

Tue, 08 Dec 2020 17:46:53 +0000

We know that you’ve probably heard of object oriented programming (OOP), but outside of designing games, when is best to use it for data science? We haven’t used OOP much until recently when we refactored a data science project code base. In this article, we give a brief refresher for OOP and discuss our top reasons for using OOP with a few examples.

When we code, we generally skew towards more of a functional style of programming, but sometimes we get frustrated with code that is too long. This means that we focus on building our code using functional modules of code and functions. However, at some point, our functions become too large to keep track of and we consider refactoring into classes.

Staying Up-To-Date on AI/ML

Tue, 24 Nov 2020 16:14:54 +0000

Great Email Newsletters on AI/ML

All newsletters are released weekly.

Import AI - AI newsletter that summarizes recent news articles and research; I enjoy how honest and succinct this newsletter is; also like that the implications of new algorithms are always discussed by Jack, who is an advocate for improved ML model explicability and data privacy.
AI News Weekly - curated list of AI-related articles; presents a brief summary of each article describing the latest news, applied use cases and ethics; has ads
AI Applied use Cases Top News - curated list of the latest AI news articles; subscribers can customize their newsletters according to topics of interest; weekly
The Batch - newsletter from deeplearning.ai more academic with discussion of recent AI research papers; great breakdown of topics e.g. what’s new, why it matters and the implications
TOPBOTS - list of various AI articles recently released on TOPBOT.com; articles cover a wide array of topics e.g. NLP, ML in marketing and interpretability; articles tend to be more technical
PyImageSearch - articles on using Python to tackle real-world computer vision and deep learning problems; very practical, well-done computer vision blog. I highly recommend Adrian’s free email courses.
ODSC - newsletter with articles spanning various topics geared to beginners, experts and everyone in between. They also offer webinars (free and paid) as well as online meetups.

Podcasts

Data Skeptic has short episodes that cover various ML concepts
TWIML AI - Formally This Week in ML & AI is hosted by Sam Charrington who asks really strong questions to the latest and greatest AI/ML researchers.
Practical AI - aims to make AI accessible to everyone

Find Practical Information on AI/ML

R vs Python

Mon, 26 Oct 2020 21:13:35 +0000

I generally reach for Python when building data science pipelines, however I discovered R before I decided to invest in learning Python. R has saved me lots of time when it came to quickly and easily preparing nice-looking plots for research. It begs the question of where is R better than Python for certain purposes? We will discuss the benefits and downsides of Python and R so that we can reach for the appropriate tool when needed, instead of treating all problems like nails when a screwdriver is required.

Export Images to PowerPoint in Python

Mon, 26 Oct 2020 16:44:51 +0000

Data Scientists spend a significant amount of time visualizing data for storytelling or conveying insights to end users of a data product. Often, the ability to succinctly and accurately explain the methods used and insights derived hinges on the medium of communication and time taken to prepare visualizations. In order to limit time spent on manually building reports or PowerPoint slides, one page at a time, we could automate the process. Let’s automatically create a slide deck containing plots created in Python. It’s not too hard, but there are a few tips that definitely make the process easier. We will use the python-pptx library read in template PowerPoint files and create new slides with exciting content. Python-pptx has the capability to add text, SmartArt, and other media formats to slides based on a template file.

Exporting Richly Formatted Text In Python

Wed, 07 Oct 2020 22:36:50 +0000

Today, I wondered whether I could automatically save an image of colored text from the Python console. I was looking for a way to display very long strings that automatically wrapped to its container, so I avoided the dreaded run-on string that never ends. Also, could I save the image elegantly with high resolution? In this article, I will discuss potential approaches for getting a file with nice-looking colored text that can be programmatically generated.

Fun Hacks for your Python Console

Sat, 12 Sep 2020 12:37:09 +0000

Hack #1: Color Text in Your Terminal

You can change the colors of text shown in your Python terminal console using ANSI escape character sequences! Or you can use the colorama library to make the process a bit easier and more streamlined. Colorama works across all the platforms i.e. Windows, Mac OS and Unix.

First, we import the modules that we need from colorama. [code] from colorama import Fore, Back, Style [/code]

Building Recommendations Systems

Mon, 07 Sep 2020 18:00:48 +0000

Recommendations systems are good for matching users to their favorite products and are incredibly popular. In fact you have likely used a recommendation system at least once in your life. For example, Amazon uses recommendation systems to suggest new exciting products to purchase based on users’ previous purchase patterns and those similar users. Netflix also utilizes recommendation systems to suggest new TV Shows and movies.

Before we get into recommendation systems, it is important to briefly cover two general-purpose approaches for identifying target customer groups and making product recommendations. These two approaches are called Clustering and Association Rules.

Recurrent Neural Networks in PyTorch

Mon, 07 Sep 2020 17:13:17 +0000

Feed forward networks cannot learn from the past, but Recurrent Neural Networks (RNNs) can learn by accepting data in a sequence. Examples of applications for RNNs include the text autocomplete feature on your phone and performing language translations.

Recurrent Neurons (RNs) act as the building blocks of RNNs. The difference between RNs and feed forward neurons is that RNs accept input x, at time t, as well as a hidden state or output from time t-1 from another RN. The output of a RN is a vector, unlike for a feed forward neuron. RNNs are trained (i.e. their weights are calculated) using backpropagation via Gradient Descent Optimization in time. Output from a single RNN layer at time instance,t is an input to the next layer. Each layer in the RNN represents an instance in time.

Fastai’s Practical Deep Learning for Coders Course Release - 2020 Update!

Thu, 03 Sep 2020 17:48:01 +0000

On Aug 21st, 2020, fastai released a new version of their Practical Deep Learning for Coders -Part 1 course. This course is a must-take for new and intermediate deep learning practitioners. It is well done and teaches you intuition without drowning you in theory. The only prerequisites are some high-school math, and a year of coding experience (preferably in Python). This course is free and can be done without any installation, by taking advantage of the Colab and/or Gradient platforms, which provide free, GPU-powered Python notebooks. Go here to learn more about the course.

How Genetic Algorithms Work

Mon, 10 Aug 2020 16:44:04 +0000

Genetic algorithms (GAs) are inspired by biology where only the fittest genes survive. It is based on Charles Darwin’s Natural Selection theory. We start with 2 parent chromosomes that each contain an ordered set of genes. Each parent contributes some of their genes when they mate to create children chromosomes. There is a randomness to the mating process so that each child has a diverse set of genes. This diversity is created by the crossover and mutation processes. Over time and with sufficient genetic diversity, the fittest genes, representing optimal characteristics for the species to survive, be come dominant and are propagated. This is nature’s way of optimizing over genetic diversity and we can co-opt this approach for tackling other optimization problems.

Regression & Classification Models in PyTorch

Tue, 05 May 2020 23:41:00 +0000

The purpose of this article is to share what I learned from a recent PyTorch course. We will share general machine learning tips as well as insights specific to deep learning library PyTorch.

PyTorch is a deep learning library for Python and was created by Facebook in 2016. PyTorch is good for deep learning beginners. There are several other popular deep learning frameworks such as TensorFlow, Keras, Chainer, and ONNX. TensorFlow was developed by Google and now includes Keras (previously a separate framework). I chose to deepen my knowledge in PyTorch because it is easy to learn and is commonly used for deep learning. Fastai is another deep learning open source library that is really awesome and intuitive to use. While using Fastai, I found myself curious about the inner workings so it’s a plus that fastai wraps PyTorch as I can gain a better understanding of both libraries at once. Two birds, one stone!