As Large Language Models (LLMs) transition from simple chatbots to autonomous agents, the methods we use to feed them data must evolve. While Retrieval-Augmented Generation (RAG) remains the industry standard for grounding models in external data, its “vanilla” implementation—converting text chunks into vectors for semantic lookup—often falters when faced with interconnected documents, technical jargon, or multi-hop queries. For Machine Learning Engineers (MLEs) and Product Managers (PMs), understanding the shift toward Agentic Search is critical. This approach moves away from static lookups toward dynamic, iterative, and hierarchical strategies that mirror how a human expert navigates a complex knowledge base. ...

Inside Claude Code: How Sub-Agents and Parallel Execution Define Next-Gen Coding Agents Introduction: The Evolution of Coding Agents Coding agents represent a fundamental shift in how developers interact with their codebases. Unlike traditional autocomplete tools or simple code generation models, modern coding agents operate autonomously across multiple files, maintain context over extended sessions, and can break down complex tasks into manageable subtasks. These systems leverage Large Language Models (LLMs) in sophisticated agentic loops where the model can call tools, observe results, and iteratively work toward task completion. ...

Picture this: generating web or phone apps is no longer a daunting task - you can simply describe your desired functionality in plain English and watch as lines of high-quality code are generated before your eyes. The ability to understand, learn and create code using cutting-edge code Generative AI (GenAI) tools has far-reaching implications, such as dramatically reducing time and effort required for software development, allowing developers to spend more time on the more creative aspects of coding. Instead of manually researching how to use various libraries, developers can manage multiple AI bots that perform coding tasks for them using powerful Large Language Models (LLMs) built on state-of-the-art deep learning techniques and trained on vast datasets. With the ability to convert human language into optimized, high quality code with astonishing accuracy and speed, the future of coding looks incredibly bright and filled with amazing innovation. ...

What is GenAI? Generative AI (GenAI) empowers end-users to generate content, such as images and text, quickly and easily. Entrepreneurs are taking advantage of this technology to create a growing number of startups that utilize GenAI models for various aspects of content creation. In the coming year, we can expect to see a proliferation of new products that build on GenAI models like titans GPT-3 and Stable Diffusion. The GenAI renaissance is just beginning and the recent boom in niche end-user applications for this technology is just the tip of the iceberg. These models will serve as the foundation for many future applications ushering in a new GenAI-economy replete with add-ons to existing software and entirely new offerings for end-users. With GenAI, the possibilities for content creation are endless and entrepreneurs are poised to capitalize on this powerful technology to revolutionize the way we create and consume media. ...

I recently looked into how to implement a deterministic rule-based model on batches of data in Python and was surprised by the complexity of potential solutions I found. I want to implement a set of rules that when not obeyed will trigger an alert. It is basically a framework for applying a glorified set if-else/switch statements on different variables. Sounds simple, right? But not necessarily, depending on the customer’s needs. For instance, the solution becomes tricky if chaining these rules is needed which may create unpredictable system states. Let’s start by defining what a rule engine is and then discuss the potential rabbit holes that developers can fall into before delving in solution ideas. For several machine learning (ML) problems, we often need to first establish a baseline for making predictions. Without a baseline, how do we measure performance improvement or quantify when we need a new model? As a starting point to tackling data science problems, we can use heuristics or simplistic rules to establish this baseline. This allows us to directly embed domain knowledge into the solution, without spending lots of time training models. Usually, when tackling a data science problem, we get as much labelled data as possible and throw it into our favorite ML model for fitting, where the model’s parameters and hyperparameters are learned from the training data. However, there isn’t a formalized, general-purpose way of hard-coding rules from the domain expert into a machine learning model. When we take on a new data science challenge along with our business partners, we need to first establish baseline performance using a non-ML model such as a simple rule-based system. A rule engine is a set of production rules, which each has a criteria and an action. The rules are basically if-then statements that can be evaluated in any order. Rule Engines can be used as alternatives to or in tandem with more complex ML models (which is pretty common). For the purpose of this article, I will focus on the implementation of the rule engine, but people often use the term to mean a system that helps users (usually non-coders) build and evaluate rules. Other names for rule engines include expert systems, domain language systems or business rule policy. ...

Currently, Python is the dominant programming language of data science and machine learning and is popular for more general scripting. It’s pretty awesome compared to its predecessors like C/C++, FORTRAN due to its ease of use, flexibility and readability. Python also has an active and robust library culture after over 30 years of existence. However, Python has some weaknesses that newer languages like Julia, Go and Rust readily address. Pythons Challenge Areas ...

These days, I highly prefer coding in Python as compared to other languages that I previously used like Matlab or R. However, I have always wondered when data science teams should use one programming language over another for certain tasks. If all team members know R and Python equally well and need to train a machine learning model, which language should they use? How could they use both Python and R without redundancies? We will discuss how to best leverage both R and Python for building data science workflows. Firstly, it really helps to know the strengths and weaknesses of Python and R. Python has overtaken R in popularity for machine learning, but R is pretty awesome at visualizing data as plots and/or dashboards. Deciding whether to exclusively use Python or R on a data science project is a big hairy topic and the answer depends on a number of factors, but this article provides lots of guidance to help data scientists make an informed decision. ...

I recently took a deep dive into Hadoop for a project where I needed to automate the population of tables using JSONs and CSVs. Inevitably, I made some mistakes along the way and would like to share the lessons learned. By sharing them, I hope to save you some time! Here are 10 mistakes to avoid making when using Hive and/or Impala: You must invalidate metadata in Impala, if you are working with tables in Hive & Impala. Hive and Impala work from the same data i.e. tables in Hadoop Distributed File System (HDFS), metadata in the Metastore. Impala caches the metadata of tables such that updates like drops or changing the structure of a table doesn’t get picked up automatically, so you must execute the invalidate metadata command. You can call invalidate metadata table; or invalidate metadata; to update for all tables if you have appropriate privileges. This command doesn’t work in Hive so only use Impala or call an Impala query from bash. Although Hive and Impala live on the same Hadoop cluster, the metadata for tables created using Hive does not get automatically updated on the Impala side. Create views for external products to use instead of tables as best practice. Views can simplify complex logic or joins over multiple tables. They show the underlying logic of the query so are great for supporting legacy code. Views are also more secure as they permit the surfacing of selected columns and hiding of others from the user. Must flatten arrays and structs before creating views for dashboards. Some dashboards only accept primitive data types in tables e.g. string and int and do not accept complex or user defined ones like arrays of structs. To flatten arrays/structs in a column-wide manner, use Impala as it lends itself more easily to this. You could use Hive with concat_ws or posexplode depending on the nature of your data i.e. how nested and type. Impala uses the dot notation to easily access nested data types like array. When flattening complex data types in Impala, use left outer joins to ensure that such columns aren’t removed while flattening them. I spent way too much time debugging why whole rows were removed from my table because of empty arrays ([]) in a single column. 🤦🏾‍♀️ Use partitions only when the table will be searched regularly by this field. Don’t overdo it! Partitions are directories on HDFS. Use buckets when you have a finite number of categories like states in the US. All tables with complex data types must be stored as parquet to successfully extract or flatten in Impala. It can be tricky to create parquet tables in Impala so do it in Hive first and don’t forget to invalidate in order to actually be able see the change to the table. Always check the structure of your JSONs before feeding them into HDFS. I got the following error: “Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=2)”. This obscure error meant that the JSON must start with {. I used an API that didn’t return all responses as strict JSONs, so I had to reformat all responses and check that all were wrapped in curly braces. Dynamic partitioning is a godsend when creating external tables based on JSONs or CSVs. This avoids the need for manual parsing of the data file while reading files into Hive. It will locate the field that you wish to partition based on the schema specified. Magic! Add metadata fields like timeLastModified or scriptname to your tables. This enables your data files and their content to be tracked or logged after it populates a Hive table. You should also consider adding comments to tables describing source queries and fields in the table. Sources: https://www.simplilearn.com/working-with-hive-and-impala-tutorial

Many of us in Data Science come from math, biology, chemistry or engineering or other non-Computer Science backgrounds, which may mean that we don’t have much experience writing and maintaining large code bases. Recently, I found myself getting frustrated with the structure of some of my code and searching for a better way to structure my code base. Let’s explore some ways that we can improve our Software Engineering skills as Data Scientists! ...