When we first set out to replace our rule-based tagging system with machine learning for sustainable investing news articles, we targeted higher accuracy, reduced maintenance overhead, and improved scalability. The purpose of this article is to highlight key aspects of our successful deployment, the necessary infrastructure upgrades, and the technical insights we gained and would like to share with machine learning engineers at other firms.
Our rule-based system has processed millions of financial articles over several years using a carefully engineered keyword matching and Boolean logic framework. The architecture was straightforward:
Input processing: Text preprocessing pipeline with tokenization, stemming, and named entity recognition
Feature extraction: Keyword density calculations, n-gram matching, and financial terminology detection
Classification logic: Nested if-then rules with weighted scoring across sustainable investing categories
Our legacy, rule-based system achieved reasonable performance metrics in categorizing financial articles across environmental, social, and governance topics. It had served us well, but as that sector became more nuanced over time and article volumes grew, the limitations became clear: constant rule maintenance, missed edge cases, and lessened ability to understand context and sentiment.
For the ML replacement, we implemented a multi-label classification system using a fine-tuned BERT-based transformer architecture. Here is the model setup:
Base model: DistilBERT for inference speed optimization
Architecture: multi-task learning with shared encoder and separate classification heads for the sustainable investing categories
Training data: previously tagged articles spanning three years
Framework: PyTorch with Hugging Face Transformers
The new machine learning model we deployed showed impressive performance gains: higher precision, better recall, and the ability to catch subtle themes that our rule-based system had missed.
As we began integrating the model into our production systems, something unexpected happened: the ML model revealed that sustainable investing articles often span multiple categories with varying degrees of relevance. This created new technical challenges to resolve.
The rule-based system had been transparent by design. We knew exactly why an article received a particular tag because we had written the rules. When an article about renewable energy investments was tagged as environmental, the logic was obvious.
But our ML model was capturing patterns we hadn't explicitly programmed, surfacing connections between sustainable investing themes and business contexts that were more sophisticated than our original categorization framework.
The model's superior performance came with a trade-off: reduced interpretability. As a result, we strategically reimagined our approach to presenting sustainable investing insights to clients. Instead of simple binary tags, we began exploring:
Confidence scoring: So that we can provide higher-quality data to our clients, we now consider confidence levels that help us understand the strength of sustainable investing relevance rather than a yes or no for a specific category such as environmental, for example.
Multi-dimensional tagging: The model revealed that articles often span multiple categories in complex ways. We considered providing this greater granularity to our clients and how could we showcase it in the best way.
Contextual insights: We began surfacing not just what sustainable investing topics are present, but why they're relevant and how they interconnect with broader market themes.
Category-specific thresholds: Instead of single confidence thresholds, we implemented category-specific thresholds optimized for different precision-recall trade-offs based on downstream use cases.
Unlike our binary rule-based approach, the ML model revealed that sustainable investing articles often span multiple categories with varying degrees of relevance. This created new technical challenges. For example, with monthly model retraining we needed robust versioning and rollback capabilities. And implantation required:
Model registry: MLflow with automated tagging and metadata tracking
Staging deployments: staging traffic routing to new models with performance comparison
A/B testing framework: side-by-side comparison of model versions on production traffic
The transition to machine learning from a rule-based system brought distinct technical challenges that shaped our engineering strategy and product outcomes. Here are considerations for reliable and impactful deployments in the future.
Design for explainability: Because interpretability is essential, architecture decisions should prioritize explainable predictions over marginal accuracy gains.
Invest in confidence calibration: Raw model confidence scores are rarely well-calibrated. Post-processing calibration is essential for production systems where confidence thresholds drive business decisions.
Comprehensive monitoring: Traditional accuracy metrics aren't sufficient. Monitor feature drift, prediction drift, and business-relevant metrics continuously.
Label space evolution: Multi-label problems in production often reveal that your original label taxonomy was oversimplified. Design systems that can adapt to evolving label structures.
Infrastructure complexity: ML models require significantly more complex infrastructure than rule-based systems. Plan for model versioning, A/B testing, and rollback capabilities from day one.
Model deployment is also a product decision: The shift from rules to ML wasn't just an engineering change; it requires product strategy evolution and a client-centric vision.
Our tagging model deployment was ultimately successful. For ML engineers considering similar transitions, the technical challenges are solvable but require fundamental shifts in system architecture, monitoring approaches, and operational processes. The investment in proper ML infrastructure pays dividends not just in model performance, but in system maintainability and business agility.
This blog post is for informational purposes only. The information contained in this blog post is not legal, tax, or investment advice. FactSet does not endorse or recommend any investments and assumes no liability for any consequence relating directly or indirectly to any action or inaction taken based on the information contained in this article.