ML Model Repositories: The Next Big Supply Chain Attack Target

Vulnerabilities in ML Model Repositories

ML model repositories offer fertile ground for threat actors due to their accessibility and the trust placed in the models they host.

• Accessibility and Trust: ML repositories, such as Hugging Face, serve as central hubs where developers can access a wide range of pre-trained models and datasets. The ease of access and the reputation of these repositories create a sense of trust among users, making them attractive targets for malicious activity.
• Lack of Comprehensive Security Controls: Despite their popularity, ML repositories often lack robust security controls comparable to traditional open-source repositories. While efforts are made to implement measures such as malware scanning and vulnerability assessments, these controls may not be sufficient to detect sophisticated attacks.

Attackers can exploit vulnerabilities in the repository infrastructure or inject malware directly into uploaded models, posing significant risks to organizations that rely on these models for critical decision-making processes.

• Exploitable Vulnerabilities: Threat actors target vulnerabilities within the repository infrastructure to infiltrate and compromise systems. Weaknesses in namespace registration, model upload processes, and validation mechanisms provide avenues for attackers to upload malicious models undetected.
• Injection of Malware: Malicious actors may inject malware directly into uploaded models, leveraging the trust associated with the repository to propagate their malicious payloads. These injected models may appear legitimate to unsuspecting users, increasing the likelihood of their adoption and subsequent compromise of systems.

While ML models are intended to enhance efficiency and accuracy in various applications, they also present an enticing opportunity for attackers to infiltrate and compromise systems.

• Risks to Critical Decision-Making Processes: Organizations that rely on ML models for critical decision-making processes are particularly vulnerable to supply chain attacks. The compromise of a single model could have far-reaching consequences, leading to data breaches, financial losses, and reputational damage.

The upcoming Black Hat Asia presentation titled “Confused Learning: Supply Chain Attacks through Machine Learning Models” will shed light on the multiple techniques that threat actors can employ to distribute malware via ML models on platforms like Hugging Face.

• Addressing Emerging Threats: The presentation underscores the urgency for organizations to implement stringent inspection measures to safeguard against potential threats. By staying informed about emerging attack techniques and enhancing security controls, organizations can mitigate the risks associated with ML model repositories and ensure the integrity of their machine learning pipelines.

Read More: GitHub’s Autofix: An AI-Powered Tool to Automatically Fix Code Vulnerabilities

Targeting Machine Learning Pipelines

Hugging Face, one of the prominent ML model repositories, facilitates the sharing and deployment of ML models across diverse applications. Despite implementing security controls such as malware scanning and vulnerability assessments, the platform remains vulnerable to various attack vectors.

• Role of Hugging Face in ML Model Sharing: Hugging Face serves as a vital platform for developers seeking to access and integrate ML models into their projects seamlessly. By providing a centralized repository for ML tools and datasets, the platform enables collaboration and innovation within the ML community. However, the inherent accessibility of Hugging Face also exposes it to potential exploitation by malicious actors.
• Persistent Vulnerabilities in ML Repositories: Despite efforts to bolster security controls, ML repositories like Hugging Face remain susceptible to exploitation. Threat actors leverage weaknesses in namespace registration, typosquatting, and model confusion to upload malicious models undetected. These vulnerabilities pose a significant threat to unsuspecting users who may unknowingly integrate compromised models into their projects.

Adrian Wood, a security engineer at Dropbox, highlights the ease with which attackers can register deceptive namespaces resembling trusted organizations and lure users into uploading models to compromised repositories.

• Deceptive Namespace Registration: Attackers can register namespaces that mimic those of trusted organizations, deceiving users into uploading models to compromised repositories. This tactic exploits the inherent trust associated with reputable organizations, making it more likely for users to overlook potential risks.
• Techniques of Exploitation: In addition to deceptive namespace registration, threat actors employ techniques such as typosquatting and model confusion to camouflage malicious models as legitimate ones. Typosquatting involves registering domain names or repository namespaces with minor misspellings, while model confusion exploits similarities between legitimate and malicious models to evade detection.

Furthermore, techniques such as typosquatting and model confusion allow threat actors to camouflage malicious models as legitimate ones, increasing the likelihood of their adoption by unsuspecting developers. These tactics underscore the importance of exercising caution and implementing robust security measures when sourcing ML models from repositories.

• Importance of Security Awareness: The prevalence of deceptive tactics underscores the critical importance of security awareness and diligence when sourcing ML models from repositories. Developers must exercise caution and conduct thorough due diligence to verify the authenticity and integrity of models before integration into their projects. By adopting a proactive approach to security, organizations can mitigate the risks posed by malicious actors and safeguard the integrity of their ML pipelines.

Instances of Malware on ML Repositories

Recent incidents highlight the evolving threat landscape surrounding ML repositories, with threat actors actively exploiting vulnerabilities to propagate malware. Researchers at JFrog uncovered a malicious ML model on Hugging Face earlier this year, capable of executing arbitrary code upon loading. This instance underscores the potential repercussions of unverified model usage, as unsuspecting users inadvertently grant attackers full control over their systems.

• Uncovering Malicious ML Model on Hugging Face: Researchers at JFrog identified a malicious ML model hosted on Hugging Face, a popular repository for ML tools and models. This model, discovered earlier this year, posed a significant threat to unsuspecting users due to its ability to execute arbitrary code upon loading. The discovery serves as a stark reminder of the risks associated with unverified model usage in production environments.
• Implications of Unverified Model Usage: The presence of malicious models within repositories like Hugging Face highlights the potential repercussions of utilizing unverified models in ML pipelines. Inadvertently loading a compromised model can grant threat actors full control over systems, compromising sensitive data and undermining organizational security measures. As such, the incident underscores the critical importance of implementing robust security measures and thorough inspection processes when sourcing models from repositories.

Adrian Wood’s demonstration involving the injection of malware into models using the Keras library and Tensorflow further emphasizes the ease with which attackers can compromise ML pipelines. By exploiting vulnerabilities within widely-used libraries, threat actors can execute malicious code while maintaining the intended functionality of the model, evading detection mechanisms.

• Demonstration by Adrian Wood: Adrian Wood, a security engineer at Dropbox, demonstrated the injection of malware into ML models using widely-used libraries such as Keras and Tensorflow. This demonstration highlighted the inherent vulnerabilities within ML pipelines, where threat actors can clandestinely inject malicious code while preserving the model’s intended functionality. Such tactics underscore the importance of comprehensive security assessments and ongoing monitoring to mitigate the risks associated with ML model repositories.
• Significance of Comprehensive Security Measures: The incidents involving malicious models on ML repositories underscore the critical importance of implementing comprehensive security measures to safeguard against supply chain attacks. Organizations must prioritize thorough inspection processes, ongoing monitoring, and collaboration within the ML community to enhance collective defenses against evolving cyber threats. By adopting a proactive approach to cybersecurity, organizations can mitigate the risks posed by malicious actors and ensure the integrity of ML-driven applications.