A few years ago, I fell into the world of anime from which I\u2019d never escape. As my watchlist was growing thinner and thinner, finding the next best anime became harder and harder. There are so many hidden gems out there, but how do I discover them? That\u2019s when I thought\u2014why not let Machine Learning sensei do the hard work? Sounds exciting, right?<\/p>\n
In our digital era, recommendation systems<\/a> are the silent\u00a0entertainment\u00a0heroes that power our daily online experiences. Whether it involves suggesting television series, creating a personalized music playlist, or recommending products based on browsing history, these algorithms operate in the background to improve user engagement.<\/p>\n This guide walks you through building a production-ready\u00a0anime recommendation engine that runs 24\/7 without the need for traditional cloud platforms. With hands-on use cases, code snippets, and a detailed exploration of the architecture, you\u2019ll be equipped to build and deploy your own recommendation engine.<\/p>\n This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n The foundation of any recommendation system<\/a> lies in quality data. For this project, datasets were sourced from Kaggle and then stored in the Hugging Face Datasets Hub for streamlined access and integration. The primary datasets used include:<\/p>\n Before we begin, ensure that you have completed the following steps:<\/p>\n Before diving into the code, it is essential to understand how the various components of the system interact. Check out the below project architecture.<\/p>\n This project adopts a modular folder structure designed to align with industry standards, ensuring scalability and maintainability.<\/p>\n The constant\/__init__.py<\/a> file defines all essential constants, such as file paths, directory names, and model filenames. These constants standardize configurations across the data ingestion, transformation, and model training stages. This ensures consistency, maintainability, and easy access to key project configurations.<\/p>\n The utils\/main_utils\/utils.py<\/a> file contains utility functions for operations such as saving\/loading data, exporting dataframes, saving models, and uploading models to Hugging Face. These reusable functions streamline processes throughout the project.<\/p>\n The entity\/config_entity.py<\/a> file holds configuration details for different stages of the training pipeline. This includes paths for data ingestion, transformation, and model training for both collaborative and content-based recommendation systems. These configurations ensure a structured and organized workflow throughout the project.<\/p>\n The entity\/artifact_entity.py<\/a> file defines classes for artifacts generated at various stages. These artifacts help track and manage intermediate outputs such as processed datasets and trained models.<\/p>\n In this project, we implement three types of recommendation systems to enhance the anime recommendation experience:\u00a0\u00a0<\/p>\n Each approach plays a unique role in delivering personalized recommendations. By breaking down each component, we will gain a deeper understanding.<\/p>\n This Collaborative Recommendation System<\/a> suggests items to users based on the preferences and behaviours of other users. It operates under the assumption that if two users have shown similar interests in the past, they are likely to have similar preferences in the future. This approach is widely used in platforms like Netflix, Amazon, and anime recommendation engines<\/a> to provide personalized suggestions.\u00a0In our case, we apply this recommendation technique to identify users with similar preferences and suggest anime based on their shared interests.<\/p>\n We will follow the below workflow to build our recommendation system. Each step is carefully structured to ensure seamless integration, starting with data collection, followed by transformation, and finally training a model to generate meaningful recommendations.\u00a0<\/p>\n Data ingestion is the process of collecting, importing, and transferring data from various sources into a data storage system or pipeline for further processing and analysis. It is a crucial first step in any data-driven application, as it enables the system to access and work with the raw data required to generate insights, train models, or perform other tasks.<\/p>\n We define a DataIngestion class in components\/data_ingestion.py<\/a> file which handles the process of fetching datasets from Hugging Face Datasets Hub, and loading them into Pandas DataFrames. It utilizes DataIngestionConfig<\/a>\u00a0to obtain the necessary file paths and configurations for the ingestion process. The ingest_data method loads the anime and user rating datasets, exports them as CSV files to the feature store, and returns a DataIngestionArtifact <\/a>containing the paths of the ingested files.\u00a0This class encapsulates the data ingestion logic, ensuring that data is properly fetched, stored, and made accessible for further stages of the pipeline.<\/p>\n Data transformation is the process of converting raw data into a format or structure that is suitable for analysis, modelling, or integration into a system. It is a crucial step in the data preprocessing pipeline, especially for machine learning, as it helps ensure that the data is clean, consistent, and formatted in a way that models can effectively use.<\/p>\n In components\/data_transformation.py<\/a> file, we implement the DataTransformation class to manage the transformation of raw data into a cleaned and merged dataset, ready for further processing. The class includes methods to read data from CSV files, merge two datasets (anime and ratings), clean and filter the merged data. Specifically, the merge_data method combines the datasets based on a common column (anime_id), while the clean_filter_data method handles tasks like replacing missing values, converting columns to numeric types, filtering rows based on conditions, and removing unnecessary columns. The initiate_data_transformation method coordinates the entire transformation process, storing the resulting transformed dataset in the specified location using\u00a0DataTransformationArtifact<\/a>\u00a0entity.<\/p>\n The Collaborative filtering is widely used in recommendation systems, where predictions are made based on user-item interactions rather than explicit features of the items.<\/p>\n The CollaborativeAnimeRecommender <\/a>class is designed to provide personalized anime recommendations using collaborative filtering techniques. It employs three different models:<\/p>\n The class processes raw user ratings, constructs interaction matrices, and trains the models to generate tailored recommendations. The recommender provides predictions for individual users, recommends similar anime titles, and suggests new shows based on user similarity. By leveraging collaborative filtering techniques, this system enhances user experience by offering personalized and relevant anime recommendations.<\/p>\n The CollaborativeModelTrainer<\/a>\u00a0automates the training, saving, and deployment of the models. It ensures that trained models are stored locally and also uploaded to Hugging Face, making them easily accessible for generating recommendations.<\/p>\n This content-based recommendation system suggests items to users by analyzing the attributes of items such as genre, keywords, or descriptions to generate recommendations based on similarity.<\/p>\n For example, in an anime recommendation system, if a user enjoys a particular anime, the model identifies similar anime based on attributes like genre, voice actors, or themes. Techniques such as TF-IDF (Term Frequency-Inverse Document Frequency), cosine similarity, and machine learning models help in ranking and suggesting relevant items.<\/p>\n Unlike collaborative filtering, which depends on user interactions, content-based filtering is independent of other users\u2019 preferences, making it effective even in cases with fewer user interactions (cold start problem).<\/p>\n We use the artifacts from the data ingestion component discussed earlier to train the content-based recommender.<\/p>\n The Content-Based recommender is responsible for training recommendation models that analyze item attributes to generate personalized suggestions. It processes data, extracts relevant features, and builds models that identify similarities between items based on their content.\u00a0\u00a0<\/p>\n The ContentBasedRecommender <\/a>class leverages TF-IDF (Term Frequency-Inverse Document Frequency) and Cosine Similarity to suggest anime based on their genre similarities. The model first processes the dataset by removing missing values and converting textual genre information into numerical feature vectors using TF-IDF vectorization. It then computes the cosine similarity between anime titles to measure their content similarity. The trained model is saved and later used to provide personalized recommendations by retrieving the most similar anime based on a given title.<\/p>\n The ContentBasedModelTrainer<\/a>\u00a0class is responsible for automating the training and deployment of a content-based recommendation model. It loads the processed anime dataset from the data ingestion artifact, initializes the ContentBasedRecommender, and trains it using TF-IDF vectorization and cosine similarity. The trained model is then saved and uploaded to Hugging Face.<\/p>\n It is common for newcomers to anime to seek out the most popular titles first. This top anime recommendation system is designed to help those new to the anime world easily discover popular, highly rated, and top-ranked anime all in one place by using simple sorting and filtering.\u00a0\u00a0<\/p>\n We utilize the artifacts from the previously discussed data ingestion component in this recommendation system.<\/p>\n The PopularityBasedFiltering<\/a>\u00a0class is responsible for ranking and sorting anime using predefined popularity-based parameters. It analyzes the dataset by evaluating attributes such as rating, number of favorites, community size, and ranking position. The class includes specialized functions to extract top-performing anime within each category, ensuring a structured approach to filtering. Additionally, it manages missing data and refines the output for readability. By providing data-driven insights, this class plays a crucial role in identifying popular and highly-rated anime for recommendation purposes.<\/p>\n The PopularityBasedRecommendor<\/a>\u00a0class is responsible for recommending anime based on different popularity metrics. It utilizes an anime dataset stored in feature_store_anime_file_path, which was a DataIngestionArtifact. The class integrates the PopularityBasedFiltering class to generate anime recommendations according to various filtering criteria, such as top-ranked anime, most popular choices, community favorites, and highest-rated shows. By selecting a specific filter_type, users can retrieve the best match based on their preferred criteria.<\/p>\n This Machine Learning Training Pipeline is designed to automate and streamline the process of building recommender models efficiently. The pipeline follows a structured workflow, beginning with data ingestion from Hugging face, followed by data transformation to preprocess and prepare the data for model training. It incorporates different modelling techniques, such as collaborative filtering, content-based approaches and Popularity-based filtering, ensuring optimal performance. The final trained models are stored in a Model Hub, enabling seamless deployment and continuous refinement. This structured approach ensures scalability, efficiency, and reproducibility in machine learning workflows.<\/p>\n Now that we\u2019ve completed creating the pipeline, run the training_pipeline.py file using the below code to view the artifacts generated in the previous steps.<\/p>\n The recommendation\u00a0application<\/a> is built using Streamlit, a lightweight and interactive framework for creating data-driven web apps. It is deployed on Hugging Face Spaces, allowing users to explore and interact with the anime recommendation system seamlessly. This setup provides an intuitive UI for discovering anime recommendations in real time.\u00a0Each time you push new changes, Hugging Face will redeploy your app automatically.<\/p>\n The Dockerfile<\/a>\u00a0sets up a lightweight Python environment using the official Python 3.10 slim-buster image. It configures the working directory, copies application files, and installs dependencies from requirements.txt. Finally, it exposes port 8501 and runs the Streamlit app, making it accessible within the containerized environment.<\/p>\n The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion.<\/strong><\/p>\n Congratulations! You have completed building the Recommendation app in no time. From acquiring data and preprocessing it to model training and deployment, this project highlights the power of getting things out there into the world! But hold up\u2026 we\u2019re not done yet! \ud83d\udca5 There\u2019s a whole lot more fun to come! You\u2019re now ready to build on something even cooler, like a Movie Recommendation app!\u00a0 \u00a0 \u00a0<\/p>\n This is just the beginning of our adventure together, so buckle up\u2014there are many more exciting projects ahead! Let\u2019s keep learning and building!\u00a0<\/p>\nLearning Objectives<\/h2>\n
\n
Anime Recommendation System with Hugging Face: Data Collection<\/h2>\n
\n
Pre-requisites for Anime Recommendation App<\/h2>\n
1. Sign Up and Log In<\/h3>\n
\n
2. Create a New Space<\/h3>\n
\n
3. Clone the Space Repository<\/h3>\n
\n
git clone https:\/\/huggingface.co\/spaces\/your-username\/your-space-name<\/code><\/pre>\n4. Set Up the Virtual Environment<\/h4>\n
\n
# Creating the Virtual environment\n\n## For macOS and Linux:\npython3 -m venv env \n## For Windows:\npython -m venv env\n\n\n# Activation the environment\n\n## For macOS and Linux:\nsource env\/bin\/activate\n## For Windows:\n.\\env\\Scripts\\activate<\/code><\/pre>\n5. Install Dependencies<\/h4>\n
\n
pip install -r requirements.txt<\/code><\/pre>\n![\"project](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-07_135539.webp\")
<\/figure>\nFolder Structure<\/h2>\n
ANIME-RECOMMENDATION-SYSTEM\/ # Project directory\n\u251c\u2500\u2500 anime_recommender\/ # Main package containing all the modules\n\u2502 \u2502\u2500\u2500 __init__.py # Package initialization\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 components\/ # Core components of the recommendation system\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py # Package initialization\n\u2502 \u2502 \u2502\u2500\u2500 collaborative_recommender.py # Collaborative filtering model\n\u2502 \u2502 \u2502\u2500\u2500 content_based_recommender.py # Content-based filtering model\n\u2502 \u2502 \u2502\u2500\u2500 data_ingestion.py # Fetches and loads data\n\u2502 \u2502 \u2502\u2500\u2500 data_transformation.py # Preprocesses and transforms the data\n\u2502 \u2502 \u2502\u2500\u2500 top_anime_recommenders.py # Filters top animes\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 constant\/ \n\u2502 \u2502 \u2502\u2500\u2500 __init__.py # Stores constant values used across the project\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 entity\/ # Defines structured entities like configs and artifacts\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502\u2500\u2500 artifact_entity.py # Data structures for model artifacts\n\u2502 \u2502 \u2502\u2500\u2500 config_entity.py # Configuration parameters and settings\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 exception\/ # Custom exception handling\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502\u2500\u2500 exception.py # Handles errors and exceptions\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 loggers\/ # Logging and monitoring setup\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502\u2500\u2500 logging.py # Configures log settings\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 model_trainer\/ # Model training scripts\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502\u2500\u2500 collaborative_modelling.py # Train collaborative filtering model\n\u2502 \u2502 \u2502\u2500\u2500 content_based_modelling.py # Train content-based model\n\u2502 \u2502 \u2502\u2500\u2500 top_anime_filtering.py # Filters top animes based on ratings\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 pipelines\/ # End-to-end ML pipelines\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502\u2500\u2500 training_pipeline.py # Training pipeline\n\u2502 \u2502\n\u2502 \u251c\u2500\u2500 utils\/ # Utility functions\n\u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u251c\u2500\u2500 main_utils\/\n\u2502 \u2502 \u2502 \u2502\u2500\u2500 __init__.py \n\u2502 \u2502 \u2502 \u2502\u2500\u2500 utils.py # Utility functions for specific processing\n\u251c\u2500\u2500 notebooks\/ # Jupyter notebooks for EDA and experimentation\n\u2502 \u251c\u2500\u2500 EDA.ipynb # Exploratory Data Analysis\n\u2502 \u251c\u2500\u2500 final_ARS.ipynb # Final implementation notebook \n\u251c\u2500\u2500 .gitattributes # Git configuration for handling file formats\n\u251c\u2500\u2500 .gitignore # Specifies files to ignore in version control\n\u251c\u2500\u2500 app.py # Main Streamlit app \n\u251c\u2500\u2500 Dockerfile # Docker configuration for containerization\n\u251c\u2500\u2500 README.md # Project documentation\n\u251c\u2500\u2500 requirements.txt # Dependencies and libraries\n\u251c\u2500\u2500 run_pipeline.py # Runs the entire training pipeline\n\u251c\u2500\u2500 setup.py # Setup script for package installation<\/code><\/pre>\nConstants<\/h3>\n
\"\"\"Defining common constant variables for training pipeline\"\"\"\nPIPELINE_NAME: str = \"AnimeRecommender\"\nARTIFACT_DIR: str = \"Artifacts\"\nANIME_FILE_NAME: str = \"Animes.csv\"\nRATING_FILE_NAME:str = \"UserRatings.csv\"\nMERGED_FILE_NAME:str = \"Anime_UserRatings.csv\" \nANIME_FILE_PATH:str = \"krishnaveni76\/Animes\"\nRATING_FILE_PATH:str = \"krishnaveni76\/UserRatings\"\nANIMEUSERRATINGS_FILE_PATH:str = \"krishnaveni76\/Anime_UserRatings\"\nMODELS_FILEPATH = \"krishnaveni76\/anime-recommendation-models\"\n\n\"\"\"Data Ingestion related constant start with DATA_INGESTION VAR NAME\"\"\" \nDATA_INGESTION_DIR_NAME: str = \"data_ingestion\"\nDATA_INGESTION_FEATURE_STORE_DIR: str = \"feature_store\"\nDATA_INGESTION_INGESTED_DIR: str = \"ingested\"\n\n\"\"\"Data Transformation related constant start with DATA_VALIDATION VAR NAME\"\"\"\nDATA_TRANSFORMATION_DIR:str = \"data_transformation\"\nDATA_TRANSFORMATION_TRANSFORMED_DATA_DIR:str = \"transformed\" \n\n\"\"\"Model Trainer related constant start with MODEL TRAINER VAR NAME\"\"\" \nMODEL_TRAINER_DIR_NAME: str = \"trained_models\"\n\nMODEL_TRAINER_COL_TRAINED_MODEL_DIR: str = \"collaborative_recommenders\"\nMODEL_TRAINER_SVD_TRAINED_MODEL_NAME: str = \"svd.pkl\"\nMODEL_TRAINER_ITEM_KNN_TRAINED_MODEL_NAME: str = \"itembasedknn.pkl\"\nMODEL_TRAINER_USER_KNN_TRAINED_MODEL_NAME: str = \"userbasedknn.pkl\"\n\nMODEL_TRAINER_CON_TRAINED_MODEL_DIR:str = \"content_based_recommenders\"\nMODEL_TRAINER_COSINESIMILARITY_MODEL_NAME:str = \"cosine_similarity.pkl\"<\/code><\/pre>\nUtils<\/h3>\n
def export_data_to_dataframe(dataframe: pd.DataFrame, file_path: str) -> pd.DataFrame: \n dir_path = os.path.dirname(file_path)\n os.makedirs(dir_path, exist_ok=True)\n dataframe.to_csv(file_path, index=False, header=True) \n return dataframe \n\ndef load_csv_data(file_path: str) -> pd.DataFrame: \n df = pd.read_csv(file_path) \n return df \n\ndef save_model(model: object, file_path: str) -> None: \n os.makedirs(os.path.dirname(file_path), exist_ok=True)\n with open(file_path, \"wb\") as file_obj:\n joblib.dump(model, file_obj) \n\ndef load_object(file_path: str) -> object: \n if not os.path.exists(file_path):\n error_msg = f\"The file: {file_path} does not exist.\" \n raise Exception(error_msg)\n with open(file_path, \"rb\") as file_obj: \n return joblib.load(file_obj) \n \ndef upload_model_to_huggingface(model_path: str, repo_id: str, filename: str): \n api = HfApi()\n api.upload_file(path_or_fileobj=model_path,path_in_repo=filename,=repo_id,repo_type=\"model\" ) <\/code><\/pre>\nConfiguration\u00a0Setup<\/h3>\n
class TrainingPipelineConfig:\n def __init__(self, timestamp=datetime.now()):\n timestamp = timestamp.strftime(\"%m_%d_%Y_%H_%M_%S\")\n self.pipeline_name = PIPELINE_NAME\n self.artifact_dir = os.path.join(ARTIFACT_DIR, timestamp)\n self.model_dir=os.path.join(\"final_model\")\n self.timestamp: str = timestamp\n\nclass DataIngestionConfig:\n def __init__(self, training_pipeline_config: TrainingPipelineConfig):\n self.data_ingestion_dir: str = os.path.join(training_pipeline_config.artifact_dir, DATA_INGESTION_DIR_NAME)\n self.feature_store_anime_file_path: str = os.path.join(self.data_ingestion_dir, DATA_INGESTION_FEATURE_STORE_DIR, ANIME_FILE_NAME) \n self.feature_store_userrating_file_path: str = os.path.join(self.data_ingestion_dir, DATA_INGESTION_FEATURE_STORE_DIR, RATING_FILE_NAME)\n self.anime_filepath: str = ANIME_FILE_PATH\n self.rating_filepath: str = RATING_FILE_PATH \n\nclass DataTransformationConfig: \n def __init__(self,training_pipeline_config:TrainingPipelineConfig): \n self.data_transformation_dir:str = os.path.join(training_pipeline_config.artifact_dir,DATA_TRANSFORMATION_DIR)\n self.merged_file_path:str = os.path.join(self.data_transformation_dir,DATA_TRANSFORMATION_TRANSFORMED_DATA_DIR,MERGED_FILE_NAME)\n\nclass CollaborativeModelConfig: \n def __init__(self,training_pipeline_config:TrainingPipelineConfig): \n self.model_trainer_dir:str = os.path.join(training_pipeline_config.artifact_dir,MODEL_TRAINER_DIR_NAME)\n self.svd_trained_model_file_path:str = os.path.join(self.model_trainer_dir,MODEL_TRAINER_COL_TRAINED_MODEL_DIR,MODEL_TRAINER_SVD_TRAINED_MODEL_NAME)\n self.user_knn_trained_model_file_path:str = os.path.join(self.model_trainer_dir,MODEL_TRAINER_COL_TRAINED_MODEL_DIR,MODEL_TRAINER_USER_KNN_TRAINED_MODEL_NAME)\n self.item_knn_trained_model_file_path:str = os.path.join(self.model_trainer_dir,MODEL_TRAINER_COL_TRAINED_MODEL_DIR,MODEL_TRAINER_ITEM_KNN_TRAINED_MODEL_NAME)\n\nclass ContentBasedModelConfig: \n def __init__(self,training_pipeline_config:TrainingPipelineConfig): \n self.model_trainer_dir:str = os.path.join(training_pipeline_config.artifact_dir,MODEL_TRAINER_DIR_NAME)\n self.cosine_similarity_model_file_path:str = os.path.join(self.model_trainer_dir,MODEL_TRAINER_CON_TRAINED_MODEL_DIR,MODEL_TRAINER_COSINESIMILARITY_MODEL_NAME)<\/code><\/pre>\nArtifacts entity<\/h3>\n
@dataclass\nclass DataIngestionArtifact: \n feature_store_anime_file_path:str\n feature_store_userrating_file_path:str\n\n@dataclass\nclass DataTransformationArtifact:\n merged_file_path:str\n\n@dataclass\nclass CollaborativeModelArtifact:\n svd_file_path:str\n item_based_knn_file_path:str\n user_based_knn_file_path:str\n \n@dataclass\nclass ContentBasedModelArtifact:\n cosine_similarity_model_file_path:str<\/code><\/pre>\nRecommendation System \u2013 Model Training<\/h2>\n
\n
1. Collaborative Recommendation System<\/h2>\n
![\"Collaborative](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-08_173153.webp\")
<\/figure>\nA. Data Ingestion<\/h3>\n
Data Ingestion Component<\/h4>\n
class DataIngestion: \n def __init__(self, data_ingestion_config: DataIngestionConfig): \n self.data_ingestion_config = data_ingestion_config \n\n def fetch_data_from_huggingface(self, dataset_path: str, split: str = None) -> pd.DataFrame: \n dataset = load_dataset(dataset_path, split=split)\n df = pd.DataFrame(dataset['train'])\n return df \n\n def ingest_data(self) -> DataIngestionArtifact: \n anime_df = self.fetch_data_from_huggingface(self.data_ingestion_config.anime_filepath)\n rating_df = self.fetch_data_from_huggingface(self.data_ingestion_config.rating_filepath)\n\n export_data_to_dataframe(anime_df, file_path=self.data_ingestion_config.feature_store_anime_file_path)\n export_data_to_dataframe(rating_df, file_path=self.data_ingestion_config.feature_store_userrating_file_path)\n\n dataingestionartifact = DataIngestionArtifact(\n feature_store_anime_file_path=self.data_ingestion_config.feature_store_anime_file_path,\n feature_store_userrating_file_path=self.data_ingestion_config.feature_store_userrating_file_path\n ) \n return dataingestionartifact <\/code><\/pre>\nB. Data\u00a0Transformation<\/h3>\n
Data Transformation Component<\/h4>\n
class DataTransformation: \n def __init__(self,data_ingestion_artifact:DataIngestionArtifact,data_transformation_config:DataTransformationConfig):\n self.data_ingestion_artifact = data_ingestion_artifact\n self.data_transformation_config = data_transformation_config \n \n @staticmethod\n def read_data(file_path)->pd.DataFrame: \n return pd.read_csv(file_path) \n \n @staticmethod\n def merge_data(anime_df: pd.DataFrame, rating_df: pd.DataFrame) -> pd.DataFrame: \n merged_df = pd.merge(rating_df, anime_df, on=\"anime_id\", how=\"inner\") \n return merged_df \n\n @staticmethod\n def clean_filter_data(merged_df: pd.DataFrame) -> pd.DataFrame: \n merged_df['average_rating'].replace('UNKNOWN', np.nan)\n merged_df['average_rating'] = pd.to_numeric(merged_df['average_rating'], errors=\"coerce\")\n merged_df['average_rating'].fillna(merged_df['average_rating'].median())\n merged_df = merged_df[merged_df['average_rating'] > 6]\n cols_to_drop = [ 'username', 'overview', 'type', 'episodes', 'producers', 'licensors', 'studios', 'source', 'rank', 'popularity', 'favorites', 'scored by', 'members' ]\n cleaned_df = merged_df.copy()\n cleaned_df.drop(columns=cols_to_drop, inplace=True) \n return cleaned_df \n \n def initiate_data_transformation(self)->DataTransformationArtifact: \n anime_df = DataTransformation.read_data(self.data_ingestion_artifact.feature_store_anime_file_path)\n rating_df = DataTransformation.read_data(self.data_ingestion_artifact.feature_store_userrating_file_path) \n merged_df = DataTransformation.merge_data(anime_df, rating_df)\n transformed_df = DataTransformation.clean_filter_data(merged_df)\n export_data_to_dataframe(transformed_df, self.data_transformation_config.merged_file_path)\n data_transformation_artifact = DataTransformationArtifact( merged_file_path=self.data_transformation_config.merged_file_path) \n return data_transformation_artifact <\/code><\/pre>\nC. Collaborative Recommender<\/h3>\n
Collaborative Modelling<\/h4>\n
\n
class CollaborativeAnimeRecommender: \n def __init__(self, df): \n self.df = df\n self.svd = None\n self.knn_item_based = None\n self.knn_user_based = None\n self.prepare_data() \n \n def prepare_data(self): \n self.df = self.df.drop_duplicates()\n reader = Reader(rating_scale=(1, 10))\n self.data = Dataset.load_from_df(self.df[['user_id', 'anime_id', 'rating']], reader)\n self.anime_pivot = self.df.pivot_table(index='name', columns=\"user_id\", values=\"rating\").fillna(0)\n self.user_pivot = self.df.pivot_table(index='user_id', columns=\"name\", values=\"rating\").fillna(0)\n \n def train_svd(self): \n self.svd = SVD()\n cross_validate(self.svd, self.data, cv=5)\n trainset = self.data.build_full_trainset()\n self.svd.fit(trainset) \n \n def train_knn_item_based(self): \n item_user_matrix = csr_matrix(self.anime_pivot.values)\n self.knn_item_based = NearestNeighbors(metric=\"cosine\", algorithm='brute')\n self.knn_item_based.fit(item_user_matrix) \n \n def train_knn_user_based(self): \n user_item_matrix = csr_matrix(self.user_pivot.values)\n self.knn_user_based = NearestNeighbors(metric=\"cosine\", algorithm='brute')\n self.knn_user_based.fit(user_item_matrix) \n \n def print_unique_user_ids(self): \n unique_user_ids = self.df['user_id'].unique() \n return unique_user_ids \n \n def get_svd_recommendations(self, user_id, n=10, svd_model=None)-> pd.DataFrame: \n svd_model = svd_model or self.svd\n if svd_model is None:\n raise ValueError(\"SVD model is not provided or trained.\") \n if user_id not in self.df['user_id'].unique():\n return f\"User ID '{user_id}' not found in the dataset.\" \n anime_ids = self.df['anime_id'].unique() \n predictions = [(anime_id, svd_model.predict(user_id, anime_id).est) for anime_id in anime_ids]\n predictions.sort(key=lambda x: x[1], reverse=True) \n recommended_anime_ids = [pred[0] for pred in predictions[:n]] \n recommended_anime = self.df[self.df['anime_id'].isin(recommended_anime_ids)].drop_duplicates(subset=\"anime_id\") \n recommended_anime = recommended_anime.head(n) \n return pd.DataFrame({ 'Anime Name': recommended_anime['name'].values, 'Genres': recommended_anime['genres'].values, 'Image URL': recommended_anime['image url'].values, 'Rating': recommended_anime['average_rating'].values})\n \n def get_item_based_recommendations(self, anime_name, n_recommendations=10, knn_item_model=None): \n knn_item_based = knn_item_model or self.knn_item_based\n if knn_item_based is None:\n raise ValueError(\"Item-based KNN model is not provided or trained.\") \n if anime_name not in self.anime_pivot.index:\n return f\"Anime title '{anime_name}' not found in the dataset.\" \n query_index = self.anime_pivot.index.get_loc(anime_name) \n distances, indices = knn_item_based.kneighbors( self.anime_pivot.iloc[query_index, :].values.reshape(1, -1), n_neighbors=n_recommendations + 1 ) \n recommendations = []\n for i in range(1, len(distances.flatten())): \n anime_title = self.anime_pivot.index[indices.flatten()[i]]\n distance = distances.flatten()[i]\n recommendations.append((anime_title, distance)) \n recommended_anime_titles = [rec[0] for rec in recommendations] \n filtered_df = self.df[self.df['name'].isin(recommended_anime_titles)].drop_duplicates(subset=\"name\") \n filtered_df = filtered_df.head(n_recommendations) \n return pd.DataFrame({ 'Anime Name': filtered_df['name'].values, 'Image URL': filtered_df['image url'].values, 'Genres': filtered_df['genres'].values, 'Rating': filtered_df['average_rating'].values }) \n \n def get_user_based_recommendations(self, user_id, n_recommendations=10, knn_user_model=None)-> pd.DataFrame: \n knn_user_based = knn_user_model or self.knn_user_based\n if knn_user_based is None:\n raise ValueError(\"User-based KNN model is not provided or trained.\") \n user_id = float(user_id) \n if user_id not in self.user_pivot.index:\n return f\"User ID '{user_id}' not found in the dataset.\" \n user_idx = self.user_pivot.index.get_loc(user_id) \n distances, indices = knn_user_based.kneighbors( self.user_pivot.iloc[user_idx, :].values.reshape(1, -1), n_neighbors=n_recommendations + 1 ) \n user_rated_anime = set(self.user_pivot.columns[self.user_pivot.iloc[user_idx, :] > 0]) \n all_neighbor_ratings = []\n for i in range(1, len(distances.flatten())): \n neighbor_idx = indices.flatten()[i]\n neighbor_rated_anime = self.user_pivot.iloc[neighbor_idx, :]\n neighbor_ratings = neighbor_rated_anime[neighbor_rated_anime > 0]\n all_neighbor_ratings.extend(neighbor_ratings.index) \n anime_counter = Counter(all_neighbor_ratings) \n recommendations = [(anime, count) for anime, count in anime_counter.items() if anime not in user_rated_anime]\n recommendations.sort(key=lambda x: x[1], reverse=True) \n recommended_anime_titles = [rec[0] for rec in recommendations[:n_recommendations]]\n filtered_df = self.df[self.df['name'].isin(recommended_anime_titles)].drop_duplicates(subset=\"name\") \n filtered_df = filtered_df.head(n_recommendations) \n return pd.DataFrame({ 'Anime Name': filtered_df['name'].values, 'Image URL': filtered_df['image url'].values, 'Genres': filtered_df['genres'].values, 'Rating': filtered_df['average_rating'].values }) <\/code><\/pre>\nCollaborative\u00a0Model Trainer Component<\/h4>\n
class CollaborativeModelTrainer: \n def __init__(self, collaborative_model_trainer_config: CollaborativeModelConfig, data_transformation_artifact: DataTransformationArtifact):\n self.collaborative_model_trainer_config = collaborative_model_trainer_config\n self.data_transformation_artifact = data_transformation_artifact \n\n def initiate_model_trainer(self) -> CollaborativeModelArtifact: \n df = load_csv_data(self.data_transformation_artifact.merged_file_path)\n recommender = CollaborativeAnimeRecommender(df) \n # Train and save SVD model \n recommender.train_svd()\n save_model(model=recommender.svd,file_path= self.collaborative_model_trainer_config.svd_trained_model_file_path)\n upload_model_to_huggingface(\n model_path=self.collaborative_model_trainer_config.svd_trained_model_file_path,\n repo_id=MODELS_FILEPATH,\n filename=MODEL_TRAINER_SVD_TRAINED_MODEL_NAME\n ) \n svd_model = load_object(self.collaborative_model_trainer_config.svd_trained_model_file_path) \n svd_recommendations = recommender.get_svd_recommendations(user_id=436, n=10, svd_model=svd_model) \n\n # Train and save Item-Based KNN model \n recommender.train_knn_item_based()\n save_model(model=recommender.knn_item_based, file_path=self.collaborative_model_trainer_config.item_knn_trained_model_file_path)\n upload_model_to_huggingface(\n model_path=self.collaborative_model_trainer_config.item_knn_trained_model_file_path,\n repo_id=MODELS_FILEPATH,\n filename=MODEL_TRAINER_ITEM_KNN_TRAINED_MODEL_NAME\n ) \n item_knn_model = load_object(self.collaborative_model_trainer_config.item_knn_trained_model_file_path)\n item_based_recommendations = recommender.get_item_based_recommendations(\n anime_name=\"One Piece\", n_recommendations=10, knn_item_model=item_knn_model\n ) \n\n # Train and save User-Based KNN model \n recommender.train_knn_user_based()\n save_model(model=recommender.knn_user_based,file_path= self.collaborative_model_trainer_config.user_knn_trained_model_file_path)\n upload_model_to_huggingface(\n model_path=self.collaborative_model_trainer_config.user_knn_trained_model_file_path,\n repo_id=MODELS_FILEPATH,\n filename=MODEL_TRAINER_USER_KNN_TRAINED_MODEL_NAME\n ) \n user_knn_model = load_object(self.collaborative_model_trainer_config.user_knn_trained_model_file_path)\n user_based_recommendations = recommender.get_user_based_recommendations(\n user_id=817, n_recommendations=10, knn_user_model=user_knn_model\n ) \n return CollaborativeModelArtifact(\n svd_file_path=self.collaborative_model_trainer_config.svd_trained_model_file_path,\n item_based_knn_file_path=self.collaborative_model_trainer_config.item_knn_trained_model_file_path,\n user_based_knn_file_path=self.collaborative_model_trainer_config.user_knn_trained_model_file_path\n ) <\/code><\/pre>\n2. Content-Based Recommendation System<\/h2>\n
![\"Content-Based](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-08_193215.webp\")
<\/figure>\nA. Data Ingestion<\/h3>\n
B. Content-Based Recommender<\/h3>\n
Content-Based Modelling<\/h4>\n
class ContentBasedRecommender: \n def __init__(self, df): \n self.df = df.dropna() \n self.indices = pd.Series(self.df.index, index=self.df['name']).drop_duplicates() \n self.tfv = TfidfVectorizer( min_df=3, strip_accents=\"unicode\", analyzer=\"word\", token_pattern=r'\\w{1,}', ngram_range=(1, 3), stop_words=\"english\" )\n self.tfv_matrix = self.tfv.fit_transform(self.df['genres']) \n self.cosine_sim = cosine_similarity(self.tfv_matrix, self.tfv_matrix) \n \n def save_model(self, model_path): \n os.makedirs(os.path.dirname(model_path), exist_ok=True)\n with open(model_path, 'wb') as f:\n joblib.dump((self.tfv, self.cosine_sim), f) \n \n def get_rec_cosine(self, title, model_path, n_recommendations=5): \n with open(model_path, 'rb') as f:\n self.tfv, self.cosine_sim = joblib.load(f) \n if self.df is None: \n raise ValueError(\"The DataFrame is not loaded, cannot make recommendations.\") \n if title not in self.indices.index: \n return f\"Anime title '{title}' not found in the dataset.\" \n idx = self.indicesHow to Build an Anime Recommendation System?\n cosinesim_scores = list(enumerate(self.cosine_sim[idx]))\n cosinesim_scores = sorted(cosinesim_scores, key=lambda x: x[1], reverse=True)[1:n_recommendations + 1]\n anime_indices = [i[0] for i in cosinesim_scores] \n return pd.DataFrame({ 'Anime name': self.df['name'].iloc[anime_indices].values, 'Image URL': self.df['image url'].iloc[anime_indices].values, 'Genres': self.df['genres'].iloc[anime_indices].values, 'Rating': self.df['average_rating'].iloc[anime_indices].values }) <\/code><\/pre>\nContent-Based Model Trainer Component<\/h4>\n
class ContentBasedModelTrainer: \n def __init__(self, content_based_model_trainer_config: ContentBasedModelConfig, data_ingestion_artifact: DataIngestionArtifact):\n self.content_based_model_trainer_config = content_based_model_trainer_config\n self.data_ingestion_artifact = data_ingestion_artifact \n\n def initiate_model_trainer(self) -> ContentBasedModelArtifact: \n df = load_csv_data(self.data_ingestion_artifact.feature_store_anime_file_path) \n recommender = ContentBasedRecommender(df=df ) \n recommender.save_model(model_path=self.content_based_model_trainer_config.cosine_similarity_model_file_path)\n upload_model_to_huggingface(\n model_path=self.content_based_model_trainer_config.cosine_similarity_model_file_path,\n repo_id=MODELS_FILEPATH,\n filename=MODEL_TRAINER_COSINESIMILARITY_MODEL_NAME\n ) \n cosine_recommendations = recommender.get_rec_cosine(title=\"One Piece\", model_path=self.content_based_model_trainer_config.cosine_similarity_model_file_path, n_recommendations=10) \n content_model_trainer_artifact = ContentBasedModelArtifact( cosine_similarity_model_file_path=self.content_based_model_trainer_config.cosine_similarity_model_file_path )\n return content_model_trainer_artifact<\/code><\/pre>\n3. Top Anime Recommendation System<\/h2>\n
![\"Top](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-08_195018.webp\")
<\/figure>\nA. Data Ingestion<\/h3>\n
B. Top Anime Recommender Component<\/h3>\n
Top anime filtering<\/h4>\n
class PopularityBasedFiltering: \n def __init__(self, df): \n self.df = df\n self.df['average_rating'] = pd.to_numeric(self.df['average_rating'], errors=\"coerce\")\n self.df['average_rating'].fillna(self.df['average_rating'].median()) \n \n def popular_animes(self, n=10): \n sorted_df = self.df.sort_values(by=['popularity'], ascending=True)\n top_n_anime = sorted_df.head(n)\n return self._format_output(top_n_anime)\n \n def top_ranked_animes(self, n=10): \n self.df['rank'] = self.df['rank'].replace('UNKNOWN', np.nan).astype(float)\n df_filtered = self.df[self.df['rank'] > 1]\n sorted_df = df_filtered.sort_values(by=['rank'], ascending=True)\n top_n_anime = sorted_df.head(n)\n return self._format_output(top_n_anime)\n \n def overall_top_rated_animes(self, n=10): \n sorted_df = self.df.sort_values(by=['average_rating'], ascending=False)\n top_n_anime = sorted_df.head(n)\n return self._format_output(top_n_anime)\n \n def favorite_animes(self, n=10): \n sorted_df = self.df.sort_values(by=['favorites'], ascending=False)\n top_n_anime = sorted_df.head(n)\n return self._format_output(top_n_anime)\n \n def top_animes_members(self, n=10): \n sorted_df = self.df.sort_values(by=['members'], ascending=False)\n top_n_anime = sorted_df.head(n)\n return self._format_output(top_n_anime)\n \n def popular_anime_among_members(self, n=10): \n sorted_df = self.df.sort_values(by=['members', 'average_rating'], ascending=[False, False]).drop_duplicates(subset=\"name\") \n popular_animes = sorted_df.head(n)\n return self._format_output(popular_animes)\n \n def top_avg_rated(self, n=10): \n self.df['average_rating'] = pd.to_numeric(self.df['average_rating'], errors=\"coerce\")\n median_rating = self.df['average_rating'].median()\n self.df['average_rating'].fillna(median_rating)\n top_animes = ( self.df.drop_duplicates(subset=\"name\").nlargest(n, 'average_rating')[['name', 'average_rating', 'image url', 'genres']] )\n return self._format_output(top_animes)\n \n def _format_output(self, anime_df): \n return pd.DataFrame({ 'Anime name': anime_df['name'].values, 'Image URL': anime_df['image url'].values, 'Genres': anime_df['genres'].values, 'Rating': anime_df['average_rating'].values })<\/code><\/pre>\nTop anime recommenders<\/h4>\n
class PopularityBasedRecommendor: \n def __init__(self,data_ingestion_artifact = DataIngestionArtifact): \n self.data_ingestion_artifact = data_ingestion_artifact \n \n def initiate_model_trainer(self,filter_type:str): \n df = load_csv_data(self.data_ingestion_artifact.feature_store_anime_file_path) \n recommender = PopularityBasedFiltering(df) \n if filter_type == 'popular_animes': \n popular_animes = recommender.popular_animes(n =10) \n elif filter_type == 'top_ranked_animes': \n top_ranked_animes = recommender.top_ranked_animes(n =10) \n elif filter_type == 'overall_top_rated_animes': \n overall_top_rated_animes = recommender.overall_top_rated_animes(n =10) \n elif filter_type == 'favorite_animes': \n favorite_animes = recommender.favorite_animes(n =10) \n elif filter_type == 'top_animes_members': \n top_animes_members = recommender.top_animes_members(n = 10) \n elif filter_type == 'popular_anime_among_members': \n popular_anime_among_members = recommender.popular_anime_among_members(n =10) \n elif filter_type == 'top_avg_rated': \n top_avg_rated = recommender.top_avg_rated(n =10) <\/code><\/pre>\nTraining Pipeline<\/h2>\n
![\"Training](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-07_185707.webp\")
<\/figure>\nclass TrainingPipeline: \n def __init__(self): \n self.training_pipeline_config = TrainingPipelineConfig()\n\n def start_data_ingestion(self) -> DataIngestionArtifact: \n data_ingestion_config = DataIngestionConfig(self.training_pipeline_config)\n data_ingestion = DataIngestion(data_ingestion_config=data_ingestion_config)\n data_ingestion_artifact = data_ingestion.ingest_data() \n return data_ingestion_artifact \n\n def start_data_transformation(self, data_ingestion_artifact: DataIngestionArtifact) -> DataTransformationArtifact: \n data_transformation_config = DataTransformationConfig(self.training_pipeline_config)\n data_transformation = DataTransformation(\n data_ingestion_artifact=data_ingestion_artifact,\n data_transformation_config=data_transformation_config\n )\n data_transformation_artifact = data_transformation.initiate_data_transformation() \n return data_transformation_artifact \n\n def start_collaborative_model_training(self, data_transformation_artifact: DataTransformationArtifact) -> CollaborativeModelArtifact: \n collaborative_model_config = CollaborativeModelConfig(self.training_pipeline_config)\n collaborative_model_trainer = CollaborativeModelTrainer(\n collaborative_model_trainer_config=collaborative_model_config,\n data_transformation_artifact=data_transformation_artifact )\n collaborative_model_trainer_artifact = collaborative_model_trainer.initiate_model_trainer() \n return collaborative_model_trainer_artifact \n\n def start_content_based_model_training(self, data_ingestion_artifact: DataIngestionArtifact) -> ContentBasedModelArtifact: \n content_based_model_config = ContentBasedModelConfig(self.training_pipeline_config)\n content_based_model_trainer = ContentBasedModelTrainer(\n content_based_model_trainer_config=content_based_model_config,\n data_ingestion_artifact=data_ingestion_artifact )\n content_based_model_trainer_artifact = content_based_model_trainer.initiate_model_trainer() \n return content_based_model_trainer_artifact \n\n def start_popularity_based_filtering(self, data_ingestion_artifact: DataIngestionArtifact): \n filtering = PopularityBasedRecommendor(data_ingestion_artifact=data_ingestion_artifact)\n recommendations = filtering.initiate_model_trainer(filter_type=\"popular_animes\") \n return recommendations \n\n def run_pipeline(self): \n # Data Ingestion\n data_ingestion_artifact = self.start_data_ingestion() \n # Content-Based Model Training\n content_based_model_trainer_artifact = self.start_content_based_model_training(data_ingestion_artifact) \n # Popularity-Based Filtering\n popularity_recommendations = self.start_popularity_based_filtering(data_ingestion_artifact) \n # Data Transformation\n data_transformation_artifact = self.start_data_transformation(data_ingestion_artifact) \n # Collaborative Model Training\n collaborative_model_trainer_artifact = self.start_collaborative_model_training(data_transformation_artifact)<\/code><\/pre>\npython training_pipeline.py <\/code><\/pre>\nStreamlit App<\/h2>\n
![\"streamlit\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Screenshot_2025-02-11_132016-thumbnail_webp-600x300-1.webp\")
<\/figure>\nDocker Integration for Deployment<\/h2>\n
# Use the official Python image as a base\nFROM python:3.10-slim-buster\n\n# Set the working directory in the container\nWORKDIR \/app\n\n# Copy the app files into the container\nCOPY . .\n\n# Install required packages\nRUN pip install -r requirements.txt\n\n# Expose the port that Streamlit uses\nEXPOSE 8501\n\n# Run the Streamlit app\nCMD [\"streamlit\", \"run\", \"app.py\", \"--server.port=8501\", \"--server.address=0.0.0.0\"] <\/code><\/pre>\nKey Takeaways<\/h2>\n
\n
Conclusion<\/h2>\n
Frequently Asked Questions<\/h2>\n
![\"Krishnaveni](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_wE6UHKH.webp\")
<\/p>\n