Machine Learning Practice

Some practices using statistical machine learning technique based on some dataset.

To see more detail or example about deep learning, you can checkout my Deep Learning repository.

Environment

• Using Python 3

(most of the relative path links are according to the repository root)

Dependencies

• numpy: For low-level math operations
• pandas: For data manipulation
• sklearn - Scikit Learn: For evaluation metrics, some data preprocessing

For comparison purpose

• sklearn: For machine learning models
• cvxopt: For convex optimization problem (for SVM)

NLP related

• gensim: Topic Modelling
• hmmlearn: Hidden Markov Models in Python, with scikit-learn like API
• jieba: Chinese text segementation library
• pyHanLP: Chinese NLP library (Python API)
• nltk: Natural Language Toolkit

Projects

Machine Learning Categories

Consider the learning task

• Surpervised Learning
  • Classification - Discrete
  • Regression - Continuous
• Unsupervised Learning
  • Clustering - Discrete
  • Dimensionality Reduction - Continuous
  • Association Rule Learning
• Semi-supervised Learning
• Reinforcement Learning

Cosider the desired output of a ML system

• Classification
  • Logistic Regression (optimization algo.)
  • k-Nearest Neighbors (kNN)
  • Support Vector Machine (SVM) - Deduction (optimization algo.)
  • Naive Bayes
  • Decision Tree (ID3, C4.5, CART)
• Regression
  • Linear Regression (optimization algo.)
  • Tree (CART)
• Clustering
  • k-Means
  • Hierarchical Clustering
• Association Rule Learning
  • Apriori
  • Eclat
  • FP-growth - Frequent itemsets mining
• Dimensionality Reduction
  • Principal Compnent Analysis (PCA)
  • Single Value Decomposition (SVD) - LSA, LSI, Recommendation System
  • ISOMAP

Ensemble Method (Meta-algorithm)

• Bagging
  • Random Forests
• Boosting
  • AdaBoost <- With some basic boosting notes
  • Gradient Boosting
    • Gradient Boosting Decision Tree (GBDT) (aka. Multiple Additive Regression Tree (MART))
  • XGBoost

Others

• Hidden Markov Model (HMM)
• Bayesian Network (aka. Probabilistic Directed Acyclic Graphical Model)
• Conditional Random Field (CRF)
• Probabilistic Latent Semantic Analysis (PLSA)
• Latent Dirichlet Allocation (LDA)
• Vector Space Model (VSM)

Heuristic Algorithm

• SMO --> SVM
• EM --> HMM

Machine Learning Concepts

General Case

• Data Preprocessing
  • Normalization
  • Training and Test Sets - Splitting Data
  • Missing Value
  • Dimensionality Reduction
• Model Expansion
  • Binary to Multi-class
• Fitting and Model Complexity
  • Overfitting
  • Underfitting
  • Generalization
  • Regularization
• Reducing Loss
  • Learning Rate
  • Gradient Descent
• Other Learning Method
  • Cost-sensitive Learning
  • Lazy Learning
  • Incremental Learning (Online Learning)
  • Multi-label Classification

Categorized

• Classification
  • Data Preprocessing
    • Label Encoding
  • Real-world Problem
    • Cost-sensitive Learning
    • Classification Imbalance
  • Evaluation Metrics
    • Classification Metrics
  • Binary to Multi-class Expension
• Regression
  • Evaluation Metrics
    • Regression Metrics
• Clustering
  • Evaluation Metrics
    • Clustering Metrics

Specific Field

• Data Mining - Knowledge Discovering
• Recommendation System
  • Collaborative Filtering
• Information Retrieval - Topic Modelling
  • Latent Semantic Analysis (LSA/LSI/SVD)
  • Latent Dirichlet Allocation (LDA)
  • Random Projections (RP)
  • Hierarchical Dirichlet Process (HDP)
  • word2vec

Machine Learning Mathematics

Topic

• Kernel Usages
• Convex Optimization

Categories

• Linear Algebra
  • Orthogonality
  • Eigenvalues
  • Hessian Matrix
  • Quadratic Form
  • Markov Chain - HMM
• Calculus
  • Multivariable Deratives
    • Quadratic Approximations
    • Lagrange Multipliers and Constrained Optimization - SVM SMO
    • Lagrange Duality
• Probability and Statistics
  • Statistical Estimation
    • Maximum Likelihood Estimation (MLE)

Basics

• Algebra
• Trigonometry

Application

(from A to Z)

• Decision Tree
  • Entropy
• HMM
  • Markov Chain
• Naive Bayes
  • Bayes' Theorem
• PCA
  • Orthogonal Transformations
  • Eigenvalues
• SVD
  • Eigenvalues
• SVM
  • Convex Optimization
  • Constrained Optimization
  • Lagrange Multipliers
  • Kernel

Books Recommendation

Machine Learning

• Machine Learning in Action
  • Source Code
• 統計學習方法 (李航)
• 機器學習 (周志華)

Mathematics

• Linear Algebra with Applications (Steven Leon)
• Convex Optimization (Stephen Boyd & Lieven Vandenberghe)
• Numerical Linear Algebra (L. Trefethen & D. Bau III)

Resources

Tutorial

Videos

• Google - Machine Learning Recipes with Josh Gordon
  • Josh Gordon's Repository
• Youtube - Machine Learning Fun and Easy
• Siraj Raval - The Math of Intelligence
  • Siraj Raval's Repository
• bilibili - 機器學習 - 白板推導系列
• bilibili - 機器學習升級版

Documentations

• ApacheCN (ML, DL, NLP)
  • Github - AiLearning
  • Official Website - ApacheCN 中文開源組織
• Machine learning 101 (infographics)
  • Machine learning overview (infographic)
  • Machine learning methods (infographic)
  • Machine learning evolution (infographic)

Interactive Learning

• Google Machine Learning Crash Course
• Kaggle Learn Machine Learning
• Microsoft Professional Program - Artificial Intelligence track

MOOC

• Stanford Andrew Ng - CS229
  • Coursera

Github

• Machine Learning from Scratch (eriklindernoren/ML-From-Scratch)
• Machine learning Resources

Textbook Implementation

• Machine Learning in Action
  • Jack-Cherish/Machine-Learning
• Learning From Data
  • Doraemonzzz/Learning-from-data
• 統計學習方法 (李航)
  • WenDesi/lihang_book_algorithm
    • blog
  • fengdu78/lihang-code
  • wzyonggege/statistical-learning-method
  • Dod-o/Statistical-Learning-Method_Code
    • blog
• Stanford Andrew Ng CS229
  • fengdu78/Coursera-ML-AndrewNg-Notes
  • fengdu78/deeplearning_ai_books

Datasets

• UCI Machine Learning Repository
• Awesome Public Datasets
• Kaggle Datasets
• The MNIST Database of handwritten digits
• 資料集平台 Data Market
• AI Challenger Datasets
• Peking University Open Research Data
• Open Images Dataset
  • github
• Alibaba Cloud Tianchi Data Lab

Machine Learning Platform

• Kaggle
• OpenML

Machine Learning Tool

• AutoML
  • Auto-sklearn
    • github

(Online) Development Environment

• Google Colaboratory
• Kaggle Kernels
• Intel AI DevCloud

jupyter notebook

• Extension plugin - pip install jupyter_contrib_nbextensions
  • VIM binding
  • Codefolding
  • ExecuteTime
  • Notify
• Jupyter Theme - pip install --upgrade jupyterthemes