9. Model Deployment

In earlier chapters, we discussed the basic components of the machine learning model training system. In this chapter, we look at the basics of model deployment, a process whereby a trained model is deployed in a runtime environment for inference. We explore the conversion from a training model into an inference model, model compression methods that adapt to hardware restrictions, model inference and performance optimization, and model security protection.

The key aspects this chapter explores are as follows:

1. Conversion and optimization from a training model to an inference model.

2. Common methods for model compression: quantization, sparsification, and knowledge distillation.

3. Model inference process and common methods for performance optimization.

4. Common methods for model security protection.

• 9.1. Overview
• 9.2. Conversion to Inference Model and Model Optimization
  • 9.2.1. Model Conversion
  • 9.2.2. Operator Fusion
  • 9.2.3. Operator Replacement
  • 9.2.4. Operator Reordering
• 9.3. Model Compression
  • 9.3.1. Quantization
  • 9.3.2. Model Sparsification
  • 9.3.3. Knowledge Distillation
• 9.4. Advanced Efficient Techniques
  • 9.4.1. Speculative Decoding
  • 9.4.2. FlashAttention
• 9.5. Model Inference
  • 9.5.1. Preprocessing and Postprocessing
  • 9.5.2. Parallel Computing
  • 9.5.3. Operator Optimization
• 9.6. Security Protection of Models
  • 9.6.1. Overview
  • 9.6.2. Model Obfuscation
• 9.7. Chapter Summary
• 9.8. Further Reading