PyTorch Deep Learning Framework: Convenience + Practicality

2019-12-17

Deep learning has almost reached human-level performance in areas such as speech and text recognition, image description, and complex gaming. However, deep learning is computationally intensive, and there have been some trade-offs in the development of deep learning frameworks: improving usability often has a negative impact on speed, and vice versa.


Popular frameworks like Caffe, TensorFlow, and Theano offer fast computational performance, but they come at the cost of simplicity and flexibility.


Then there's PyTorch, primarily developed by Facebook AI and released in 2016. PyTorch is an efficient library for facilitating deep learning projects. It allows deep learning models to be represented in the familiar Python programming language, which is a huge advantage in terms of usability.PyTorch is designed based on four main principles. Firstly, considering that most data scientists are familiar with Python, PyTorch stays within this ecosystem to maintain a simple and consistent interface. Secondly, it hides many of the inherent complexities of machine learning behind an intuitive API. Thirdly, it allows researchers to manually control code execution for practical performance. Lastly, in contrast to more comprehensive and complex frameworks, PyTorch's implementation is designed to be simple yet somewhat "incomplete." This makes it easier to implement additional functionalities and adapt to new situations in PyTorch, thereby increasing flexibility.


PyTorch adopts a graph-based approach for defining the composition of layers in a model, loading data, and running optimizer execution code, all represented by generic programming. This design ensures that any new neural network architecture can be easily generalized and implemented using PyTorch.


Interoperability and scalability are other key objectives of PyTorch, allowing for bidirectional data exchange with external libraries. Users are free to interchange PyTorch components to better meet their specific project requirements.PyTorch's outstanding performance is largely attributed to the following five strategies:




  1. PyTorch Core is used to implement tensor data structures, CPU and GPU operators, basic parallel primitives, and automatic differentiation computations. Since the most compute-intensive operations are handled by the core, they can be written in efficient C++ programming language to improve performance.

  2. Strict separation between control and data flow. Although control flow is done in Python, optimized C++ code can be executed on the host CPU.

  3. Custom allocators incrementally build a cache of CUDA memory and reallocate it for subsequent allocations, preventing further use of the CUDA API.

  4. Torch.multiprocessing allows users to achieve highly parallel programs across multiple GPUs.

  5. A reference counting scheme monitors the usage count of each tensor and immediately releases underlying memory once the count reaches zero.


Compared to mainstream deep learning frameworks like Chainer, CNTK, MXNet, PaddlePaddle, and TensorFlow, PyTorch scores the fastest framework in all six benchmark tests, with scores within 17%.

RECENT AI NEWS

RECENT AI TOOLS