Scaling Python
Build apps that scale to billions
“Our app now handles thousands of users without a sweat. Thanks Julien!” – Igor Subira
Why Python?
Python is an easy to learn programming language. Yet, it is powerful, versatile and comes with a luxuriant ecosystem. That makes it a very productive programming language for beginners and experts.
Building substantial applications with Python is totally possible. Like many other technologies, mastering Python requires know-how, techniques, trade-offs to be made, and traps to avoid.
It would be awful to end up like this, right?
“Python is fast enough for our site and allows us to produce maintainable features in record times, with a minimum of developers.”
Cuong Do
Software Architect at YouTube
Python is already trusted by thousands of companies using it a very large scale.
How can this book help me?
Build for Resiliency
Today's applications run on large and distributed infrastructures. They need to handle failure without blinking. In an environment where anything could go wrong, how do you design your code to handle those failures?
Remove Bottlenecks
Learn how to plan an architecture that has no bottleneck. Get straight answers to your conception questions and solutions that work and solve concrete cases. Discover distributed systems that scale properly.
Performance
We'll talk performance: how to write faster Python code, profile it, and optimize it. Read about concrete use-cases on how to enhance slow applications, transforming them into modern applications.
Batteries Included
Use the included Docker image to replay all the examples presented in the book. You get a complete Python environment that you need in order to execute the examples. Use the notebooks to run them in a click from your Web browser.
When you build a scalable, performing and distributed system, there are hundreds of problems to solve
This book offers shortcuts.
Here's what it includes:
What's scaling?
Scaling across CPUs
Distributed Systems
Service-Oriented Architecture
CPU scaling
Using Threads
Using Processes
Using Futures
Advanced Futures Usage
Daemon Processes
Event Loops
Basic Pattern
Using Asyncio
Network Server
Queue-Based Distribution
RQ
Celery
Handling failure
Chaining Tasks
Multiple Queues
Monitoring
Designing for Failure
Naive Retrying
Retrying with Tenacity
Lock Management
Thread Locks
Processes Locks
Multiprocessing Locks
Inter-Process Locks
Using etcd for Distributed Locking
Using Tooz Locking Abstraction
Group Membership
Creating, Joining and Leaving Groups
Using Capabilities
Using Watchers Callbacks
Consistent Hash Rings
Partitioner
Building REST API
The WSGI Protocol
Streaming Data
Using ETag
Asynchronous HTTP API
Fast HTTP Client
Testing REST API
Deploying on PaaS
Heroku
Amazon Beanstalk
Google App Engine
OpenShift
Beyond PaaS
Testing Distributed System
Setting Up Environments with tox
Manage External Services with pifpaf
Using Fixtures with pifpaf
Functional Programming
The Functional Toolkit
Caching
Local Caching
Memoization
Distributed Caching
Performance
Memory and CPU Profiling
Profiling Strategy and a Case
Zero-Copy
Disassembling Code
Learn Straight From the Experts
Scaling Python includes seven interviews from experienced Python developers. They share their know-how and wisdom on the various themes covered in the book.
Mehdi Abaakouk
Mehdi is a free software hacker, working as a Senior Software Engineer at Red Hat. He contributes to OpenStack, the largest open-source project using Python, where he builds a fault-tolerant, scalable and distributed cloud platform.
Chris Dent
Chris Dent is a Staff Engineer at VMware, a member of the OpenStack Technical Committee, and an HTTP connoisseur. Chris has among other things created Gabbi, a fabulous Python testing tool for HTTP, and is responsible for HTTP REST API testing.
Joshua Harlow
Joshua is a highly experienced engineer in distributed systems, working at GoDaddy. Josh is very knowledgeable on principles that make Python application resilient and faults tolerant as he wrote and maintained several of those over the years.
Naoki Inada
Naoki is a Researcher and Architect at KLab. He also happens to be one of the CPython developers. He worked on several significant features in CPython that are used by thousands, such as the asyncio event loop.
Victor Stinner
Victor is a long time CPython developer, working on the language itself for several years now. He is well known in the community for working on making CPython faster and leads several performance-oriented projects. He likes to talk about optimizations, profiling, and performance when using Python, and how to make the right decisions.
Jason Myers
Jason is a Python developer, working as a Software Engineer at Juice Analytics. He wrote an entire book on SQLAlchemy, the well-known Python SQL library (Essential SQLAlchemy (O'Reilly)) He worked on cloud computing platforms, as a Web developer, and as a data engineer.
Alexys Jacob-Monier
Alexys is the CTO of 1000mercis and is part of the open-source software community for a few years now. He regularly gives speeches at Python conferences (PyCon) and talks about how to leverage Python when distributing applications. He writes, maintains and operates highly distributed Python applications.
"A Must Read"
The readers of Scaling Python talk about it better than anyone else.