| Sumario: | In a big data project, a plethora of information is retrieved, big numbers are crunched on our machine, or both. If the coding is sequential or synchronous, our application will struggle to execute. Two mechanisms to alleviate such bottlenecks are concurrency and parallelism. In Python, concurrency is represented by threading, whereas multiprocessing achieves parallelism. This course begins with an introduction about potential programming speed bottlenecks and solving them. You will delve into Python concepts and create a Wikipedia Reader, Yahoo Finance Reader, Queues, and Master Scheduler. You will build a multi-threaded program to grab data from the Internet and parse and save them into a local database. Implement multiprocessing in Python, which lets us use multiple CPUs in our code. Learn about threading, multiprocessing, asynchronous wait, locking, multiprocessing queues, Pool Map Multiple Arguments, writing asynchronous programs, and combining async and multiprocessing. Upon completion, we can spread our workload over all cores available on the used machine. We will combine both elements, multiprocessing with asynchronous programming, to maximize benefit and CPU resource usage and minimize the time spent waiting for IO responses. You will create multi-threaded, asynchronous, multi-process programs to make programs run faster. What You Will Learn Learn to use concurrency and parallelism in Python Write multi-threaded programs in Python to reduce coding lengths Write multi-process programs that execute even faster Understand the differences between concurrency and parallelism Create asynchronous programs in Python by adding concurrency Spread workload over all the cores available on a machine being used Audience This course is aimed at intermediate- to mastery-level seeking programmers, API developers, web developers, and application developers who know basic- to intermediate-level Python coding beforehand. The topics on concurrency and parallelism expect one to be aware of basic to intermediate understanding of coding on Python. Prior knowledge of basic Python coding is desirable for optimal benefit from this course. About The Author Maximilian Schallwig: Maximilian Schallwig is a data engineer and a proficient Python programmer. He holds a bachelor's degree in physics and a master's degree in astrophysics. He has been working on data for over five years, first as a data scientist and then as a data engineer. He can talk endlessly about big data pipelines, data infrastructure, and his unwavering devotion to Python. Even after two unsuccessful attempts in high school, he still decided to learn Python at the University. He cautiously stepped into the realm of data, beginning with a simple Google search for "what does a data scientist do" He was determined to pursue a career in data science to become a data engineer by learning about big data tools and infrastructure design to build scalable systems and pipelines. He enjoys sharing his programming skills with the rest of the world.
|
|---|
