1 线程基础
1.1 threading.Thread类
01.Thread类概述
a.基本概念
threading.Thread是Python中创建和管理线程的核心类，提供了面向对象的线程操作接口。该类封装了线程的创建、启动、等待和管理等基本功能，是Python多线程编程的基础。
b.Thread类的主要特点
a.面向对象设计
Thread类采用面向对象的设计模式，每个线程实例都是一个独立的对象，具有自己的属性和方法。这种设计使得线程管理更加直观和灵活，便于扩展和定制。
b.生命周期管理
Thread类提供了完整的线程生命周期管理功能，从线程的创建、启动、运行到结束的各个阶段都有相应的控制方法。开发者可以通过这些方法精确控制线程的执行流程。
c.丰富的配置选项
支持多种线程配置选项，包括线程名称、守护线程标记、目标函数和参数等。这些配置选项使得线程的创建更加灵活，可以适应不同的应用场景。
02.Thread类的核心属性
a.基础属性
a.name属性
线程名称标识，用于区分不同的线程实例。默认情况下，线程名称会自动生成，但开发者也可以自定义名称来提高代码的可读性和调试效率。
b.ident属性
线程标识符，是一个整数，在系统范围内唯一标识一个线程。该属性在线程启动后才会被设置，可用于线程的追踪和调试。
c.daemon属性
守护线程标记，决定线程是否为守护线程。守护线程会在主线程结束时自动退出，非守护线程则会阻止主线程结束。
b.状态相关属性
a.is_alive()方法
判断线程是否处于活动状态，返回布尔值。活动状态指线程已经启动但尚未结束，包括运行中、阻塞或等待等状态。
b.native_id属性
原生线程标识符，在Python 3.8及以上版本中可用，返回操作系统的原生线程ID。
03.Thread类的核心方法
a.线程创建方法
a.__init__构造方法
初始化线程对象，设置线程的基本属性如目标函数、参数、名称等。构造方法不会立即创建线程，只是准备线程的配置信息。
b.线程启动方法
a.start()方法
启动线程，调用目标函数开始执行。该方法会创建一个新的线程并开始运行，每个线程只能调用一次start()方法。
b.run()方法
线程的实际执行内容，默认会调用目标函数。开发者可以通过重写run()方法来自定义线程的执行逻辑。
c.线程等待方法
a.join()方法
等待线程执行结束，主线程会阻塞直到被等待的线程完成。该方法可以设置超时参数，避免无限等待。
1.2 创建线程的两种方式
01.函数式创建线程
a.方法概述
通过传递目标函数给Thread构造函数来创建线程，这是最直接和常用的线程创建方式。开发者将需要在新线程中执行的函数作为参数传递，线程启动后会自动调用该函数。
b.函数式创建的语法结构
a.基本语法格式
Thread(target=function_name, args=(arg1, arg2), kwargs={key: value})
这种语法结构清晰明了，target参数指定目标函数，args传递位置参数，kwargs传递关键字参数。
b.参数说明
target参数必须是一个可调用对象，可以是函数、方法或实现了__call__方法的类实例。args参数是一个元组，包含传递给目标函数的位置参数，注意单参数时需要加逗号。
c.函数式创建的使用场景
a.简单任务执行
适用于执行相对简单的、一次性的任务，不需要复杂的状态管理或生命周期控制的场景。这种方式代码简洁，易于理解和维护。
b.批量任务处理
当需要同时执行多个相似任务时，函数式创建方式可以通过循环批量创建线程，每个线程执行相同的函数但处理不同的数据。
02.面向对象式创建线程
a.方法概述
通过继承Thread类并重写run()方法来创建线程，这种方式更加面向对象，可以将线程的逻辑和数据封装在一个类中。这种方法提供了更好的代码组织和扩展性。
b.面向对象创建的语法结构
a.类定义结构
class CustomThread(Thread):
def __init__(self, **kwargs):
super().__init__(**kwargs)
# 初始化自定义属性
def run(self):
# 重写线程执行逻辑
这种结构遵循面向对象的设计原则，通过继承和多态实现线程的定制化。
b.构造函数重写
子类的构造函数必须调用父类的构造函数super().__init__()，以确保线程的初始化正确完成。在子类构造函数中可以添加自定义的属性初始化。
c.面向对象创建的使用场景
a.复杂线程逻辑
当线程的执行逻辑比较复杂，需要维护多个状态变量或提供多个辅助方法时，面向对象的方式更加适合。它可以将相关的数据和行为封装在一起。
b.可重用的线程类
当需要在多个地方使用相同类型的线程时，可以创建一个可重用的线程类。这提高了代码的复用性和一致性。
03.两种创建方式的对比分析
a.代码复杂度对比
a.函数式方式
代码结构简单直接，学习成本较低，适合初学者和简单的应用场景。不需要理解类的继承机制，只需要定义函数即可创建线程。
b.面向对象方式
需要理解类的继承和方法重写概念，代码结构相对复杂，但提供了更好的组织性和扩展性。适合需要复杂逻辑管理的应用场景。
b.性能与资源使用
a.资源消耗对比
两种方式在线程的资源消耗和执行性能上基本相同，因为它们最终都调用底层的线程创建机制。选择哪种方式主要取决于代码组织需求而不是性能考虑。
b.内存占用差异
面向对象方式可能会因为类的额外属性而占用稍多内存，但这种差异通常很小，不足以成为选择的决定因素。
c.维护与扩展性
a.函数式维护特点
代码分散在多个函数中，当线程逻辑变得复杂时，维护成本会增加。函数间的数据共享需要通过全局变量或参数传递实现。
b.面向对象维护特点
相关的数据和行为被封装在一个类中，便于维护和扩展。可以通过添加新的方法和属性来增强线程的功能，而不影响现有的代码结构。
1.3 线程的启动与等待
01.线程启动机制
a.start()方法详解
start()方法是Thread类的核心方法，用于启动一个新的线程并开始执行目标函数。调用该方法后，系统会分配必要的资源并调度线程的执行，每个线程只能调用一次start()方法。
b.start()方法的工作原理
a.底层执行机制
start()方法会调用操作系统提供的线程创建API，在底层创建一个新的执行流。一旦线程创建成功，它会在独立的执行上下文中运行，与主线程并发执行。
b.线程状态转换
调用start()后，线程从"新建"状态转换为"就绪"状态，等待操作系统的调度。当获得CPU时间片时，线程进入"运行"状态并开始执行run()方法。
c.启动时的注意事项
a.单次启动限制
每个线程实例只能调用一次start()方法，重复调用会抛出RuntimeError异常。这种设计保证了线程的生命周期管理的清晰性和可预测性。
b.启动时机选择
应该在所有必要的参数设置完成后再调用start()方法，一旦启动后，线程的某些属性如daemon状态将无法修改。
02.线程等待机制
a.join()方法详解
join()方法用于等待线程执行完成，调用该方法的线程会被阻塞，直到目标线程结束执行。这是实现线程同步的重要机制，可以确保主线程在所有子线程完成后再继续执行。
b.join()方法的使用方式
a.无限等待模式
thread.join()会无限期等待目标线程结束，这种模式适用于必须等待线程完成才能继续的场景。使用时需要注意避免死锁情况的发生。
b.超时等待模式
thread.join(timeout=N)可以设置最大等待时间，如果在指定时间内线程未结束，join()方法会返回，主线程可以继续执行。这种模式提供了更好的可控性。
c.join()方法的实现机制
a.阻塞原理
join()方法通过操作系统的线程同步机制实现阻塞等待，底层通常使用条件变量或信号量来协调线程间的执行顺序。
b.资源清理
当线程结束时，系统会自动清理线程相关资源，调用join()不仅可以等待线程完成，还可以确保资源得到正确释放。
03.线程生命周期管理
a.线程状态模型
a.五种基本状态
新建状态：线程对象已创建但尚未启动；就绪状态：线程已启动等待调度；运行状态：线程正在执行；阻塞状态：线程因等待资源而暂停；终止状态：线程执行完毕。
b.状态转换关系
线程状态之间的转换是有条件的，新建状态只能通过start()转换为就绪状态，终止状态是线程的最终状态，不可逆转。
b.生命周期监控
a.is_alive()方法
判断线程是否还处于活动状态，返回True表示线程已启动但尚未结束。这个方法在需要监控线程状态的场景中非常有用。
b.线程标识追踪
通过ident属性可以获取线程的唯一标识符，有助于在调试和日志记录中追踪特定线程的执行情况。
c.优雅的线程终止
a.自然结束机制
最好的线程终止方式是让run()方法自然执行完毕，这样可以确保所有的清理工作得到正确执行，资源得到适当释放。
b.强制终止风险
Python不提供直接强制终止线程的API，因为强制终止可能导致资源泄露、数据不一致等严重问题。应该通过标志位或其他协调机制实现优雅终止。
04.多线程协调策略
a.启动顺序控制
a.顺序启动模式
按照特定顺序启动多个线程，适用于有依赖关系的任务场景。通过循环或条件判断来控制线程的启动时机。
b.批量启动模式
同时启动多个线程执行并行任务，适用于可以完全并行处理的独立任务。这种方式可以最大化利用系统资源。
b.等待策略设计
a.全部等待策略
启动所有线程后等待所有线程完成，这是最常见的多线程使用模式。确保所有任务完成后才进行后续处理。
b.分组等待策略
将线程分成多个组，按组等待完成，适用于复杂的任务依赖关系。可以实现更细粒度的执行控制。
1.4 守护线程
01.守护线程概念
a.定义与特性
守护线程是一种特殊的线程，它的生命周期依赖于主线程或非守护线程。当所有非守护线程都结束时，守护线程会自动被终止，无论其是否执行完成。这种机制确保了程序能够正常退出而不会被无限阻塞。
b.守护线程的标识机制
通过Thread类的daemon属性或setDaemon()方法来设置线程的守护属性。一旦线程启动后，就无法再修改其守护状态，因此必须在调用start()方法之前设置daemon属性。
c.守护线程的创建时机
守护线程属性必须在线程启动前设置，这是一个强制性要求。在线程运行过程中修改daemon属性会引发RuntimeError异常，这种设计保证了线程状态的一致性和可预测性。
02.守护线程的工作原理
a.程序退出机制
Python程序在退出时会检查所有存活的线程，如果只剩下守护线程，程序会立即终止而不等待守护线程完成执行。这种机制避免了守护线程阻塞程序正常退出的问题。
b.资源清理考虑
由于守护线程可能被强制中断，因此不应该在守护线程中执行需要完整执行的重要任务，如文件写入、数据库事务等。这些操作应该放在非守护线程中确保完整性。
c.系统资源释放
当程序退出时，操作系统会自动回收所有线程占用的系统资源，包括内存、文件描述符等。但是，优雅的资源清理仍然需要开发者妥善处理。
03.守护线程的应用场景
a.后台监控任务
a.系统状态监控
守护线程非常适合用于监控程序运行状态、系统资源使用情况等后台任务。这些任务通常是持续性的，不需要在程序退出时完成。
b.性能数据收集
实时收集应用程序的性能指标、运行统计数据等信息，为性能分析和优化提供数据支持。这些收集任务可以随时中断而不影响程序的主要功能。
b.维护服务任务
a.缓存清理服务
定期清理过期的缓存数据，维护内存使用效率。这种清理任务可以在程序退出时立即终止，不会造成数据丢失或不一致。
b.日志轮转服务
自动管理和轮转日志文件，防止日志文件过大占用过多磁盘空间。日志服务被中断不会影响程序的核心业务逻辑。
c.辅助功能任务
a.预加载服务
在后台预加载可能需要的数据或资源，提高程序的响应速度。预加载任务的中断不会影响程序的功能完整性。
b.心跳检测服务
定期检查外部服务或连接的健康状态，确保系统的可靠性。心跳检测的临时中断不会影响主要业务流程。
04.守护线程与非守护线程的区别
a.生命周期差异
a.非守护线程特性
非守护线程是程序的主要执行单元，程序会等待所有非守护线程执行完成才会退出。这种机制保证了重要任务的完整执行。
b.守护线程特性
守护线程的生命周期受制于非守护线程，当所有非守护线程结束时，守护线程会被强制终止。这种设计适用于后台辅助任务。
b.程序退出影响
a.阻塞行为差异
非守护线程会阻塞程序退出，确保任务完成；守护线程不会阻塞程序退出，允许程序快速终止。
b.适用场景区分
关键业务逻辑使用非守护线程，确保完整性；后台维护任务使用守护线程，避免阻塞程序退出。
c.资源管理策略
a.清理机制差异
非守护线程可以确保资源得到完整清理；守护线程需要考虑到被强制中断的可能性，采用更加健壮的资源管理策略。
b.状态一致性保证
非守护线程更容易保证数据状态的一致性；守护线程需要使用原子操作或其他同步机制来维护数据一致性。
05.守护线程的最佳实践
a.设计原则
a.无状态设计
守护线程应该设计为无状态或状态可恢复的形式，避免因为强制中断导致数据不一致或系统状态异常。
b.幂等操作
守护线程执行的操作应该是幂等的，即重复执行不会产生副作用，这样即使线程被中断，重新启动后也不会有问题。
b.错误处理策略
a.异常捕获机制
守护线程应该有完善的异常处理机制，避免因未处理的异常导致线程意外终止，影响后台服务的稳定性。
b.日志记录策略
守护线程应该记录重要的操作和状态信息，便于调试和监控。日志记录应该是线程安全的，避免并发写入导致的数据混乱。
c.性能优化考虑
a.资源使用控制
守护线程应该合理控制资源使用，避免过度消耗CPU或内存资源影响主程序的性能。
b.执行频率调节
根据实际需求调整守护线程的执行频率，避免过于频繁的执行造成系统负担。
2 GIL全局解释器锁
2.1 GIL的工作原理
01.GIL概念定义
a.基本定义
GIL全局解释器锁是Python解释器中的一个互斥锁，它保护对Python对象的访问，确保在任何时刻只有一个线程在执行Python字节码。这是CPython解释器实现线程安全的核心机制。
b.设计目的
GIL的主要目的是简化内存管理，防止多个线程同时访问Python对象时发生数据竞争和内存泄露。它使得C扩展模块的开发更加简单，不需要考虑复杂的线程安全问题。
c.作用范围
GIL只作用于Python字节码的执行，对于纯Python代码来说，多线程无法实现真正的并行执行。但是对于I/O密集型任务和C扩展操作，GIL会被释放，允许其他线程运行。
02.GIL的实现机制
a.互斥锁原理
GIL本质上是一个操作系统级别的互斥锁，使用线程同步机制确保同一时间只有一个线程能够获得解释器的控制权。当一个线程获得GIL后，其他线程必须等待它释放后才能执行。
b.锁的获取和释放
a.获取时机
线程在开始执行Python字节码之前必须先获取GIL，这个操作是自动完成的，开发者无法直接控制。获取GIL的过程是原子的，保证了线程安全性。
b.释放条件
GIL会在特定条件下自动释放，包括线程执行时间过长、遇到I/O操作、显式调用释放函数等。这些机制确保了系统的响应性和公平性。
c.线程切换机制
a.时间片轮转
CPython实现了基于时间片的线程调度，默认情况下一个线程执行100个字节码指令后就会释放GIL，让其他线程有机会执行。这个间隔可以通过sys.setcheckinterval()调整。
b.协作式调度
Python的线程调度是协作式的，线程不会强制被中断，而是在合适的时机主动释放GIL。这种机制避免了强制中断导致的数据不一致问题。
03.GIL的性能影响
a.CPU密集型任务
a.单核执行限制
在CPU密集型任务中，由于GIL的存在，多线程程序在单核处理器上的性能提升有限，甚至可能因为线程切换开销而导致性能下降。
b.多核资源浪费
在多核处理器上，GIL使得Python多线程无法利用多个CPU核心，即使有多个线程同时运行，也只有一个核心在执行Python代码，造成了计算资源的浪费。
b.I/O密集型任务
a.I/O等待优化
对于I/O密集型任务，GIL的影响较小，因为线程在等待I/O操作时会释放GIL，允许其他线程执行。这使得多线程在处理网络请求、文件操作等场景中仍然有效。
b.并发性能提升
在I/O密集型应用中，多线程可以通过并发处理多个I/O操作来提高整体性能，GIL不会成为瓶颈，因为大部分时间线程都在等待I/O完成。
04.GIL的历史发展
a.设计背景
GIL的设计源于早期Python版本的需要，当时内存管理机制较为简单，GIL提供了一种简单有效的线程安全保证方案。这个设计决策使得Python能够快速支持多线程编程。
b.技术演进
随着Python的发展，GIL的实现不断优化，包括改进调度算法、优化锁机制、引入更好的I/O处理等。但是GIL的基本架构和核心原理保持不变。
c.移除GIL的挑战
移除GIL是一个复杂的技术挑战，涉及内存管理、垃圾回收、C扩展兼容性等多个方面。虽然有一些替代方案被提出，但到目前为止，CPython仍然保留GIL机制。
2.2 GIL对多线程的影响
01.并发与并行概念区别
a.并发执行
并发是指程序的多个任务在宏观上同时执行，但在微观上是通过快速切换执行权来实现的。在Python中，由于GIL的存在，多线程实现的是并发而不是真正的并行。
b.并行执行
并行是指多个任务在同一时刻真正同时执行，需要多个处理器核心的支持。Python的多线程由于GIL限制无法实现并行，但多进程可以实现真正的并行。
c.GIL对并发的限制
GIL限制了Python多线程的并发能力，在同一时刻只有一个线程能够执行Python字节码，这降低了多线程在CPU密集型任务中的效果。
02.多线程性能分析
a.CPU密集型应用
a.性能下降现象
在CPU密集型应用中，多线程的性能可能不如单线程，因为线程切换和GIL竞争带来了额外的开销。每个线程都需要竞争GIL，这种竞争会消耗CPU时间。
b.资源争用问题
多个线程频繁竞争GIL会导致大量的上下文切换，增加系统负载。在高频率的锁竞争中，系统的整体性能会显著下降。
b.I/O密集型应用
a.GIL释放机制
在I/O操作期间，线程会释放GIL，让其他线程有机会执行。这使得多线程在I/O密集型应用中能够有效提高整体吞吐量。
b.并发优势
虽然不能并行执行，但多线程可以在等待I/O操作完成时执行其他任务，提高了程序的响应性和资源利用率。
03.线程竞争与调度
a.GIL竞争机制
a.公平性问题
GIL的调度不是完全公平的，某些线程可能获得更多的执行时间，特别是在长时间运行的线程和短暂线程之间的竞争中。
b.优先级影响
操作系统的线程调度优先级会影响GIL的获取，高优先级的线程更容易获得GIL，可能导致低优先级线程饥饿。
b.线程切换开销
a.上下文切换成本
每次GIL的释放和获取都需要进行线程上下文切换，这包括保存和恢复线程状态，更新调度信息等操作，都会消耗CPU时间。
b.缓存失效问题
线程切换会导致CPU缓存失效，新线程需要重新加载代码和数据到缓存中，这会降低执行效率。
04.多线程适用场景
a.适用场景分析
a.I/O密集型任务
文件读写、网络请求、数据库操作等I/O密集型任务适合使用多线程，因为这些操作会释放GIL，允许其他线程执行。
b.事件驱动应用
GUI应用、Web服务器等需要同时处理多个事件的应用适合使用多线程，每个线程可以处理不同的事件源。
b.不适用场景
a.纯计算任务
数学计算、图像处理等纯CPU密集型任务不适合使用多线程，因为GIL限制了并行执行效果，多进程是更好的选择。
b.实时性要求高
对实时性要求高的应用不适合使用多线程，因为GIL的调度延迟和线程切换开销会影响响应时间。
05.替代方案对比
a.多进程方案
a.并行能力
多进程可以真正利用多核CPU实现并行执行，每个进程有自己独立的GIL，避免了进程间的GIL竞争。
b.资源开销
进程的创建和管理开销比线程大，内存占用也更多，但可以通过进程池来减少这种开销。
b.异步编程方案
a.单线程并发
异步编程在单线程内实现并发，避免了GIL问题，适合I/O密集型任务，可以提高程序的性能和响应性。
b.编程复杂度
异步编程的复杂度较高，需要理解事件循环、协程等概念，对开发者的要求更高。
2.3 CPU密集型与IO密集型
01.CPU密集型任务特征
a.任务定义
CPU密集型任务是指主要消耗CPU计算资源的任务，如数学计算、数据加密、图像处理、科学计算等。这类任务的特点是计算量大，而I/O操作相对较少。
b.GIL影响分析
在CPU密集型任务中，GIL成为严重的性能瓶颈，因为多个线程无法真正并行执行Python代码，只能通过时间片轮转实现并发，性能提升有限甚至可能下降。
c.资源利用特点
CPU密集型任务会大量使用CPU核心，但在GIL限制下，多线程只能利用单个CPU核心，其他核心处于空闲状态，造成计算资源的浪费。
02.IO密集型任务特征
a.任务定义
IO密集型任务是指主要消耗I/O等待时间的任务，如文件读写、网络请求、数据库访问、用户输入等。这类任务的特点是大部分时间都在等待I/O操作完成。
b.GIL影响分析
在I/O密集型任务中，GIL的影响相对较小，因为线程在等待I/O时会释放GIL，允许其他线程执行。多线程可以有效利用等待时间提高整体性能。
c.并发优势体现
多线程在I/O密集型任务中能够实现真正的并发，当一个线程等待I/O时，其他线程可以继续执行，大大提高了系统的吞吐量和响应性。
03.任务类型识别方法
a.性能分析工具
a.使用cProfile分析
通过Python内置的cProfile模块分析程序的执行时间分布，识别哪些函数消耗了大量的CPU时间，从而判断任务类型。
b.使用top/htop监控
在Linux系统中，使用top或htop工具监控CPU使用率，如果CPU使用率持续很高，说明是CPU密集型任务。
b.代码特征判断
a.计算密集型特征
包含大量数学运算、循环、递归、算法处理等代码，很少涉及文件操作或网络请求。这类代码通常在Python解释器内部执行。
b.I/O密集型特征
包含大量文件操作、网络请求、数据库查询、GUI事件处理等代码。这些操作会触发系统调用，导致GIL释放。
04.性能优化策略
a.CPU密集型任务优化
a.多进程方案
使用multiprocessing模块创建多个进程，每个进程有独立的GIL，可以真正利用多核CPU实现并行计算，显著提高性能。
b.C扩展优化
将计算密集的部分用C语言重写为扩展模块，在C扩展中可以释放GIL，实现真正的并行执行。NumPy、SciPy等库就是这种策略的典型应用。
c.Just-In-Time编译
使用PyPy、Numba等JIT编译器，将Python代码编译为本地机器码，部分绕过GIL的限制，提高执行效率。
b.I/O密集型任务优化
a.线程池方案
使用ThreadPoolExecutor管理线程池，避免频繁创建和销毁线程的开销，同时控制并发线程数量，防止资源过度消耗。
b.异步编程方案
使用asyncio、aiohttp等异步框架，在单线程内实现高效的I/O并发，避免线程切换开销和GIL竞争。
c.连接池优化
对于数据库连接、网络连接等资源，使用连接池技术减少连接建立和关闭的开销，提高I/O操作效率。
05.混合型任务处理
a.任务分解策略
a.功能模块分离
将程序中的CPU密集型和I/O密集型部分分离到不同的模块中，根据各自的特点选择最适合的并发方案。
b.管道式处理
使用生产者-消费者模式，将I/O操作和计算操作分离，通过队列连接，实现任务的流水线处理。
b.架构设计考虑
a.分层并发设计
在系统设计中，对不同层次的任务采用不同的并发策略，I/O层使用多线程或异步，计算层使用多进程。
b.资源调度优化
合理分配系统资源，根据任务类型动态调整线程和进程的数量，实现最优的资源利用效率。
2.4 绕过GIL的方法
01.多进程并行方案
a.multiprocessing模块
multiprocessing是Python标准库中的多进程模块，每个进程都有独立的Python解释器和GIL，可以实现真正的并行执行。进程间通过管道、队列、共享内存等方式进行通信。
b.进程间通信机制
a.Queue队列通信
使用multiprocessing.Queue实现进程间的数据传递，Queue提供了线程安全的进程间通信机制，支持多种数据类型的传递。
b.Pipe管道通信
multiprocessing.Pipe提供双向通信通道，适合点对点的进程间通信，性能比Queue更高但功能相对简单。
c.共享内存
使用multiprocessing.Value和multiprocessing.Array实现进程间的共享内存，适合大量数据的共享，但需要注意同步问题。
c.进程池管理
a.Pool对象使用
multiprocessing.Pool提供了进程池管理功能，可以重用进程对象，避免频繁创建销毁进程的开销，提高系统效率。
b.任务分配策略
进程池自动处理任务的分配和负载均衡，开发者只需要关注业务逻辑，不需要手动管理进程的创建和销毁。
02.异步编程方案
a.asyncio框架
asyncio是Python的异步编程框架，通过事件循环和协程实现单线程并发。由于是单线程执行，完全避免了GIL问题，适合I/O密集型任务。
b.协程机制
a.async/await语法
使用async定义协程函数，await等待异步操作完成。这种语法使得异步代码看起来像同步代码，降低了编程复杂度。
b.事件循环调度
asyncio的事件循环负责调度和执行协程，当遇到I/O操作时自动切换到其他协程，实现高效的I/O并发。
c.异步生态
a.aiohttp网络库
提供异步HTTP客户端和服务器功能，是构建高性能Web应用的重要组件。
b.asyncpg数据库库
提供异步PostgreSQL数据库访问，支持高效的数据库连接池和查询操作。
03.C扩展开发方案
a.扩展模块开发
使用C或C++编写Python扩展模块，在扩展中可以释放GIL，实现真正的并行计算。NumPy、SciPy等科学计算库就是采用这种方案。
b.GIL释放机制
a.Py_BEGIN_ALLOW_THREADS宏
在C扩展中，通过这个宏释放GIL，让其他线程可以执行Python代码。长时间运行的C代码应该使用这个机制。
b.Py_END_ALLOW_THREADS宏
在C代码执行完成后，使用这个宏重新获取GIL，确保后续的Python操作是线程安全的。
c.现有高性能库
a.NumPy数组计算
NumPy的数组操作在C层面执行，可以释放GIL，实现高效的并行数值计算。
b.Pandas数据处理
Pandas的许多操作都使用了优化的C代码，可以绕过GIL的限制，提供高性能的数据处理能力。
04.JIT编译方案
a.Pypy解释器
PyPy是Python的JIT编译器实现，可以将Python代码编译为本地机器码执行。在某些场景下可以绕过GIL限制，提高执行效率。
b.Numba编译器
a.即时编译优化
Numba可以将Python函数编译为高效的机器码，特别适合数值计算和科学计算任务。
b.并行执行支持
Numba支持自动并行化，可以将循环和数组操作编译为并行代码，充分利用多核CPU。
c.Cython工具
a.Python到C的转换
Cython可以将Python代码转换为C代码，然后编译为扩展模块，实现接近C语言的性能。
b.类型声明优化
通过添加类型声明，Cython可以生成更高效的C代码，减少运行时开销。
05.替代Python解释器
a.Jython解释器
Jython是运行在JVM上的Python实现，使用Java的线程机制，没有GIL限制。适合需要真正多线程并发的Java环境应用。
b.IronPython解释器
IronPython是运行在.NET平台上的Python实现，使用.NET的线程机制，同样没有GIL限制。适合与.NET生态系统集成的应用。
c.GraalPython解释器
基于GraalVM的Python实现，可以将Python代码编译为本地代码执行，部分绕过GIL限制，提供更好的性能。
06.架构设计策略
a.微服务架构
将应用拆分为多个微服务，每个服务独立运行，通过消息队列或HTTP API通信。这样可以在服务级别实现并行，避免单进程的GIL限制。
b.分布式计算
使用分布式计算框架如Dask、Ray等，将任务分配到多个节点上并行执行，实现更大规模的并行计算。
c.混合架构模式
结合多种技术方案，如CPU密集型任务使用多进程，I/O密集型任务使用异步编程，网络服务使用微服务架构，实现最优的系统性能。
3 线程池
3.1 ThreadPoolExecutor
01.ThreadPoolExecutor概述
a.基本概念
ThreadPoolExecutor是Python标准库concurrent.futures模块中的一个线程池实现，它提供了一种高效的方式来管理和重用线程对象，避免了频繁创建和销毁线程的开销。
b.设计理念
线程池的核心思想是预先创建一定数量的线程并维护它们，当有任务提交时，从池中获取空闲线程执行任务，任务完成后线程返回池中等待下一个任务。
c.主要优势
减少线程创建和销毁的开销，控制并发线程数量避免资源过度消耗，提供统一的任务提交和结果获取接口，简化多线程编程的复杂性。
02.ThreadPoolExecutor核心参数
a.max_workers参数
a.参数作用
指定线程池中最大工作线程数量，这是线程池最重要的参数之一，直接影响程序的性能和资源使用。
b.基本使用示例
---
# 基本的ThreadPoolExecutor创建示例
import concurrent.futures
import time
import threading
def worker_task(task_id):
"""简单的工作任务"""
thread_name = threading.current_thread().name
time.sleep(0.5) # 模拟工作
return f"任务 {task_id} 在 {thread_name} 中完成"
# 创建一个最多包含4个工作线程的线程池
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
print(f"线程池已创建，最大工作线程数: 4")
# 提交一些任务
futures = []
for i in range(8):
future = executor.submit(worker_task, i)
futures.append(future)
# 等待所有任务完成
for future in concurrent.futures.as_completed(futures):
result = future.result()
print(f"任务完成: {result}")
---
c.默认值设置
在Python 3.8+中，默认值为min(32, (os.cpu_count() or 1) + 4)，这个默认值考虑了系统的CPU核心数和实际需求。
d.参数调优示例
---
# 不同max_workers参数的性能对比示例
import os
import time
import concurrent.futures
def cpu_intensive_task(n):
"""CPU密集型任务：计算斐波那契数列"""
if n <= 1:
return n
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b
# 测试不同的线程池大小
cpu_count = os.cpu_count()
worker_counts = [1, cpu_count, cpu_count * 2, cpu_count * 4]
for workers in worker_counts:
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
futures = [executor.submit(cpu_intensive_task, 35) for _ in range(20)]
results = [future.result() for future in futures]
end_time = time.time()
print(f"工作线程数: {workers}, 耗时: {end_time - start_time:.2f}秒")
---
b.thread_name_prefix参数
a.命名作用
为线程池中的线程设置名称前缀，便于调试和监控线程的执行情况。
b.命名示例
---
# 使用thread_name_prefix参数的示例
import concurrent.futures
import threading
def task_with_thread_info(task_id):
"""显示当前线程信息的任务"""
thread_name = threading.current_thread().name
thread_id = threading.current_thread().ident
return f"任务 {task_id} 在线程 {thread_name} (ID: {thread_id}) 中执行"
# 创建带自定义前缀的线程池
with concurrent.futures.ThreadPoolExecutor(
max_workers=3,
thread_name_prefix="Worker"
) as executor:
# 提交任务
futures = [executor.submit(task_with_thread_info, i) for i in range(5)]
# 获取结果
for future in concurrent.futures.as_completed(futures):
print(future.result())
---
c.默认行为
如果不指定，系统会自动生成线程名称，使用前缀可以更好地识别不同线程池的线程。
c.initializer和initargs参数
a.初始化函数
每个工作线程启动时会调用initializer函数，可以用于线程级别的初始化工作。
b.初始化示例
---
# 使用initializer的示例
import concurrent.futures
import threading
import random
# 线程本地存储，每个线程都有自己的实例
thread_local_data = threading.local()
def thread_initializer():
"""线程初始化函数"""
# 为每个线程生成唯一的随机种子
thread_local_data.random_seed = random.randint(1, 1000)
thread_local_data.task_count = 0
print(f"线程 {threading.current_thread().name} 初始化完成，随机种子: {thread_local_data.random_seed}")
def task_with_local_data(task_id):
"""使用线程本地数据的任务"""
thread_local_data.task_count += 1
# 使用线程特定的随机种子
random.seed(thread_local_data.random_seed)
random_value = random.random()
thread_name = threading.current_thread().name
return f"任务 {task_id} - 线程: {thread_name}, 随机值: {random_value:.3f}, 执行次数: {thread_local_data.task_count}"
# 创建带初始化函数的线程池
with concurrent.futures.ThreadPoolExecutor(
max_workers=3,
initializer=thread_initializer
) as executor:
# 提交多个任务
futures = [executor.submit(task_with_local_data, i) for i in range(10)]
# 获取结果
for future in concurrent.futures.as_completed(futures):
print(future.result())
---
03.ThreadPoolExecutor核心方法
a.submit()方法
a.方法功能
将任务提交到线程池执行，立即返回Future对象，不等待任务完成。
b.submit()完整示例
---
# submit()方法的详细使用示例
import concurrent.futures
import time
import urllib.request
def download_url(url, timeout=10):
"""下载网页内容"""
try:
print(f"开始下载: {url}")
start_time = time.time()
with urllib.request.urlopen(url, timeout=timeout) as response:
content = response.read()
size = len(content)
end_time = time.time()
duration = end_time - start_time
return {
'url': url,
'size': size,
'duration': duration,
'status': 'success'
}
except Exception as e:
return {
'url': url,
'error': str(e),
'status': 'failed'
}
# URL列表
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/2',
'https://httpbin.org/delay/3',
'https://httpbin.org/status/200',
'https://httpbin.org/json'
]
# 使用submit()方法并行下载
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# 提交所有下载任务
future_to_url = {executor.submit(download_url, url): url for url in urls}
print("所有任务已提交，等待完成...")
# 处理完成的任务
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
result = future.result()
if result['status'] == 'success':
print(f"✓ {url}: {result['size']} bytes, 耗时 {result['duration']:.2f}s")
else:
print(f"✗ {url}: {result['error']}")
except Exception as exc:
print(f"✗ {url} 产生异常: {exc}")
---
c.返回值处理
返回Future对象，可以通过该对象获取任务执行结果或异常信息。
b.map()方法
a.批量处理功能
对多个输入数据并行执行同一个函数，类似于内置的map()函数，但是并行执行。
b.map()详细示例
---
# map()方法的详细使用示例
import concurrent.futures
import time
import math
def process_number(n):
"""处理单个数字的计算密集型任务"""
# 模拟一些计算工作
result = math.factorial(n % 20)
time.sleep(0.1) # 模拟处理时间
return f"处理 {n}: 阶乘结果为 {result}"
def process_file(filename):
"""处理文件的示例"""
try:
# 模拟文件处理
time.sleep(0.2)
return f"文件 {filename} 处理完成"
except Exception as e:
return f"文件 {filename} 处理失败: {e}"
# 示例1: 简单的数据处理
numbers = list(range(1, 11))
print("=== 使用 map() 处理数字 ===")
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
# map() 会保持输入输出顺序
results = list(executor.map(process_number, numbers))
for result in results:
print(result)
print("\n=== 使用 map() 处理文件 ===")
# 示例2: 文件批量处理
filenames = [f"data_{i}.txt" for i in range(1, 6)]
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# 设置超时时间
try:
results = list(executor.map(process_file, filenames, timeout=2))
for result in results:
print(result)
except concurrent.futures.TimeoutError:
print("某些任务超时了")
---
c.执行顺序保证
map()方法会按照输入的顺序返回结果，保证了输出顺序与输入顺序的一致性。
c.shutdown()方法
a.优雅关闭示例
---
# shutdown()方法的详细使用示例
import concurrent.futures
import time
def long_running_task(task_id, duration):
"""长时间运行的任务"""
print(f"任务 {task_id} 开始执行，预计耗时 {duration} 秒")
time.sleep(duration)
print(f"任务 {task_id} 完成")
return f"任务 {task_id} 结果"
# 手动管理线程池的生命周期
executor = concurrent.futures.ThreadPoolExecutor(max_workers=3)
try:
# 提交一些任务
futures = []
for i in range(5):
duration = 2 + i * 0.5 # 递增的执行时间
future = executor.submit(long_running_task, i, duration)
futures.append(future)
print("所有任务已提交")
# 获取部分结果
completed_count = 0
for future in concurrent.futures.as_completed(futures[:3]): # 只等前3个
result = future.result()
print(f"提前完成的任务: {result}")
completed_count += 1
print(f"已完成 {completed_count} 个任务，准备关闭线程池")
finally:
# 优雅关闭：等待所有已提交的任务完成
print("调用 shutdown(wait=True)...")
executor.shutdown(wait=True) # 等待所有任务完成
print("线程池已关闭")
# 对比：强制关闭
print("\n=== 强制关闭示例 ===")
executor2 = concurrent.futures.ThreadPoolExecutor(max_workers=2)
# 提交一些长时间任务
futures2 = [executor2.submit(long_running_task, i, 3) for i in range(3)]
print("等待1秒后强制关闭...")
time.sleep(1)
# 强制关闭：不等待未完成的任务
executor2.shutdown(wait=False)
print("线程池已强制关闭")
# 注意：已提交但未完成的任务可能会被中断
---
04.ThreadPoolExecutor上下文管理
a.with语句完整示例
---
# with语句的完整使用示例
import concurrent.futures
import time
import random
def database_operation(query_id):
"""模拟数据库操作"""
try:
# 模拟数据库连接和查询
print(f"查询 {query_id}: 连接数据库...")
time.sleep(0.5)
# 模拟查询执行
print(f"查询 {query_id}: 执行查询...")
time.sleep(random.uniform(0.2, 0.8))
# 模拟结果处理
result = f"查询 {query_id} 的结果: {random.randint(100, 999)}"
print(f"查询 {query_id}: 处理结果完成")
return result
except Exception as e:
print(f"查询 {query_id} 失败: {e}")
raise
def cleanup_resources():
"""清理资源的函数"""
print("清理数据库连接...")
time.sleep(0.1)
print("资源清理完成")
# 使用with语句自动管理线程池
print("=== 使用 with 语句的示例 ===")
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
print("线程池已创建")
# 提交数据库查询任务
query_ids = ['SELECT_USERS', 'GET_ORDERS', 'FETCH_PRODUCTS', 'UPDATE_STATS']
# 提交所有任务
future_to_query = {
executor.submit(database_operation, query_id): query_id
for query_id in query_ids
}
print(f"已提交 {len(future_to_query)} 个查询任务")
# 获取结果
results = []
for future in concurrent.futures.as_completed(future_to_query):
query_id = future_to_query[future]
try:
result = future.result()
results.append(result)
print(f"✓ {query_id}: {result}")
except Exception as exc:
print(f"✗ {query_id} 产生异常: {exc}")
print(f"成功完成 {len(results)} 个查询")
# 在with块内，线程池仍然可用
print("在with块内提交额外任务...")
extra_future = executor.submit(database_operation, "EXTRA_QUERY")
print(extra_future.result())
# 模拟一个异常情况
# raise ValueError("模拟异常") # 取消注释来测试异常处理
except Exception as e:
print(f"发生异常: {e}")
finally:
# with语句结束后，线程池自动关闭
print("with语句执行完毕，线程池已自动关闭")
cleanup_resources()
# 这里线程池已经不可用
try:
executor.submit(database_operation, "TEST") # 这会抛出异常
except Exception as e:
print(f"预期的异常（线程池已关闭）: {e}")
---
b.自动资源管理
使用with语句可以确保线程池在使用完毕后自动关闭，即使发生异常也能正确清理资源。
3.2 submit与map方法
01.submit方法详解
a.方法签名与参数
submit(fn, *args, **kwargs)方法接受一个函数和其参数，立即返回Future对象而不等待任务完成。这种异步提交方式使得主线程可以继续执行其他工作。
b.submit方法详细示例
---
# submit()方法的详细使用示例
import concurrent.futures
import time
import requests
import json
def fetch_weather_data(city):
"""获取天气数据的API调用"""
try:
# 模拟API调用（使用免费的天气API）
api_url = f"http://api.openweathermap.org/data/2.5/weather?q={city}&appid=demo"
print(f"开始获取 {city} 的天气数据...")
start_time = time.time()
# 这里使用模拟数据，实际应用中应该调用真实API
# response = requests.get(api_url, timeout=5)
# data = response.json()
# 模拟API调用延迟
time.sleep(1)
# 模拟返回数据
data = {
'city': city,
'temperature': 20 + len(city) % 15, # 模拟温度
'humidity': 40 + len(city) % 40, # 模拟湿度
'weather': 'sunny' if len(city) % 2 == 0 else 'cloudy'
}
end_time = time.time()
duration = end_time - start_time
return {
'success': True,
'data': data,
'duration': duration,
'timestamp': time.time()
}
except Exception as e:
return {
'success': False,
'error': str(e),
'city': city
}
def process_weather_result(result):
"""处理天气数据的业务逻辑"""
if result['success']:
data = result['data']
processed_data = {
'city': data['city'],
'celsius': data['temperature'],
'fahrenheit': data['temperature'] * 9/5 + 32,
'humidity': data['humidity'],
'conditions': data['weather'],
'fetch_duration': result['duration']
}
return processed_data
else:
return {'error': result['error'], 'city': result['city']}
# 城市列表
cities = ['Beijing', 'Shanghai', 'Guangzhou', 'Shenzhen', 'Hangzhou']
print("=== submit() 方法示例：异步获取天气数据 ===")
# 创建线程池
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# 提交所有天气数据获取任务
future_to_city = {executor.submit(fetch_weather_data, city): city for city in cities}
print(f"已提交 {len(future_to_city)} 个天气数据获取任务")
# 收集完成的任务结果
weather_results = []
failed_requests = []
# as_completed() 会按照任务完成的顺序返回Future对象
for future in concurrent.futures.as_completed(future_to_city):
city = future_to_city[future]
try:
result = future.result(timeout=10) # 设置超时时间
if result['success']:
weather_results.append(result)
print(f"✓ {city}: 数据获取成功 ({result['duration']:.2f}s)")
else:
failed_requests.append(result)
print(f"✗ {city}: {result['error']}")
except concurrent.futures.TimeoutError:
print(f"✗ {city}: 请求超时")
except Exception as exc:
print(f"✗ {city}: 产生异常 {exc}")
print(f"\n成功获取 {len(weather_results)} 个城市的数据，失败 {len(failed_requests)} 个")
# 对获取到的数据进行后续处理
print("\n=== 处理天气数据 ===")
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as processor:
# 提交数据处理任务
processing_futures = [processor.submit(process_weather_result, result)
for result in weather_results]
# 获取处理结果
for future in concurrent.futures.as_completed(processing_futures):
try:
processed_data = future.result()
if 'error' not in processed_data:
data = processed_data
print(f"处理完成 - {data['city']}: {data['celsius']}°C/{data['fahrenheit']:.1f}°F, "
f"湿度: {data['humidity']}%, 天气: {data['conditions']}")
else:
print(f"处理失败: {processed_data['error']}")
except Exception as e:
print(f"数据处理异常: {e}")
---
c.返回值Future对象
a.Future对象概念
Future代表异步执行的结果，提供了检查任务状态、获取结果、处理异常等功能。它是并发编程中的重要抽象。
b.结果获取机制
通过Future.result()方法可以阻塞等待任务完成并获取结果，如果任务执行失败，会抛出相应的异常。
02.map方法详解
a.map方法详细示例
---
# map()方法的详细使用示例
import concurrent.futures
import time
import math
import hashlib
def calculate_hash(filename, chunk_size=8192):
"""计算文件的哈希值"""
# 模拟文件内容
file_content = f"This is the content of {filename}. " * 100
content_bytes = file_content.encode('utf-8')
# 计算MD5哈希
hash_md5 = hashlib.md5()
# 模拟分块读取文件
for i in range(0, len(content_bytes), chunk_size):
chunk = content_bytes[i:i + chunk_size]
hash_md5.update(chunk)
time.sleep(0.001) # 模拟I/O延迟
return {
'filename': filename,
'md5_hash': hash_md5.hexdigest(),
'size': len(content_bytes)
}
def process_image(filename, operations):
"""模拟图片处理操作"""
# 模拟不同类型的图片处理
result = {'filename': filename, 'operations': operations}
for op in operations:
if op == 'resize':
# 模拟图片缩放
time.sleep(0.2)
result['resized'] = f"{filename}_resized"
elif op == 'compress':
# 模拟图片压缩
time.sleep(0.3)
result['compressed'] = f"{filename}_compressed"
elif op == 'filter':
# 模拟滤镜应用
time.sleep(0.1)
result['filtered'] = f"{filename}_filtered"
return result
def mathematical_computation(n):
"""数学计算任务"""
operations = [
('factorial', math.factorial(n % 10)),
('sqrt', math.sqrt(max(1, n))),
('log', math.log(max(1, n))),
('power', n ** 2)
]
# 模拟计算密集型操作
time.sleep(0.05) # 模拟计算时间
return {
'input': n,
'operations': dict(operations)
}
print("=== map() 方法示例1：批量文件哈希计算 ===")
# 文件列表
file_list = [f"document_{i}.pdf" for i in range(1, 11)]
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
# 使用map()并行计算文件哈希
print("开始计算文件哈希值...")
start_time = time.time()
# map() 保持输入输出顺序
hash_results = list(executor.map(calculate_hash, file_list))
end_time = time.time()
print(f"哈希计算完成，耗时: {end_time - start_time:.2f}秒")
# 显示结果（保持原有顺序）
for i, result in enumerate(hash_results, 1):
print(f"{i:2d}. {result['filename']}: {result['md5_hash'][:8]}... "
f"({result['size']} bytes)")
print("\n=== map() 方法示例2：图片批量处理 ===")
# 图片处理任务列表
image_tasks = [
('image1.jpg', ['resize', 'compress']),
('image2.jpg', ['resize', 'filter']),
('image3.jpg', ['compress', 'filter']),
('image4.jpg', ['resize', 'compress', 'filter']),
('image5.jpg', ['compress'])
]
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# 分离文件名和操作列表
filenames, operations = zip(*image_tasks)
print("开始批量处理图片...")
start_time = time.time()
# 使用map()并行处理图片
image_results = list(executor.map(process_image, filenames, operations))
end_time = time.time()
print(f"图片处理完成，耗时: {end_time - start_time:.2f}秒")
# 显示处理结果
for result in image_results:
print(f"图片: {result['filename']}")
for key, value in result.items():
if key != 'filename' and key != 'operations':
print(f" ✓ {key}: {value}")
print("\n=== map() 方法示例3：数学计算 ===")
# 数字列表
numbers = list(range(1, 21))
with concurrent.futures.ThreadPoolExecutor(max_workers=6) as executor:
print("开始数学计算...")
start_time = time.time()
# 使用map()并行计算
math_results = list(executor.map(mathematical_computation, numbers))
end_time = time.time()
print(f"数学计算完成，耗时: {end_time - start_time:.2f}秒")
# 显示部分结果
for i, result in enumerate(math_results[:10], 1):
ops = result['operations']
print(f"{i:2d}. 输入: {result['input']}, "
f"阶乘: {ops['factorial']}, "
f"平方根: {ops['sqrt']:.2f}")
---
b.顺序保证特性
a.输入输出顺序一致
map方法保证结果的返回顺序与输入数据的顺序完全一致，这对于需要保持顺序的场景非常重要。
b.内部实现机制
内部使用队列来维护任务顺序，即使某些任务先完成，也会等待前面的任务完成后再返回结果。
03.submit与map方法对比
a.性能对比示例
---
# submit() 和 map() 的性能对比示例
import concurrent.futures
import time
def dummy_task(task_id):
"""模拟任务"""
time.sleep(0.1)
return f"Task {task_id} completed"
def test_submit_method(task_count, workers):
"""测试submit方法的性能"""
print(f"\n=== submit() 方法测试 ===")
print(f"任务数量: {task_count}, 工作线程: {workers}")
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
# 使用submit提交任务
futures = [executor.submit(dummy_task, i) for i in range(task_count)]
# 等待所有任务完成
results = [future.result() for future in futures]
end_time = time.time()
print(f"提交方式: submit(), 耗时: {end_time - start_time:.2f}秒")
print(f"第一个结果: {results[0]}")
print(f"最后一个结果: {results[-1]}")
return end_time - start_time
def test_map_method(task_count, workers):
"""测试map方法的性能"""
print(f"\n=== map() 方法测试 ===")
print(f"任务数量: {task_count}, 工作线程: {workers}")
start_time = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
# 使用map执行任务
results = list(executor.map(dummy_task, range(task_count)))
end_time = time.time()
print(f"提交方式: map(), 耗时: {end_time - start_time:.2f}秒")
print(f"第一个结果: {results[0]}")
print(f"最后一个结果: {results[-1]}")
return end_time - start_time
# 测试不同场景
task_counts = [10, 50, 100]
worker_counts = [2, 4, 8]
print("=== submit() vs map() 性能对比 ===")
for task_count in task_counts:
for workers in worker_counts:
submit_time = test_submit_method(task_count, workers)
map_time = test_map_method(task_count, workers)
improvement = ((submit_time - map_time) / submit_time) * 100
print(f"性能差异: map() 比 submit() {'快' if improvement > 0 else '慢'} {abs(improvement):.1f}%")
---
b.使用场景对比
a.submit适用场景
适合处理不同类型的任务，需要独立控制每个任务的执行和结果获取，灵活性更高。
b.map适用场景
适合批量处理相同类型的任务，特别是需要保持输入输出顺序的场景，代码更简洁。
04.实际应用示例
a.submit应用场景：复杂的业务流程
---
# submit() 在复杂业务流程中的应用示例
import concurrent.futures
import time
import random
class OrderProcessor:
"""订单处理器"""
def __init__(self):
self.inventory = {
'product_1': 100,
'product_2': 50,
'product_3': 75,
'product_4': 200
}
def validate_order(self, order_id, products):
"""验证订单"""
print(f"验证订单 {order_id}...")
time.sleep(0.2) # 模拟验证时间
for product, quantity in products.items():
if product not in self.inventory:
return {'valid': False, 'error': f'产品 {product} 不存在'}
if self.inventory[product] < quantity:
return {'valid': False, 'error': f'产品 {product} 库存不足'}
return {'valid': True, 'order_id': order_id}
def process_payment(self, order_id, amount):
"""处理支付"""
print(f"处理订单 {order_id} 的支付，金额: {amount}...")
time.sleep(random.uniform(0.5, 1.5)) # 模拟支付处理时间
# 90% 成功率
if random.random() < 0.9:
return {'success': True, 'order_id': order_id, 'amount': amount}
else:
return {'success': False, 'order_id': order_id, 'error': '支付失败'}
def update_inventory(self, order_id, products):
"""更新库存"""
print(f"更新订单 {order_id} 的库存...")
time.sleep(0.1) # 模拟数据库更新时间
for product, quantity in products.items():
self.inventory[product] -= quantity
return {'success': True, 'order_id': order_id, 'remaining_inventory': dict(self.inventory)}
def send_confirmation(self, order_id, customer_email):
"""发送确认邮件"""
print(f"发送订单 {order_id} 的确认邮件到 {customer_email}...")
time.sleep(0.3) # 模拟邮件发送时间
return {'success': True, 'order_id': order_id, 'email_sent': True}
def process_complex_order(order):
"""处理复杂订单的完整流程"""
processor = OrderProcessor()
order_id = order['order_id']
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# 步骤1：验证订单
validate_future = executor.submit(
processor.validate_order,
order_id,
order['products']
)
try:
validation_result = validate_future.result(timeout=2)
if not validation_result['valid']:
return {'status': 'failed', 'reason': validation_result['error']}
except concurrent.futures.TimeoutError:
return {'status': 'failed', 'reason': '订单验证超时'}
# 步骤2：处理支付（可以并行执行其他操作）
payment_future = executor.submit(
processor.process_payment,
order_id,
order['amount']
)
# 步骤3：预处理（可以与支付并行）
prep_future = executor.submit(
prepare_shipping,
order_id,
order['shipping_address']
)
try:
payment_result = payment_future.result(timeout=5)
prep_result = prep_future.result(timeout=3)
if not payment_result['success']:
return {'status': 'failed', 'reason': payment_result['error']}
except concurrent.futures.TimeoutError:
return {'status': 'failed', 'reason': '支付或预处理超时'}
# 步骤4：更新库存和发送确认（并行执行）
inventory_future = executor.submit(
processor.update_inventory,
order_id,
order['products']
)
confirmation_future = executor.submit(
processor.send_confirmation,
order_id,
order['customer_email']
)
# 等待所有最终步骤完成
try:
inventory_result = inventory_future.result(timeout=2)
confirmation_result = confirmation_future.result(timeout=3)
return {
'status': 'success',
'order_id': order_id,
'inventory_updated': inventory_result['success'],
'confirmation_sent': confirmation_result['success'],
'prep_result': prep_result
}
except concurrent.futures.TimeoutError:
return {'status': 'partial', 'reason': '某些后续步骤超时'}
def prepare_shipping(order_id, address):
"""准备发货"""
print(f"准备订单 {order_id} 的发货信息...")
time.sleep(0.4)
return {'order_id': order_id, 'address': address, 'ready': True}
# 订单数据
orders = [
{
'order_id': 'ORD001',
'products': {'product_1': 2, 'product_2': 1},
'amount': 299.99,
'customer_email': '[email protected]',
'shipping_address': '北京市朝阳区xxx街道xxx号'
},
{
'order_id': 'ORD002',
'products': {'product_3': 3, 'product_4': 1},
'amount': 599.99,
'customer_email': '[email protected]',
'shipping_address': '上海市浦东新区xxx路xxx号'
},
{
'order_id': 'ORD003',
'products': {'product_1': 1, 'product_2': 2, 'product_4': 1},
'amount': 449.99,
'customer_email': '[email protected]',
'shipping_address': '广州市天河区xxx大道xxx号'
}
]
print("=== 复杂订单处理流程示例 ===")
# 并行处理多个复杂订单
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
# 为每个复杂订单创建处理任务
order_futures = {executor.submit(process_complex_order, order): order['order_id']
for order in orders}
print(f"提交了 {len(order_futures)} 个复杂订单处理任务")
# 处理完成的订单
for future in concurrent.futures.as_completed(order_futures):
order_id = order_futures[future]
try:
result = future.result()
print(f"\n订单 {order_id} 处理结果:")
print(f" 状态: {result['status']}")
if result['status'] == 'success':
print(f" 库存更新: ✓")
print(f" 确认邮件: ✓")
print(f" 发货准备: ✓")
else:
print(f" 失败原因: {result.get('reason', '未知')}")
except Exception as exc:
print(f"订单 {order_id} 处理异常: {exc}")
---
3.3 Future对象
01.Future对象概述
a.基本定义
Future对象代表异步执行的操作结果，是concurrent.futures模块中的核心抽象。它封装了异步操作的状态、结果和异常信息。
b.Future对象基础示例
---
# Future对象的基础使用示例
import concurrent.futures
import time
import threading
def demonstrate_future_basics():
"""演示Future对象的基本概念"""
print("=== Future对象基础概念演示 ===")
def simple_task(task_id, duration):
"""简单的任务函数"""
thread_name = threading.current_thread().name
print(f"任务 {task_id} 在线程 {thread_name} 中开始执行...")
time.sleep(duration)
result = f"任务 {task_id} 完成，耗时 {duration} 秒"
print(f"任务 {task_id} 在线程 {thread_name} 中完成")
return result
# 创建线程池
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 创建Future对象")
# 提交任务并获取Future对象
future1 = executor.submit(simple_task, "A", 2)
future2 = executor.submit(simple_task, "B", 1)
print(f" Future对象1: {future1}")
print(f" Future对象2: {future2}")
print("\n2. 检查Future状态")
print(f" Future1 是否完成: {future1.done()}")
print(f" Future2 是否完成: {future2.done()}")
print("\n3. 等待任务完成")
# 等待较短的任务先完成
result2 = future2.result()
print(f" 获取Future2结果: {result2}")
print(f" Future2 是否完成: {future2.done()}")
print("\n4. 获取另一个任务结果")
result1 = future1.result()
print(f" 获取Future1结果: {result1}")
print(f" Future1 是否完成: {future1.done()}")
demonstrate_future_basics()
---
02.Future对象核心方法
a.result()方法详细示例
---
# result()方法的详细使用示例
import concurrent.futures
import time
import random
def demonstrate_result_method():
"""演示result()方法的各种用法"""
print("=== result() 方法演示 ===")
def successful_task(task_id):
"""成功完成的任务"""
time.sleep(1)
return {"task_id": task_id, "status": "success", "data": f"result_{task_id}"}
def failing_task(task_id):
"""会失败的任务"""
if random.random() < 0.5:
time.sleep(0.5)
raise ValueError(f"任务 {task_id} 遇到了预期错误")
else:
time.sleep(1.5)
return {"task_id": task_id, "status": "success"}
def timeout_task(timeout_duration):
"""会超时的任务"""
time.sleep(timeout_duration + 1) # 故意超时
return "这个结果永远不会返回"
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
print("1. 基本result()调用")
success_future = executor.submit(successful_task, "SUCCESS_001")
result = success_future.result()
print(f" 结果: {result}")
print("\n2. 带超时的result()调用")
timeout_future = executor.submit(timeout_task, 2)
try:
result = timeout_future.result(timeout=1)
print(f" 结果: {result}")
except concurrent.futures.TimeoutError:
print(" 超时: 任务在1秒内未完成")
print("\n3. 处理异常的result()调用")
fail_future = executor.submit(failing_task, "FAIL_001")
try:
result = fail_future.result(timeout=2)
print(f" 结果: {result}")
except ValueError as e:
print(f" 捕获到异常: {e}")
print("\n4. 多次调用result()的演示")
# result()可以多次调用，返回相同结果
multi_future = executor.submit(successful_task, "MULTI_001")
result1 = multi_future.result()
result2 = multi_future.result() # 第二次调用
print(f" 第一次调用: {result1}")
print(f" 第二次调用: {result2}")
print(f" 两次结果是否相同: {result1 == result2}")
demonstrate_result_method()
---
b.done()方法详细示例
---
# done()方法的详细使用示例
import concurrent.futures
import time
def demonstrate_done_method():
"""演示done()方法的用法"""
print("=== done() 方法演示 ===")
def check_status_task(task_id):
"""状态检查任务"""
time.sleep(2)
return f"任务 {task_id} 完成"
def quick_task(task_id):
"""快速任务"""
time.sleep(0.5)
return f"快速任务 {task_id} 完成"
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 创建不同时长的任务")
long_future = executor.submit(check_status_task, "LONG")
quick_future = executor.submit(quick_task, "QUICK")
print(" 长任务是否完成: ", long_future.done())
print(" 快速任务是否完成: ", quick_future.done())
print("\n2. 等待一段时间后检查状态")
time.sleep(1)
print(" 长任务是否完成: ", long_future.done())
print(" 快速任务是否完成: ", quick_future.done())
print("\n3. 获取快速任务结果")
quick_result = quick_future.result()
print(f" 快速任务结果: {quick_result}")
print(f" 快速任务是否完成: {quick_future.done()}")
print("\n4. 继续等待长任务")
time.sleep(2)
print(" 长任务是否完成: ", long_future.done())
long_result = long_future.result()
print(f" 长任务结果: {long_result}")
demonstrate_done_method()
---
c.cancel()方法详细示例
---
# cancel()方法的详细使用示例
import concurrent.futures
import time
import threading
def demonstrate_cancel_method():
"""演示cancel()方法的用法"""
print("=== cancel() 方法演示 ===")
def long_task(task_id):
"""长时间运行的任务"""
thread_name = threading.current_thread().name
print(f"长任务 {task_id} 在线程 {thread_name} 中开始执行...")
time.sleep(5)
print(f"长任务 {task_id} 在线程 {thread_name} 中完成")
return f"长任务 {task_id} 完成"
def short_task(task_id):
"""短时间任务"""
time.sleep(1)
return f"短任务 {task_id} 完成"
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 取消未开始的任务")
future1 = executor.submit(long_task, "CANCEL_TEST")
time.sleep(0.1) # 给任务一点时间，但不要让它开始执行
print(f" 任务取消前是否完成: {future1.done()}")
cancelled = future1.cancel()
print(f" 尝试取消任务: {'成功' if cancelled else '失败'}")
print(f" 任务取消后是否完成: {future1.done()}")
try:
result = future1.result(timeout=1)
print(f" 任务结果: {result}")
except concurrent.futures.CancelledError:
print(" 任务已被取消，调用result()抛出CancelledError")
print("\n2. 无法取消已开始的任务")
future2 = executor.submit(short_task, "NO_CANCEL")
time.sleep(0.2) # 确保任务已经开始
cancelled = future2.cancel()
print(f" 尝试取消已开始的任务: {'成功' if cancelled else '失败'}")
try:
result = future2.result(timeout=2)
print(f" 任务结果: {result}")
except Exception as e:
print(f" 任务异常: {e}")
demonstrate_cancel_method()
---
03.Future对象状态管理
a.状态转换详细示例
---
# Future状态转换的完整演示
import concurrent.futures
import time
import threading
def demonstrate_state_transitions():
"""演示Future对象的状态转换"""
print("=== Future状态转换演示 ===")
def monitor_task(task_id, duration):
"""可监控的任务"""
thread_name = threading.current_thread().name
print(f"[{thread_name}] 任务 {task_id} 开始执行")
time.sleep(duration)
result = f"任务 {task_id} 在 {thread_name} 中完成"
print(f"[{thread_name}] 任务 {task_id} 执行完成")
return result
def print_future_state(future, name):
"""打印Future对象的状态信息"""
states = []
if future.cancelled():
states.append("CANCELLED")
elif future.running():
states.append("RUNNING")
elif future.done():
states.append("DONE")
else:
states.append("PENDING")
status_info = f"{name}: {', '.join(states)}"
if future.done() and not future.cancelled():
try:
# 检查是否有异常
future.result(timeout=0)
status_info += " (SUCCESS)"
except Exception:
status_info += " (FAILED)"
return status_info
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 创建Future对象")
future1 = executor.submit(monitor_task, "TASK_A", 3)
future2 = executor.submit(monitor_task, "TASK_B", 1)
print(f" 初始状态 - {print_future_state(future1, 'Future1')}")
print(f" 初始状态 - {print_future_state(future2, 'Future2')}")
print("\n2. 监控状态变化")
start_time = time.time()
max_duration = 5
while time.time() - start_time < max_duration:
time.sleep(0.5)
print(f" {print_future_state(future1, 'Future1')}")
print(f" {print_future_state(future2, 'Future2')}")
print(" " + "-" * 50)
if future1.done() and future2.done():
break
print("\n3. 最终状态和结果")
print(f" Future1 - {print_future_state(future1, 'Future1')}")
print(f" Future2 - {print_future_state(future2, 'Future2')}")
try:
result1 = future1.result()
print(f" Future1 结果: {result1}")
except Exception as e:
print(f" Future1 异常: {e}")
try:
result2 = future2.result()
print(f" Future2 结果: {result2}")
except Exception as e:
print(f" Future2 异常: {e}")
demonstrate_state_transitions()
---
b.状态检查方法对比示例
---
# 状态检查方法的对比示例
import concurrent.futures
import time
def demonstrate_state_methods():
"""演示各种状态检查方法"""
print("=== 状态检查方法对比 ===")
def test_task(task_id):
"""测试任务"""
time.sleep(2)
if task_id % 3 == 0:
raise ValueError(f"任务 {task_id} 故意失败")
return f"任务 {task_id} 成功完成"
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
futures = []
for i in range(1, 6):
future = executor.submit(test_task, i)
futures.append(future)
print("初始状态检查:")
for i, future in enumerate(futures, 1):
print(f" Future{i}: done={future.done()}, running={future.running()}, cancelled={future.cancelled()}")
print("\n等待所有任务完成...")
# 等待所有任务完成
concurrent.futures.wait(futures)
print("\n最终状态检查:")
for i, future in enumerate(futures, 1):
done = future.done()
cancelled = future.cancelled()
running = future.running()
status = f"Future{i}: done={done}, running={running}, cancelled={cancelled}"
if done and not cancelled:
try:
future.result(timeout=0)
status += " (SUCCESS)"
except Exception:
status += " (FAILED)"
print(f" {status}")
demonstrate_state_methods()
---
04.Future对象回调机制
a.回调函数详细示例
---
# add_done_callback()方法的详细使用示例
import concurrent.futures
import time
import threading
def demonstrate_callbacks():
"""演示Future回调机制"""
print("=== Future回调机制演示 ===")
def callback_handler(future):
"""回调函数处理器"""
thread_name = threading.current_thread().name
print(f"[{thread_name}] 回调被触发")
try:
result = future.result()
print(f"[{thread_name}] 任务成功完成: {result}")
except Exception as e:
print(f"[{thread_name}] 任务失败: {e}")
def success_task(task_id):
"""成功任务"""
time.sleep(2)
return f"成功任务 {task_id} 完成"
def fail_task(task_id):
"""失败任务"""
time.sleep(1)
raise RuntimeError(f"失败任务 {task_id} 出错")
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 为成功任务添加回调")
success_future = executor.submit(success_task, "SUCCESS_001")
success_future.add_done_callback(callback_handler)
print("2. 为失败任务添加回调")
fail_future = executor.submit(fail_task, "FAIL_001")
fail_future.add_done_callback(callback_handler)
print("3. 添加多个回调函数")
def callback1(future):
print("回调1: 任务完成")
def callback2(future):
print("回调2: 处理任务结果")
multi_future = executor.submit(success_task, "MULTI_001")
multi_future.add_done_callback(callback1)
multi_future.add_done_callback(callback2)
print("4. 等待所有任务完成")
# 等待所有任务完成，这样回调才有机会执行
for future in [success_future, fail_future, multi_future]:
try:
future.result()
except Exception as e:
pass # 忽略异常，因为回调已经处理了
print("所有任务完成")
demonstrate_callbacks()
---
b.回调链和高级回调示例
---
# 回调链和高级回调模式示例
import concurrent.futures
import time
import threading
class TaskPipeline:
"""任务管道类"""
def __init__(self):
self.results = {}
def step1_callback(self, future):
"""第一步回调"""
step1_result = future.result()
task_id = step1_result['task_id']
print(f"步骤1完成 - 任务 {task_id}")
# 将结果存储，并触发下一步
self.results[f"{task_id}_step1"] = step1_result
return step1_result
def step2_callback(self, future):
"""第二步回调"""
step1_result = future.result()
task_id = step1_result['task_id']
print(f"步骤2开始 - 任务 {task_id}")
# 模拟第二步处理
step2_result = {
'task_id': task_id,
'step1_data': step1_result,
'processed': True,
'timestamp': time.time()
}
self.results[f"{task_id}_step2"] = step2_result
print(f"步骤2完成 - 任务 {task_id}")
return step2_result
def final_callback(self, future):
"""最终回调"""
step2_result = future.result()
task_id = step2_result['task_id']
print(f"最终处理完成 - 任务 {task_id}")
final_result = {
'task_id': task_id,
'all_steps': True,
'final_timestamp': time.time()
}
self.results[f"{task_id}_final"] = final_result
return final_result
def demonstrate_advanced_callbacks():
"""演示高级回调模式"""
print("=== 高级回调模式演示 ===")
def pipeline_task(task_id, data):
"""管道任务"""
time.sleep(1)
return {'task_id': task_id, 'data': data, 'status': 'step1_complete'}
pipeline = TaskPipeline()
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("创建任务管道...")
# 创建初始任务
initial_future = executor.submit(pipeline_task, "PIPELINE_001", "测试数据")
# 构建回调链
future1 = initial_future
future1.add_done_callback(lambda f: pipeline.step1_callback(f))
# 注意：在实际应用中，可能需要创建新的Future来执行下一步
# 这里简化处理，只演示回调机制
print("等待任务完成...")
try:
initial_future.result()
time.sleep(0.5) # 等待回调执行
print("\n管道结果:")
for key, value in pipeline.results.items():
print(f" {key}: {value}")
except Exception as e:
print(f"任务异常: {e}")
demonstrate_advanced_callbacks()
---
05.Future对象高级应用
a.异常处理高级示例
---
# Future异常处理的高级示例
import concurrent.futures
import time
import traceback
class FutureErrorHandler:
"""Future错误处理器"""
def __init__(self):
self.error_count = 0
self.success_count = 0
def handle_future_with_retry(self, executor, func, *args, max_retries=3, **kwargs):
"""带重试机制的Future处理"""
last_exception = None
for attempt in range(max_retries + 1):
try:
print(f"尝试执行任务 (第 {attempt + 1} 次)")
future = executor.submit(func, *args, **kwargs)
result = future.result(timeout=10)
self.success_count += 1
return result
except Exception as e:
last_exception = e
self.error_count += 1
print(f"第 {attempt + 1} 次尝试失败: {e}")
if attempt < max_retries:
print(f"准备第 {attempt + 2} 次重试...")
time.sleep(1) # 等待后重试
# 所有重试都失败了
print(f"任务在 {max_retries + 1} 次尝试后仍然失败")
raise last_exception
def batch_process_with_error_handling(self, executor, tasks):
"""批量处理并处理错误"""
futures = {}
results = {}
errors = {}
# 提交所有任务
for task_id, (func, args, kwargs) in tasks.items():
try:
future = executor.submit(func, *args, **kwargs)
futures[future] = task_id
except Exception as e:
errors[task_id] = str(e)
# 处理完成的任务
for future in concurrent.futures.as_completed(futures):
task_id = futures[future]
try:
result = future.result()
results[task_id] = result
except Exception as e:
error_info = {
'error': str(e),
'traceback': traceback.format_exc(),
'task_id': task_id
}
errors[task_id] = error_info
return results, errors
def demonstrate_advanced_error_handling():
"""演示高级错误处理"""
print("=== 高级错误处理演示 ===")
def unreliable_task(task_id, success_rate=0.7):
"""不可靠的任务（有一定概率失败）"""
import random
time.sleep(1)
if random.random() < success_rate:
return f"任务 {task_id} 成功完成"
else:
raise RuntimeError(f"任务 {task_id} 随机失败")
def network_request(url, should_fail=False):
"""模拟网络请求"""
time.sleep(0.5)
if should_fail:
raise ConnectionError(f"连接 {url} 失败")
return f"从 {url} 获取到数据"
handler = FutureErrorHandler()
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
print("1. 带重试机制的任务处理")
try:
result = handler.handle_future_with_retry(
executor, unreliable_task, "RETRY_TASK", 0.3, max_retries=3
)
print(f"重试成功: {result}")
except Exception as e:
print(f"重试失败: {e}")
print(f"\n统计: 成功 {handler.success_count} 次, 失败 {handler.error_count} 次")
print("\n2. 批量任务错误处理")
tasks = {
'task_1': (unreliable_task, ("TASK_1", 0.8), {}),
'task_2': (unreliable_task, ("TASK_2", 0.3), {}),
'task_3': (network_request, ("http://example.com/api1", False), {}),
'task_4': (network_request, ("http://example.com/api2", True), {}),
'task_5': (unreliable_task, ("TASK_3", 0.6), {}),
}
results, errors = handler.batch_process_with_error_handling(executor, tasks)
print(f"\n批量处理结果:")
print(f"成功任务: {len(results)}")
for task_id, result in results.items():
print(f" {task_id}: {result}")
print(f"\n失败任务: {len(errors)}")
for task_id, error in errors.items():
if isinstance(error, dict):
print(f" {task_id}: {error['error']}")
else:
print(f" {task_id}: {error}")
demonstrate_advanced_error_handling()
---
3.4 线程池大小设置
01.线程池大小的重要性
a.性能影响
线程池大小直接影响程序的性能表现，过小的线程池无法充分利用系统资源，过大的线程池会导致过度竞争和资源浪费。
b.资源利用率
合理的线程池大小能够最大化CPU和I/O资源的利用率，避免线程空闲或过度切换带来的性能损失。
c.系统稳定性
适当的线程池大小设置有助于维护系统的稳定性，防止因线程过多导致的内存耗尽或系统崩溃。
02.CPU密集型任务优化
a.线程数量计算公式
对于CPU密集型任务，推荐的线程数量为CPU核心数或CPU核心数+1。公式为：threads = cpu_count 或 cpu_count + 1
b.GIL影响考虑
由于Python GIL的存在，多线程在CPU密集型任务中无法真正并行，因此线程数不宜过多，通常设置为CPU核心数即可。
c.负载均衡策略
在多核系统中，合理分配线程到不同核心，避免某个核心过载而其他核心空闲的情况。
d.性能测试验证
通过基准测试验证不同线程数量下的性能表现，选择最优的线程池大小配置。
03.I/O密集型任务优化
a.线程数量策略
对于I/O密集型任务，可以设置较大的线程数量，通常为CPU核心数的2-5倍，因为大部分时间线程都在等待I/O操作。
b.阻塞时间考虑
线程数量应该根据I/O操作的阻塞时间来调整，阻塞时间越长，可以设置的线程数越多。
c.资源限制权衡
需要考虑系统资源限制，包括内存、文件描述符等，避免因线程过多导致资源耗尽。
d.动态调整机制
实现动态线程池大小调整，根据系统负载和I/O等待情况自动优化线程数量。
04.混合型任务处理
a.任务类型分析
首先分析应用中CPU密集型和I/O密集型任务的比例，根据比例确定线程池大小。
b.分层策略
采用分层线程池设计，为不同类型的任务设置不同大小的线程池，实现更精细的资源管理。
c.优先级调度
为不同重要性的任务设置不同的线程池，确保关键任务获得足够的执行资源。
d.负载监控
实时监控系统负载和任务执行情况，动态调整各线程池的大小分配。
05.系统资源考量
a.内存使用评估
每个线程都会消耗一定的内存空间，需要根据系统内存容量合理限制线程数量。
b.文件描述符限制
在Linux系统中，每个线程都会占用文件描述符，需要确保不超过系统限制。
c.上下文切换开销
线程数量过多会增加上下文切换的开销，影响系统整体性能。
d.缓存影响
过多的线程可能导致CPU缓存失效，降低程序执行效率。
06.实践经验指导
a.经验公式总结
CPU密集型：threads = cpu_count
I/O密集型：threads = cpu_count * (1 + average_wait_time / average_cpu_time)
混合型：根据实际测试调整，通常从CPU核心数开始逐步增加。
b.性能监控指标
监控CPU使用率、内存占用、线程切换频率、任务完成时间等指标来评估线程池效果。
c.渐进式调优
从较小的线程池开始，逐步增加线程数量，观察性能变化，找到最优配置点。
d.环境适应性
考虑不同部署环境的硬件配置差异，为生产环境和测试环境设置不同的线程池参数。
4 线程本地存储
4.1 threading.local
01.threading.local概述
a.基本概念
threading.local是Python标准库threading模块提供的一个线程本地存储机制，它允许每个线程拥有自己独立的数据副本，不同线程之间的数据互不影响。
b.工作原理
每个线程访问threading.local对象时，实际上访问的是该线程特有的数据副本，系统会自动根据当前线程ID来存储和检索相应的数据。
c.应用场景
主要用于需要在多线程环境中维护线程特定状态的情况，如数据库连接、用户会话信息、线程特定的配置参数等。
02.threading.local的基本使用
a.创建和访问local对象
a.对象创建方式
直接创建threading.local()实例，然后像普通对象一样添加属性，每个线程会看到自己独立的属性副本。
b.基本使用示例
---
# threading.local的基本使用示例
import threading
import time
# 创建线程本地存储对象
thread_local = threading.local()
def worker_function(thread_id):
"""工作线程函数"""
# 为当前线程设置本地数据
thread_local.thread_id = thread_id
thread_local.start_time = time.time()
thread_local.counter = 0
thread_name = threading.current_thread().name
print(f"{thread_name}: 初始化线程本地数据 - thread_id={thread_id}")
# 模拟工作过程，多次访问本地数据
for i in range(5):
thread_local.counter += 1
elapsed = time.time() - thread_local.start_time
print(f"{thread_name}: 计数器={thread_local.counter}, 运行时间={elapsed:.2f}s")
time.sleep(0.1)
print(f"{thread_name}: 最终计数器值={thread_local.counter}")
# 创建多个线程
threads = []
for i in range(3):
thread = threading.Thread(
target=worker_function,
args=(i,),
name=f"Worker-{i}"
)
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
print("所有线程已完成")
---
c.属性访问机制
通过点号操作符访问属性，系统会自动根据当前线程返回对应的值，每个线程看到的是自己的数据副本。
b.数据隔离验证
a.隔离性测试示例
---
# 验证线程本地存储数据隔离性的示例
import threading
import time
# 创建线程本地存储对象
shared_local = threading.local()
def demonstrate_isolation(thread_name, base_value):
"""演示数据隔离性"""
# 设置线程特定的数据
shared_local.value = base_value
shared_local.name = thread_name
shared_local.data = []
print(f"{thread_name}: 初始化 - value={shared_local.value}")
# 每个线程修改自己的数据
for i in range(5):
shared_local.value += i
shared_local.data.append(f"{thread_name}_item_{i}")
time.sleep(0.05)
print(f"{thread_name}: value={shared_local.value}, data_len={len(shared_local.data)}")
print(f"{thread_name}: 最终结果 - value={shared_local.value}, data={shared_local.data}")
def check_other_threads_data(thread_name):
"""尝试访问其他线程的数据"""
try:
print(f"{thread_name}: 尝试访问共享数据...")
print(f"{thread_name}: value={getattr(shared_local, 'value', '未设置')}")
print(f"{thread_name}: name={getattr(shared_local, 'name', '未设置')}")
print(f"{thread_name}: data_len={len(getattr(shared_local, 'data', []))}")
except AttributeError as e:
print(f"{thread_name}: 访问失败 - {e}")
# 创建线程
threads = []
thread_configs = [
("Thread-A", 100),
("Thread-B", 200),
("Thread-C", 300)
]
for name, value in thread_configs:
thread = threading.Thread(
target=demonstrate_isolation,
args=(name, value),
name=name
)
threads.append(thread)
thread.start()
# 等待主线程一段时间后检查数据
time.sleep(1)
print("\n=== 主线程检查共享本地对象 ===")
check_other_threads_data("MainThread")
# 等待所有工作线程完成
for thread in threads:
thread.join()
print("\n=== 所有线程完成后再次检查 ===")
check_other_threads_data("MainThread")
---
b.数据验证机制
通过检查不同线程设置相同属性名但值不同，验证每个线程确实拥有独立的数据副本。
03.threading.local的高级特性
a.动态属性管理
a.属性动态添加示例
---
# 动态管理线程本地存储属性的示例
import threading
import time
import random
class ThreadLocalStorage:
"""线程本地存储管理器"""
def __init__(self):
self.local_data = threading.local()
def setup_thread_data(self, thread_config):
"""为线程设置初始数据"""
# 动态添加属性
for key, value in thread_config.items():
setattr(self.local_data, key, value)
def get_thread_data(self, key, default=None):
"""获取线程本地数据"""
return getattr(self.local_data, key, default)
def update_thread_data(self, updates):
"""更新线程本地数据"""
for key, value in updates.items():
setattr(self.local_data, key, value)
def get_all_thread_data(self):
"""获取当前线程的所有本地数据"""
return {
key: value for key, value in self.local_data.__dict__.items()
if not key.startswith('_')
}
def dynamic_data_worker(thread_id, initial_config):
"""动态数据管理的工作线程"""
storage = ThreadLocalStorage()
# 设置初始配置
storage.setup_thread_data(initial_config)
thread_name = threading.current_thread().name
print(f"{thread_name}: 初始数据 - {storage.get_all_thread_data()}")
# 动态更新数据
for i in range(3):
updates = {
'iteration': i,
'random_value': random.randint(1, 100),
'timestamp': time.time()
}
storage.update_thread_data(updates)
current_data = storage.get_all_thread_data()
print(f"{thread_name}: 更新{i+1} - {current_data}")
time.sleep(0.2)
# 获取特定数据
final_random = storage.get_thread_data('random_value')
print(f"{thread_name}: 最终随机值 = {final_random}")
# 线程配置
thread_configs = [
{
'thread_type': 'processor',
'priority': 'high',
'batch_size': 50
},
{
'thread_type': 'collector',
'priority': 'medium',
'batch_size': 30
},
{
'thread_type': 'analyzer',
'priority': 'low',
'batch_size': 20
}
]
# 创建并启动线程
threads = []
for i, config in enumerate(thread_configs):
thread = threading.Thread(
target=dynamic_data_worker,
args=(i, config),
name=f"DynamicWorker-{i}"
)
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
---
b.属性生命周期管理
属性随着线程的创建而创建，随着线程的销毁而销毁，无需手动清理。
b.与类方法结合使用
a.类方法集成示例
---
# threading.local与类方法结合使用的示例
import threading
import time
class DatabaseConnection:
"""数据库连接管理类，使用线程本地存储"""
def __init__(self):
# 使用线程本地存储连接信息
self._local = threading.local()
def _get_connection_info(self):
"""获取当前线程的连接信息"""
if not hasattr(self._local, 'connection_id'):
# 为新线程创建连接
self._local.connection_id = f"conn_{threading.current_thread().ident}_{int(time.time() * 1000)}"
self._local.connection_count = 0
self._local.last_query_time = None
return self._local
def connect(self):
"""建立连接"""
conn_info = self._get_connection_info()
thread_name = threading.current_thread().name
print(f"{thread_name}: 建立连接 - {conn_info.connection_id}")
return conn_info.connection_id
def execute_query(self, query):
"""执行查询"""
conn_info = self._get_connection_info()
conn_info.connection_count += 1
conn_info.last_query_time = time.time()
thread_name = threading.current_thread().name
print(f"{thread_name}: 执行查询 #{conn_info.connection_count} - {query}")
# 模拟查询执行时间
time.sleep(0.1)
return f"Result for {query}"
def get_connection_stats(self):
"""获取连接统计信息"""
conn_info = self._get_connection_info()
return {
'connection_id': conn_info.connection_id,
'query_count': conn_info.connection_count,
'last_query_time': conn_info.last_query_time
}
def close(self):
"""关闭连接"""
conn_info = self._get_connection_info()
thread_name = threading.current_thread().name
print(f"{thread_name}: 关闭连接 - {conn_info.connection_id}")
# 清理线程本地数据
if hasattr(self._local, 'connection_id'):
delattr(self._local, 'connection_id')
if hasattr(self._local, 'connection_count'):
delattr(self._local, 'connection_count')
if hasattr(self._local, 'last_query_time'):
delattr(self._local, 'last_query_time')
def database_worker(worker_id, queries):
"""数据库工作线程"""
db = DatabaseConnection()
thread_name = threading.current_thread().name
print(f"{thread_name}: 工作器 {worker_id} 开始工作")
# 建立连接
connection_id = db.connect()
# 执行查询
results = []
for query in queries:
result = db.execute_query(query)
results.append(result)
# 显示统计信息
stats = db.get_connection_stats()
print(f"{thread_name}: 统计信息 - {stats}")
# 关闭连接
db.close()
return results
# 创建多个数据库工作线程
db_threads = []
query_sets = [
["SELECT * FROM users", "SELECT * FROM products", "SELECT COUNT(*) FROM orders"],
["UPDATE users SET status='active'", "INSERT INTO logs VALUES(...)", "DELETE FROM temp"],
["SELECT AVG(price) FROM products", "GROUP BY category", "ORDER BY date DESC"]
]
for i, queries in enumerate(query_sets):
thread = threading.Thread(
target=database_worker,
args=(i, queries),
name=f"DBWorker-{i}"
)
db_threads.append(thread)
thread.start()
# 等待所有数据库线程完成
for thread in db_threads:
thread.join()
print("所有数据库工作线程已完成")
---
b.实例方法中的使用
在实例方法中使用threading.local可以确保每个线程拥有独立的实例状态，避免线程间数据竞争。
04.threading.local的注意事项
a.内存管理考虑
a.内存泄漏风险
如果线程长期存活且不断向threading.local对象添加属性，可能导致内存使用持续增长。
b.内存管理示例
---
# 线程本地存储内存管理的示例
import threading
import time
import gc
class MemoryAwareLocalStorage:
"""内存感知的线程本地存储"""
def __init__(self):
self._local = threading.local()
self._max_attributes = 100 # 最大属性数量
def set_data(self, key, value):
"""设置数据，带内存限制检查"""
local_dict = self._local.__dict__
# 检查属性数量
if len(local_dict) >= self._max_attributes:
self._cleanup_old_data()
setattr(self._local, key, value)
def get_data(self, key, default=None):
"""获取数据"""
return getattr(self._local, key, default)
def _cleanup_old_data(self):
"""清理旧数据"""
local_dict = self._local.__dict__
# 简单的清理策略：删除一半的数据
keys_to_remove = list(local_dict.keys())[:len(local_dict)//2]
for key in keys_to_remove:
if not key.startswith('_'):
delattr(self._local, key)
print(f"{threading.current_thread().name}: 清理了 {len(keys_to_remove)} 个属性")
def get_memory_info(self):
"""获取内存使用信息"""
local_dict = self._local.__dict__
return {
'attribute_count': len([k for k in local_dict.keys() if not k.startswith('_')]),
'total_attributes': len(local_dict),
'thread_id': threading.current_thread().ident
}
def memory_test_worker(worker_id, data_count):
"""内存测试工作线程"""
storage = MemoryAwareLocalStorage()
thread_name = threading.current_thread().name
print(f"{thread_name}: 开始测试，将创建 {data_count} 个数据项")
# 创建大量数据
for i in range(data_count):
storage.set_data(f"data_{i}", f"value_{i}_{worker_id}")
if i % 50 == 0:
mem_info = storage.get_memory_info()
print(f"{thread_name}: 创建{i+1}项后 - {mem_info}")
final_info = storage.get_memory_info()
print(f"{thread_name}: 最终内存信息 - {final_info}")
# 强制垃圾回收
gc.collect()
# 创建内存测试线程
memory_threads = []
data_counts = [200, 150, 300]
for i, count in enumerate(data_counts):
thread = threading.Thread(
target=memory_test_worker,
args=(i, count),
name=f"MemTest-{i}"
)
memory_threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in memory_threads:
thread.join()
# 再次强制垃圾回收
gc.collect()
print("内存测试完成，已执行垃圾回收")
---
c.最佳实践建议
定期清理不需要的属性，避免在threading.local中存储大量数据，考虑使用弱引用。
b.线程安全注意事项
a.线程边界理解
threading.local本身是线程安全的，但存储的对象如果是可变的，仍需要额外的同步机制。
b.线程安全示例
---
# 线程本地存储与线程安全的综合示例
import threading
import time
from collections import deque
class ThreadSafeCounter:
"""线程安全的计数器"""
def __init__(self):
self._local = threading.local()
self._lock = threading.Lock()
self.global_count = 0
def _get_local_counter(self):
"""获取本地计数器"""
if not hasattr(self._local, 'counter'):
self._local.counter = 0
self._local.operations = deque(maxlen=100) # 限制操作历史长度
return self._local
def increment(self):
"""增加计数"""
local_data = self._get_local_counter()
local_data.counter += 1
# 记录操作
operation = f"{threading.current_thread().name}: increment to {local_data.counter}"
local_data.operations.append(operation)
# 更新全局计数（需要锁保护）
with self._lock:
self.global_count += 1
return local_data.counter
def get_local_count(self):
"""获取本地计数"""
return getattr(self._local, 'counter', 0)
def get_global_count(self):
"""获取全局计数"""
with self._lock:
return self.global_count
def get_operation_history(self):
"""获取操作历史"""
return list(getattr(self._local, 'operations', []))
def reset_local(self):
"""重置本地计数"""
if hasattr(self._local, 'counter'):
self._local.counter = 0
if hasattr(self._local, 'operations'):
self._local.operations.clear()
def safe_counter_worker(worker_id, operations_count):
"""线程安全计数器工作线程"""
counter = ThreadSafeCounter()
thread_name = threading.current_thread().name
print(f"{thread_name}: 开始执行 {operations_count} 次操作")
# 执行计数操作
results = []
for i in range(operations_count):
count = counter.increment()
results.append(count)
time.sleep(0.01) # 短暂延迟
# 显示统计信息
local_count = counter.get_local_count()
global_count = counter.get_global_count()
history = counter.get_operation_history()
print(f"{thread_name}: 本地计数={local_count}, 全局计数={global_count}")
print(f"{thread_name}: 操作历史（前3条）={history[:3]}")
print(f"{thread_name}: 操作历史（后3条）={history[-3:]}")
# 创建多个线程安全计数器工作线程
safe_threads = []
operation_counts = [20, 25, 15, 30]
for i, count in enumerate(operation_counts):
thread = threading.Thread(
target=safe_counter_worker,
args=(i, count),
name=f"SafeWorker-{i}"
)
safe_threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in safe_threads:
thread.join()
print("所有线程安全计数器工作线程已完成")
---
c.同步机制结合
当需要在线程间共享某些信息时，可以结合使用threading.local和传统同步机制。
4.2 线程独立数据
01.线程独立数据的概念
a.定义与特点
线程独立数据是指每个线程拥有自己独立的数据副本，修改一个线程的数据不会影响其他线程的数据，这种机制通过threading.local实现。
b.数据隔离机制
系统为每个线程维护独立的数据存储空间，通过线程ID来区分不同线程的数据，确保数据的线程隔离性。
c.与全局变量的对比
全局变量在所有线程间共享，而线程独立数据为每个线程提供独立的副本，避免了数据竞争和同步问题。
02.线程独立数据的基本应用
a.用户会话管理
a.会话管理示例
---
# 线程独立数据在用户会话管理中的应用示例
import threading
import time
import random
class UserSessionManager:
"""用户会话管理器，使用线程本地存储"""
def __init__(self):
self._local = threading.local()
def create_session(self, user_id, user_info):
"""为当前线程创建用户会话"""
self._local.user_id = user_id
self._local.user_info = user_info
self._local.session_id = f"sess_{user_id}_{int(time.time())}"
self._local.created_at = time.time()
self._local.last_activity = time.time()
self._local.operations = []
thread_name = threading.current_thread().name
print(f"{thread_name}: 创建会话 - {self._local.session_id} for user {user_id}")
return self._local.session_id
def add_operation(self, operation):
"""添加操作记录"""
if hasattr(self._local, 'operations'):
operation_record = {
'operation': operation,
'timestamp': time.time(),
'thread': threading.current_thread().name
}
self._local.operations.append(operation_record)
self._local.last_activity = time.time()
def get_session_info(self):
"""获取当前会话信息"""
if not hasattr(self._local, 'session_id'):
return None
return {
'session_id': self._local.session_id,
'user_id': self._local.user_id,
'user_info': self._local.user_info,
'created_at': self._local.created_at,
'last_activity': self._local.last_activity,
'operation_count': len(self._local.operations),
'thread': threading.current_thread().name
}
def get_recent_operations(self, count=5):
"""获取最近操作"""
if hasattr(self._local, 'operations'):
return self._local.operations[-count:]
return []
def clear_session(self):
"""清理会话"""
if hasattr(self._local, 'session_id'):
session_id = self._local.session_id
# 清理所有会话相关属性
for attr in list(self._local.__dict__.keys()):
if not attr.startswith('_'):
delattr(self._local, attr)
thread_name = threading.current_thread().name
print(f"{thread_name}: 清理会话 - {session_id}")
def simulate_user_session(user_id, user_name, operations):
"""模拟用户会话操作"""
session_manager = UserSessionManager()
# 创建用户会话
user_info = {
'name': user_name,
'email': f"{user_name}@example.com",
'role': random.choice(['user', 'admin', 'moderator'])
}
session_id = session_manager.create_session(user_id, user_info)
thread_name = threading.current_thread().name
print(f"{thread_name}: 用户 {user_name} 开始会话操作")
# 模拟用户操作
for i, operation in enumerate(operations):
session_manager.add_operation(operation)
time.sleep(random.uniform(0.1, 0.3))
if i % 2 == 0:
session_info = session_manager.get_session_info()
print(f"{thread_name}: 操作进度 {i+1}/{len(operations)} - "
f"操作数: {session_info['operation_count']}")
# 显示最终会话信息
final_info = session_manager.get_session_info()
recent_ops = session_manager.get_recent_operations(3)
print(f"{thread_name}: 会话完成 - 总操作数: {final_info['operation_count']}")
print(f"{thread_name}: 最近操作: {[op['operation'] for op in recent_ops]}")
# 清理会话
session_manager.clear_session()
return final_info
# 用户操作数据
user_configs = [
(1001, "Alice", ["login", "view_profile", "edit_settings", "logout"]),
(1002, "Bob", ["login", "browse_products", "add_to_cart", "checkout", "logout"]),
(1003, "Charlie", ["login", "create_post", "upload_image", "publish", "logout"]),
(1004, "Diana", ["login", "check_messages", "send_reply", "view_notifications", "logout"])
]
# 创建用户会话线程
session_threads = []
for user_id, user_name, operations in user_configs:
thread = threading.Thread(
target=simulate_user_session,
args=(user_id, user_name, operations),
name=f"UserSession-{user_name}"
)
session_threads.append(thread)
thread.start()
# 等待所有会话完成
for thread in session_threads:
thread.join()
print("所有用户会话已完成")
---
b.会话数据隔离
每个线程的用户会话数据完全独立，包括用户信息、操作记录等，确保了会话的安全性和隔离性。
b.数据库连接管理
a.连接池管理示例
---
# 线程独立数据在数据库连接管理中的应用示例
import threading
import time
import random
from contextlib import contextmanager
class DatabaseConnectionPool:
"""数据库连接池管理器"""
def __init__(self, max_connections_per_thread=5):
self._local = threading.local()
self.max_connections_per_thread = max_connections_per_thread
self.global_stats = {
'total_connections': 0,
'active_threads': set(),
'lock': threading.Lock()
}
def _get_thread_connections(self):
"""获取当前线程的连接列表"""
if not hasattr(self._local, 'connections'):
self._local.connections = []
self._local.connection_count = 0
self._local.thread_id = threading.current_thread().ident
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['active_threads'].add(self._local.thread_id)
return self._local.connections
@contextmanager
def get_connection(self):
"""获取数据库连接（上下文管理器）"""
connections = self._get_thread_connections()
thread_name = threading.current_thread().name
# 检查是否已达连接限制
if len(connections) >= self.max_connections_per_thread:
print(f"{thread_name}: 连接池已满，等待可用连接...")
# 简化处理：随机清理一个连接
if connections:
removed = connections.pop()
print(f"{thread_name}: 清理旧连接 {removed['id']}")
# 创建新连接
connection_id = f"conn_{threading.current_thread().ident}_{len(connections)+1}_{int(time.time()*1000)}"
connection = {
'id': connection_id,
'created_at': time.time(),
'query_count': 0,
'last_used': time.time()
}
connections.append(connection)
self._local.connection_count += 1
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['total_connections'] += 1
print(f"{thread_name}: 创建连接 {connection_id} (线程连接数: {len(connections)})")
try:
yield connection
finally:
# 连接使用结束
connection['last_used'] = time.time()
print(f"{thread_name}: 释放连接 {connection_id}")
def execute_query(self, query):
"""执行查询"""
with self.get_connection() as connection:
connection['query_count'] += 1
thread_name = threading.current_thread().name
print(f"{thread_name}: 通过连接 {connection['id']} 执行查询 #{connection['query_count']}: {query}")
# 模拟查询执行
time.sleep(random.uniform(0.05, 0.2))
return {
'connection_id': connection['id'],
'query': query,
'result': f"Result for {query}",
'execution_time': random.uniform(0.01, 0.1)
}
def get_thread_stats(self):
"""获取当前线程的统计信息"""
connections = self._get_thread_connections()
return {
'thread_id': threading.current_thread().ident,
'thread_name': threading.current_thread().name,
'connection_count': len(connections),
'total_queries': sum(conn['query_count'] for conn in connections),
'oldest_connection': min((conn['created_at'] for conn in connections), default=None)
}
def cleanup_old_connections(self, max_age_seconds=30):
"""清理旧连接"""
connections = self._get_thread_connections()
current_time = time.time()
old_connections = []
for conn in connections[:]:
if current_time - conn['created_at'] > max_age_seconds:
connections.remove(conn)
old_connections.append(conn['id'])
if old_connections:
thread_name = threading.current_thread().name
print(f"{thread_name}: 清理 {len(old_connections)} 个旧连接: {old_connections}")
def get_global_stats(self):
"""获取全局统计信息"""
with self.global_stats['lock']:
return {
'total_connections_created': self.global_stats['total_connections'],
'active_threads': len(self.global_stats['active_threads']),
'thread_ids': list(self.global_stats['active_threads'])
}
def database_worker(worker_id, query_count):
"""数据库工作线程"""
pool = DatabaseConnectionPool(max_connections_per_thread=3)
thread_name = threading.current_thread().name
print(f"{thread_name}: 数据库工作器 {worker_id} 开始执行 {query_count} 个查询")
# 执行查询
results = []
for i in range(query_count):
query = f"SELECT data_{i} FROM table_{worker_id} WHERE id = {random.randint(1, 100)}"
result = pool.execute_query(query)
results.append(result)
# 随机清理旧连接
if random.random() < 0.2:
pool.cleanup_old_connections(max_age_seconds=1)
time.sleep(random.uniform(0.1, 0.3))
# 显示统计信息
thread_stats = pool.get_thread_stats()
global_stats = pool.get_global_stats()
print(f"{thread_name}: 工作完成 - {thread_stats}")
print(f"{thread_name}: 全局统计 - {global_stats}")
return results
# 创建数据库工作线程
db_threads = []
worker_configs = [
(1, 8),
(2, 12),
(3, 6),
(4, 10)
]
for worker_id, query_count in worker_configs:
thread = threading.Thread(
target=database_worker,
args=(worker_id, query_count),
name=f"DBWorker-{worker_id}"
)
db_threads.append(thread)
thread.start()
# 等待所有工作线程完成
for thread in db_threads:
thread.join()
print("所有数据库工作线程已完成")
---
b.连接隔离机制
每个线程维护自己的数据库连接列表，避免了连接池竞争和连接状态混乱的问题。
03.线程独立数据的高级应用
a.配置管理器
a.动态配置示例
---
# 线程独立数据在配置管理中的应用示例
import threading
import time
import json
from typing import Dict, Any
class ThreadConfigManager:
"""线程配置管理器"""
def __init__(self):
self._local = threading.local()
self.global_configs = {
'default_timeout': 30,
'max_retries': 3,
'log_level': 'INFO',
'debug_mode': False
}
def load_config(self, config_file_path):
"""加载配置文件"""
thread_name = threading.current_thread().name
# 为当前线程创建独立配置
self._local.config = dict(self.global_configs)
self._local.config_file = config_file_path
self._local.loaded_at = time.time()
self._local.modifications = {}
# 模拟从文件加载配置
try:
# 这里模拟不同配置文件内容
simulated_configs = {
'config_1.json': {
'timeout': 10,
'database_url': 'sqlite:///db1.db',
'feature_flags': {'new_ui': True, 'beta_features': False}
},
'config_2.json': {
'timeout': 20,
'database_url': 'postgresql://localhost/db2',
'feature_flags': {'new_ui': False, 'beta_features': True}
},
'config_3.json': {
'timeout': 15,
'database_url': 'mysql://localhost/db3',
'feature_flags': {'new_ui': True, 'beta_features': True}
}
}
file_config = simulated_configs.get(config_file_path, {})
self._local.config.update(file_config)
print(f"{thread_name}: 加载配置文件 {config_file_path}")
print(f"{thread_name}: 配置内容 - {json.dumps(self._local.config, indent=2)}")
except Exception as e:
print(f"{thread_name}: 加载配置失败 - {e}")
# 使用默认配置
self._local.config = dict(self.global_configs)
def get_config(self, key, default=None):
"""获取配置值"""
if hasattr(self._local, 'config'):
return self._local.config.get(key, default)
return self.global_configs.get(key, default)
def set_config(self, key, value, persist=False):
"""设置配置值"""
if not hasattr(self._local, 'config'):
self.load_config('default.json')
self._local.config[key] = value
if persist:
if not hasattr(self._local, 'modifications'):
self._local.modifications = {}
self._local.modifications[key] = value
thread_name = threading.current_thread().name
print(f"{thread_name}: 设置配置 {key} = {value} ({'持久化' if persist else '临时'})")
def get_all_configs(self):
"""获取所有配置"""
if hasattr(self._local, 'config'):
return dict(self._local.config)
return dict(self.global_configs)
def get_config_info(self):
"""获取配置信息"""
if hasattr(self._local, 'config'):
return {
'config_file': getattr(self._local, 'config_file', 'default'),
'loaded_at': getattr(self._local, 'loaded_at', None),
'modification_count': len(getattr(self._local, 'modifications', {})),
'thread': threading.current_thread().name
}
return None
def reset_to_default(self):
"""重置为默认配置"""
thread_name = threading.current_thread().name
if hasattr(self._local, 'config'):
self._local.config = dict(self.global_configs)
self._local.modifications = {}
print(f"{thread_name}: 配置已重置为默认值")
def config_worker(worker_id, config_file, operations):
"""配置工作线程"""
config_manager = ThreadConfigManager()
thread_name = threading.current_thread().name
print(f"{thread_name}: 配置工作器 {worker_id} 开始，使用配置文件 {config_file}")
# 加载配置
config_manager.load_config(config_file)
# 执行配置操作
for i, operation in enumerate(operations):
op_type = operation['type']
key = operation['key']
value = operation.get('value', None)
if op_type == 'get':
result = config_manager.get_config(key, 'NOT_FOUND')
print(f"{thread_name}: 获取配置 {key} = {result}")
elif op_type == 'set':
persist = operation.get('persist', False)
config_manager.set_config(key, value, persist)
elif op_type == 'info':
info = config_manager.get_config_info()
print(f"{thread_name}: 配置信息 - {json.dumps(info, indent=2)}")
time.sleep(0.1)
# 显示最终配置
final_configs = config_manager.get_all_configs()
final_info = config_manager.get_config_info()
print(f"{thread_name}: 最终配置 - {json.dumps(final_configs, indent=2)}")
print(f"{thread_name}: 最终信息 - {json.dumps(final_info, indent=2)}")
return final_configs
# 配置操作数据
worker_configs = [
(1, 'config_1.json', [
{'type': 'get', 'key': 'timeout'},
{'type': 'get', 'key': 'database_url'},
{'type': 'set', 'key': 'custom_setting', 'value': 'worker_1_value', 'persist': True},
{'type': 'get', 'key': 'custom_setting'},
{'type': 'info'}
]),
(2, 'config_2.json', [
{'type': 'get', 'key': 'timeout'},
{'type': 'set', 'key': 'log_level', 'value': 'DEBUG'},
{'type': 'get', 'key': 'feature_flags'},
{'type': 'info'}
]),
(3, 'config_3.json', [
{'type': 'get', 'key': 'database_url'},
{'type': 'set', 'key': 'worker_id', 'value': 3},
{'type': 'reset'},
{'type': 'get', 'key': 'worker_id'},
{'type': 'info'}
])
]
# 创建配置工作线程
config_threads = []
for worker_id, config_file, operations in worker_configs:
thread = threading.Thread(
target=config_worker,
args=(worker_id, config_file, operations),
name=f"ConfigWorker-{worker_id}"
)
config_threads.append(thread)
thread.start()
# 等待所有配置工作线程完成
for thread in config_threads:
thread.join()
print("所有配置工作线程已完成")
---
b.配置继承机制
线程独立配置可以继承全局默认配置，同时允许每个线程根据需要进行个性化调整。
b.日志系统
a.线程日志管理示例
---
# 线程独立数据在日志系统中的应用示例
import threading
import time
import random
from datetime import datetime
from enum import Enum
class LogLevel(Enum):
DEBUG = "DEBUG"
INFO = "INFO"
WARNING = "WARNING"
ERROR = "ERROR"
CRITICAL = "CRITICAL"
class ThreadLogger:
"""线程日志记录器"""
def __init__(self):
self._local = threading.local()
self.global_log_buffer = []
self.global_buffer_lock = threading.Lock()
def init_logger(self, logger_name, log_level=LogLevel.INFO):
"""初始化线程日志器"""
self._local.logger_name = logger_name
self._local.log_level = log_level
self._local.log_buffer = []
self._local.log_count = 0
self._local.start_time = time.time()
self._local.thread_id = threading.current_thread().ident
thread_name = threading.current_thread().name
print(f"{thread_name}: 初始化日志器 '{logger_name}' - 级别: {log_level.value}")
def _should_log(self, level):
"""检查是否应该记录日志"""
if not hasattr(self._local, 'log_level'):
return False
level_priority = {
LogLevel.DEBUG: 0,
LogLevel.INFO: 1,
LogLevel.WARNING: 2,
LogLevel.ERROR: 3,
LogLevel.CRITICAL: 4
}
return level_priority[level.value] >= level_priority[self._local.log_level.value]
def _create_log_entry(self, level, message, extra_data=None):
"""创建日志条目"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
thread_name = threading.current_thread().name
log_entry = {
'timestamp': timestamp,
'level': level.value,
'logger': getattr(self._local, 'logger_name', 'Unknown'),
'thread': thread_name,
'thread_id': getattr(self._local, 'thread_id', 0),
'message': message,
'extra_data': extra_data or {}
}
return log_entry
def debug(self, message, **kwargs):
"""记录调试日志"""
if self._should_log(LogLevel.DEBUG):
self._log(LogLevel.DEBUG, message, kwargs)
def info(self, message, **kwargs):
"""记录信息日志"""
if self._should_log(LogLevel.INFO):
self._log(LogLevel.INFO, message, kwargs)
def warning(self, message, **kwargs):
"""记录警告日志"""
if self._should_log(LogLevel.WARNING):
self._log(LogLevel.WARNING, message, kwargs)
def error(self, message, **kwargs):
"""记录错误日志"""
if self._should_log(LogLevel.ERROR):
self._log(LogLevel.ERROR, message, kwargs)
def critical(self, message, **kwargs):
"""记录严重错误日志"""
if self._should_log(LogLevel.CRITICAL):
self._log(LogLevel.CRITICAL, message, kwargs)
def _log(self, level, message, extra_data):
"""内部日志记录方法"""
log_entry = self._create_log_entry(level, message, extra_data)
# 添加到线程本地缓冲区
if hasattr(self._local, 'log_buffer'):
self._local.log_buffer.append(log_entry)
self._local.log_count += 1
# 添加到全局缓冲区（需要锁保护）
with self.global_buffer_lock:
self.global_log_buffer.append(log_entry)
# 输出到控制台
formatted_message = f"[{log_entry['timestamp']}] [{log_entry['level']}] " \
f"[{log_entry['logger']}] [{log_entry['thread']}] {message}"
print(formatted_message)
def get_thread_logs(self, level_filter=None, limit=None):
"""获取当前线程的日志"""
if not hasattr(self._local, 'log_buffer'):
return []
logs = self._local.log_buffer
if level_filter:
logs = [log for log in logs if log['level'] == level_filter]
if limit:
logs = logs[-limit:]
return logs
def get_thread_stats(self):
"""获取线程日志统计"""
if not hasattr(self._local, 'log_buffer'):
return None
level_counts = {}
for log in self._local.log_buffer:
level = log['level']
level_counts[level] = level_counts.get(level, 0) + 1
return {
'logger_name': self._local.logger_name,
'total_logs': self._local.log_count,
'level_distribution': level_counts,
'duration': time.time() - self._local.start_time,
'thread': threading.current_thread().name
}
def flush_logs(self):
"""刷新日志缓冲区"""
if hasattr(self._local, 'log_buffer'):
thread_name = threading.current_thread().name
log_count = len(self._local.log_buffer)
self._local.log_buffer.clear()
self._local.log_count = 0
print(f"{thread_name}: 日志缓冲区已刷新，清理了 {log_count} 条日志")
def get_global_logs(self, limit=None):
"""获取全局日志"""
with self.global_buffer_lock:
logs = self.global_log_buffer
if limit:
logs = logs[-limit:]
return logs[:]
def logging_worker(worker_id, logger_name, log_level, operations):
"""日志工作线程"""
logger = ThreadLogger()
# 初始化日志器
logger.init_logger(logger_name, log_level)
thread_name = threading.current_thread().name
print(f"{thread_name}: 日志工作器 {worker_id} 开始，级别: {log_level.value}")
# 执行日志操作
for i, operation in enumerate(operations):
log_level = operation.get('level', LogLevel.INFO)
message = operation.get('message', f"Operation {i+1}")
extra_data = operation.get('extra_data', {})
# 记录日志
if log_level == LogLevel.DEBUG:
logger.debug(message, **extra_data)
elif log_level == LogLevel.INFO:
logger.info(message, **extra_data)
elif log_level == LogLevel.WARNING:
logger.warning(message, **extra_data)
elif log_level == LogLevel.ERROR:
logger.error(message, **extra_data)
elif log_level == LogLevel.CRITICAL:
logger.critical(message, **extra_data)
time.sleep(random.uniform(0.05, 0.2))
# 随机刷新日志
if random.random() < 0.3:
logger.flush_logs()
# 显示统计信息
stats = logger.get_thread_stats()
thread_logs = logger.get_thread_logs(limit=5)
print(f"{thread_name}: 日志统计 - {json.dumps(stats, indent=2, default=str)}")
print(f"{thread_name}: 最近日志 - {[log['message'] for log in thread_logs]}")
return stats
import json
# 日志操作配置
worker_configs = [
(1, "AuthService", LogLevel.DEBUG, [
{'level': LogLevel.INFO, 'message': "用户认证服务启动"},
{'level': LogLevel.DEBUG, 'message': "检查用户凭据", 'extra_data': {'user_id': 12345}},
{'level': LogLevel.INFO, 'message': "用户认证成功", 'extra_data': {'user_id': 12345, 'duration': 0.15}},
{'level': LogLevel.WARNING, 'message': "检测到异常登录尝试", 'extra_data': {'ip': '192.168.1.100'}},
{'level': LogLevel.INFO, 'message': "认证服务运行正常"}
]),
(2, "DataProcessor", LogLevel.INFO, [
{'level': LogLevel.INFO, 'message': "数据处理任务开始"},
{'level': LogLevel.INFO, 'message': "处理批次 1/5", 'extra_data': {'records': 1000}},
{'level': LogLevel.INFO, 'message': "处理批次 2/5", 'extra_data': {'records': 1200}},
{'level': LogLevel.ERROR, 'message': "数据验证失败", 'extra_data': {'record_id': 5678, 'error': 'Invalid format'}},
{'level': LogLevel.INFO, 'message': "数据处理完成", 'extra_data': {'total_records': 5000, 'errors': 1}}
]),
(3, "NetworkMonitor", LogLevel.WARNING, [
{'level': LogLevel.WARNING, 'message': "网络延迟增加", 'extra_data': {'latency_ms': 150}},
{'level': LogLevel.ERROR, 'message': "连接丢失", 'extra_data': {'server': 'api.example.com', 'attempts': 3}},
{'level': LogLevel.CRITICAL, 'message': "服务不可用", 'extra_data': {'downtime_seconds': 30}},
{'level': LogLevel.INFO, 'message': "连接已恢复", 'extra_data': {'recovery_time': 12.5}}
])
]
# 创建日志工作线程
logging_threads = []
for worker_id, logger_name, log_level, operations in worker_configs:
thread = threading.Thread(
target=logging_worker,
args=(worker_id, logger_name, log_level, operations),
name=f"LoggerWorker-{worker_id}"
)
logging_threads.append(thread)
thread.start()
# 等待所有日志线程完成
for thread in logging_threads:
thread.join()
print("\n=== 全局日志摘要 ===")
global_logger = ThreadLogger()
global_logs = global_logger.get_global_logs(limit=10)
for log in global_logs:
print(f"[{log['timestamp']}] [{log['level']}] [{log['logger']}] {log['message']}")
print("所有日志工作线程已完成")
---
b.日志缓冲管理
每个线程维护独立的日志缓冲区，避免日志写入竞争，同时支持批量刷新和异步处理。
4.3 应用场景
01.Web应用中的用户会话管理
a.请求处理与会话隔离
a.Web会话管理示例
---
# Web应用中线程本地存储用于会话管理的示例
import threading
import time
import random
from typing import Dict, Any, Optional
from contextlib import contextmanager
class WebSessionManager:
"""Web会话管理器，使用线程本地存储隔离用户会话"""
def __init__(self):
self._local = threading.local()
self.global_sessions = {} # 全局会话存储
self.session_lock = threading.Lock()
@contextmanager
def request_context(self, request_id, user_id=None):
"""请求上下文管理器"""
thread_name = threading.current_thread().name
# 设置当前线程的请求上下文
self._local.request_id = request_id
self._local.user_id = user_id
self._local.session_id = None
self._local.start_time = time.time()
self._local.request_data = {}
self._local.response_data = {}
self._local.log_entries = []
print(f"{thread_name}: 开始处理请求 {request_id} for user {user_id}")
try:
# 如果有用户ID，加载或创建会话
if user_id:
self._local.session_id = self._get_or_create_session(user_id)
self._load_session_data()
yield self
finally:
# 保存会话数据
if hasattr(self._local, 'session_id') and self._local.session_id:
self._save_session_data()
# 清理线程本地数据
duration = time.time() - self._local.start_time
print(f"{thread_name}: 请求 {request_id} 处理完成，耗时 {duration:.3f}s")
self._cleanup_request_context()
def _get_or_create_session(self, user_id):
"""获取或创建用户会话"""
with self.session_lock:
session_key = f"session_{user_id}"
if session_key not in self.global_sessions:
self.global_sessions[session_key] = {
'user_id': user_id,
'created_at': time.time(),
'last_access': time.time(),
'data': {},
'request_count': 0
}
session = self.global_sessions[session_key]
session['last_access'] = time.time()
session['request_count'] += 1
return session_key
def _load_session_data(self):
"""加载会话数据到线程本地存储"""
if self._local.session_id:
session = self.global_sessions.get(self._local.session_id, {})
self._local.session_data = session.get('data', {}).copy()
self._local.session_created_at = session.get('created_at')
self._local.request_count = session.get('request_count', 0)
def _save_session_data(self):
"""保存线程本地数据到会话"""
if self._local.session_id and hasattr(self._local, 'session_data'):
with self.session_lock:
session = self.global_sessions.get(self._local.session_id, {})
session['data'] = self._local.session_data
session['last_access'] = time.time()
def _cleanup_request_context(self):
"""清理请求上下文"""
attributes_to_remove = [
'request_id', 'user_id', 'session_id', 'start_time',
'request_data', 'response_data', 'log_entries',
'session_data', 'session_created_at', 'request_count'
]
for attr in attributes_to_remove:
if hasattr(self._local, attr):
delattr(self._local, attr)
def get_current_user_id(self):
"""获取当前用户ID"""
return getattr(self._local, 'user_id', None)
def get_session_data(self, key, default=None):
"""获取会话数据"""
if hasattr(self._local, 'session_data'):
return self._local.session_data.get(key, default)
return default
def set_session_data(self, key, value):
"""设置会话数据"""
if not hasattr(self._local, 'session_data'):
self._local.session_data = {}
self._local.session_data[key] = value
thread_name = threading.current_thread().name
print(f"{thread_name}: 设置会话数据 {key} = {value}")
def add_request_data(self, key, value):
"""添加请求数据"""
if not hasattr(self._local, 'request_data'):
self._local.request_data = {}
self._local.request_data[key] = value
def add_response_data(self, key, value):
"""添加响应数据"""
if not hasattr(self._local, 'response_data'):
self._local.response_data = {}
self._local.response_data[key] = value
def log(self, level, message):
"""记录请求日志"""
if not hasattr(self._local, 'log_entries'):
self._local.log_entries = []
log_entry = {
'timestamp': time.time(),
'level': level,
'message': message,
'request_id': getattr(self._local, 'request_id', 'unknown')
}
self._local.log_entries.append(log_entry)
thread_name = threading.current_thread().name
print(f"{thread_name} [{level}] {message}")
def get_request_summary(self):
"""获取请求摘要"""
return {
'request_id': getattr(self._local, 'request_id', 'unknown'),
'user_id': getattr(self._local, 'user_id', None),
'session_id': getattr(self._local, 'session_id', None),
'duration': time.time() - getattr(self._local, 'start_time', time.time()),
'request_data_keys': list(getattr(self._local, 'request_data', {}).keys()),
'response_data_keys': list(getattr(self._local, 'response_data', {}).keys()),
'log_count': len(getattr(self._local, 'log_entries', [])),
'session_data_count': len(getattr(self._local, 'session_data', {}))
}
def simulate_web_request(session_manager, request_id, user_id, operations):
"""模拟Web请求处理"""
with session_manager.request_context(request_id, user_id) as ctx:
thread_name = threading.current_thread().name
# 模拟请求处理操作
for i, operation in enumerate(operations):
op_type = operation.get('type')
if op_type == 'set_session':
key = operation.get('key')
value = operation.get('value')
ctx.set_session_data(key, value)
elif op_type == 'get_session':
key = operation.get('key')
value = ctx.get_session_data(key, 'NOT_FOUND')
ctx.log('INFO', f"获取会话数据 {key} = {value}")
elif op_type == 'add_request':
ctx.add_request_data('operation', f"step_{i+1}")
ctx.add_request_data('data', operation.get('data'))
elif op_type == 'add_response':
ctx.add_response_data('result', f"processed_{i+1}")
ctx.add_response_data('status', 'success')
elif op_type == 'log':
level = operation.get('level', 'INFO')
message = operation.get('message')
ctx.log(level, message)
# 模拟处理时间
time.sleep(random.uniform(0.05, 0.15))
# 获取请求摘要
summary = ctx.get_request_summary()
print(f"{thread_name}: 请求摘要 - {summary}")
return summary
# Web服务器模拟
def web_server_worker(worker_id, requests):
"""Web服务器工作线程"""
session_manager = WebSessionManager()
thread_name = threading.current_thread().name
print(f"{thread_name}: Web工作器 {worker_id} 开始处理 {len(requests)} 个请求")
results = []
for i, request_config in enumerate(requests):
request_id = request_config['request_id']
user_id = request_config['user_id']
operations = request_config['operations']
print(f"{thread_name}: 处理请求 {i+1}/{len(requests)}")
summary = simulate_web_request(session_manager, request_id, user_id, operations)
results.append(summary)
# 模拟请求间隔
time.sleep(random.uniform(0.1, 0.3))
return results
# 请求数据配置
request_batches = [
[
{'request_id': 'req_001', 'user_id': 1001, 'operations': [
{'type': 'log', 'level': 'INFO', 'message': '用户登录'},
{'type': 'set_session', 'key': 'cart_items', 'value': []},
{'type': 'set_session', 'key': 'preferences', 'value': {'theme': 'dark'}},
{'type': 'add_request', 'data': {'action': 'view_home'}},
{'type': 'add_response', 'status': 'success'}
]},
{'request_id': 'req_002', 'user_id': 1001, 'operations': [
{'type': 'get_session', 'key': 'cart_items'},
{'type': 'set_session', 'key': 'cart_items', 'value': ['item_1', 'item_2']},
{'type': 'add_request', 'data': {'action': 'add_to_cart'}},
{'type': 'log', 'level': 'INFO', 'message': '商品已添加到购物车'}
]}
],
[
{'request_id': 'req_003', 'user_id': 1002, 'operations': [
{'type': 'log', 'level': 'INFO', 'message': '用户登录'},
{'type': 'set_session', 'key': 'search_history', 'value': []},
{'type': 'add_request', 'data': {'action': 'search', 'query': 'laptop'}},
{'type': 'set_session', 'key': 'search_history', 'value': ['laptop']},
{'type': 'add_response', 'status': 'success'}
]},
{'request_id': 'req_004', 'user_id': 1002, 'operations': [
{'type': 'get_session', 'key': 'search_history'},
{'type': 'add_request', 'data': {'action': 'filter', 'category': 'electronics'}},
{'type': 'log', 'level': 'DEBUG', 'message': '应用筛选条件'}
]}
],
[
{'request_id': 'req_005', 'user_id': 1001, 'operations': [
{'type': 'get_session', 'key': 'cart_items'},
{'type': 'add_request', 'data': {'action': 'checkout'}},
{'type': 'log', 'level': 'INFO', 'message': '开始结账流程'},
{'type': 'add_response', 'status': 'processing'}
]}
]
]
# 创建Web服务器工作线程
web_threads = []
for i, requests in enumerate(request_batches):
thread = threading.Thread(
target=web_server_worker,
args=(i+1, requests),
name=f"WebWorker-{i+1}"
)
web_threads.append(thread)
thread.start()
# 等待所有Web线程完成
for thread in web_threads:
thread.join()
print("所有Web服务器工作线程已完成")
print("=== 全局会话状态 ===")
session_manager = WebSessionManager()
for session_key, session_data in session_manager.global_sessions.items():
print(f"{session_key}: 用户 {session_data['user_id']}, "
f"请求数 {session_data['request_count']}, "
f"数据项 {len(session_data['data'])}")
---
b.并发请求处理优势
线程本地存储确保每个请求的会话数据完全隔离，即使同一用户的多个并发请求也不会相互干扰。
02.数据库连接池管理
a.连接隔离与资源管理
a.数据库连接池示例
---
# 数据库连接池的线程本地存储管理示例
import threading
import time
import random
from contextlib import contextmanager
from typing import List, Dict, Any
class ThreadSafeConnectionPool:
"""线程安全的数据库连接池"""
def __init__(self, max_connections_per_thread=10, connection_timeout=30):
self._local = threading.local()
self.max_connections_per_thread = max_connections_per_thread
self.connection_timeout = connection_timeout
# 全局统计信息
self.global_stats = {
'total_connections_created': 0,
'total_queries_executed': 0,
'active_threads': set(),
'lock': threading.Lock()
}
def _get_thread_pool(self) -> List[Dict[str, Any]]:
"""获取当前线程的连接池"""
if not hasattr(self._local, 'connections'):
self._local.connections = []
self._local.thread_id = threading.current_thread().ident
self._local.connection_stats = {
'created': 0,
'queries': 0,
'errors': 0
}
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['active_threads'].add(self._local.thread_id)
return self._local.connections
@contextmanager
def get_connection(self, database_name="default"):
"""获取数据库连接（上下文管理器）"""
connections = self._get_thread_pool()
thread_name = threading.current_thread().name
# 检查是否已有可用连接
available_connection = None
for conn in connections:
if (conn['database'] == database_name and
not conn['in_use'] and
time.time() - conn['last_used'] < self.connection_timeout):
available_connection = conn
break
# 如果没有可用连接，创建新连接
if not available_connection:
if len(connections) >= self.max_connections_per_thread:
# 清理过期连接
self._cleanup_expired_connections()
# 如果仍然没有空间，等待或抛出异常
if len(connections) >= self.max_connections_per_thread:
print(f"{thread_name}: 连接池已满，无法创建新连接")
raise Exception("Connection pool exhausted")
# 创建新连接
connection_id = f"conn_{self._local.thread_id}_{len(connections)+1}_{int(time.time()*1000)}"
available_connection = {
'id': connection_id,
'database': database_name,
'created_at': time.time(),
'last_used': time.time(),
'in_use': False,
'query_count': 0,
'error_count': 0,
'transaction_active': False
}
connections.append(available_connection)
self._local.connection_stats['created'] += 1
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['total_connections_created'] += 1
print(f"{thread_name}: 创建新连接 {connection_id} 到数据库 {database_name}")
# 标记连接为使用中
available_connection['in_use'] = True
available_connection['last_used'] = time.time()
try:
yield available_connection
finally:
# 释放连接
available_connection['in_use'] = False
available_connection['last_used'] = time.time()
print(f"{thread_name}: 释放连接 {available_connection['id']}")
def execute_query(self, query: str, database: str = "default", params=None):
"""执行数据库查询"""
with self.get_connection(database) as connection:
thread_name = threading.current_thread().name
connection['query_count'] += 1
self._local.connection_stats['queries'] += 1
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['total_queries_executed'] += 1
print(f"{thread_name}: 通过连接 {connection['id']} 执行查询: {query}")
# 模拟查询执行
execution_time = random.uniform(0.01, 0.2)
time.sleep(execution_time)
# 模拟偶发错误
if random.random() < 0.05:
connection['error_count'] += 1
self._local.connection_stats['errors'] += 1
raise Exception(f"Query execution failed: {query}")
return {
'connection_id': connection['id'],
'query': query,
'result': f"Result set for {query}",
'execution_time': execution_time,
'rows_affected': random.randint(0, 100)
}
@contextmanager
def transaction(self, database: str = "default"):
"""数据库事务管理"""
with self.get_connection(database) as connection:
if connection['transaction_active']:
raise Exception("Nested transactions not supported")
connection['transaction_active'] = True
thread_name = threading.current_thread().name
print(f"{thread_name}: 开始事务 on connection {connection['id']}")
try:
yield connection
# 提交事务
print(f"{thread_name}: 提交事务 on connection {connection['id']}")
except Exception as e:
# 回滚事务
connection['error_count'] += 1
print(f"{thread_name}: 回滚事务 on connection {connection['id']}: {e}")
raise
finally:
connection['transaction_active'] = False
def _cleanup_expired_connections(self):
"""清理过期连接"""
connections = self._get_thread_pool()
current_time = time.time()
expired_connections = []
for conn in connections[:]:
if (not conn['in_use'] and
current_time - conn['last_used'] > self.connection_timeout):
connections.remove(conn)
expired_connections.append(conn['id'])
if expired_connections:
thread_name = threading.current_thread().name
print(f"{thread_name}: 清理 {len(expired_connections)} 个过期连接: {expired_connections}")
def get_thread_stats(self) -> Dict[str, Any]:
"""获取当前线程的连接统计"""
connections = self._get_thread_pool()
active_connections = sum(1 for conn in connections if conn['in_use'])
total_queries = sum(conn['query_count'] for conn in connections)
total_errors = sum(conn['error_count'] for conn in connections)
return {
'thread_id': self._local.thread_id,
'thread_name': threading.current_thread().name,
'total_connections': len(connections),
'active_connections': active_connections,
'idle_connections': len(connections) - active_connections,
'total_queries': total_queries,
'total_errors': total_errors,
'connection_stats': self._local.connection_stats,
'databases': list(set(conn['database'] for conn in connections))
}
def get_global_stats(self) -> Dict[str, Any]:
"""获取全局统计信息"""
with self.global_stats['lock']:
return {
'total_connections_created': self.global_stats['total_connections_created'],
'total_queries_executed': self.global_stats['total_queries_executed'],
'active_threads_count': len(self.global_stats['active_threads']),
'active_thread_ids': list(self.global_stats['active_threads'])
}
def cleanup_thread_connections(self):
"""清理当前线程的所有连接"""
if hasattr(self._local, 'connections'):
thread_name = threading.current_thread().name
connection_count = len(self._local.connections)
# 确保没有活动事务
for conn in self._local.connections:
if conn['transaction_active']:
print(f"{thread_name}: 强制回滚连接 {conn['id']} 的事务")
conn['transaction_active'] = False
self._local.connections.clear()
print(f"{thread_name}: 清理了 {connection_count} 个连接")
# 更新全局统计
with self.global_stats['lock']:
self.global_stats['active_threads'].discard(self._local.thread_id)
def database_worker(worker_id, tasks):
"""数据库工作线程"""
pool = ThreadSafeConnectionPool(max_connections_per_thread=5)
thread_name = threading.current_thread().name
print(f"{thread_name}: 数据库工作器 {worker_id} 开始处理 {len(tasks)} 个任务")
results = []
for i, task in enumerate(tasks):
task_type = task.get('type', 'query')
database = task.get('database', 'default')
try:
if task_type == 'query':
query = task.get('query', f"SELECT * FROM table_{i}")
result = pool.execute_query(query, database)
results.append(result)
elif task_type == 'transaction':
operations = task.get('operations', [])
with pool.transaction(database) as conn:
for op in operations:
op_query = op.get('query', f"UPDATE table SET value = {random.randint(1, 100)}")
result = pool.execute_query(op_query, database)
results.append(result)
elif task_type == 'mixed':
# 混合操作：一些查询，一些事务
if random.random() < 0.7:
query = task.get('query', f"SELECT * FROM mixed_table_{i}")
result = pool.execute_query(query, database)
else:
with pool.transaction(database) as conn:
for j in range(random.randint(1, 3)):
query = f"INSERT INTO log VALUES ('{thread_name}_{i}_{j}')"
result = pool.execute_query(query, database)
results.append(result)
print(f"{thread_name}: 完成任务 {i+1}/{len(tasks)}")
except Exception as e:
print(f"{thread_name}: 任务 {i+1} 失败: {e}")
continue
# 随机延迟
time.sleep(random.uniform(0.05, 0.2))
# 显示统计信息
thread_stats = pool.get_thread_stats()
global_stats = pool.get_global_stats()
print(f"{thread_name}: 工作完成统计")
print(f" 线程统计: {thread_stats}")
print(f" 全局统计: {global_stats}")
# 清理连接
pool.cleanup_thread_connections()
return results
# 数据库任务配置
worker_tasks = [
[
{'type': 'query', 'database': 'user_db', 'query': 'SELECT * FROM users WHERE active = 1'},
{'type': 'transaction', 'database': 'user_db', 'operations': [
{'query': 'UPDATE users SET last_login = NOW() WHERE id = 1'},
{'query': 'INSERT INTO login_log VALUES (1, NOW())'}
]},
{'type': 'query', 'database': 'product_db', 'query': 'SELECT * FROM products LIMIT 10'},
{'type': 'mixed', 'database': 'order_db', 'query': 'SELECT * FROM orders WHERE status = "pending"'},
{'type': 'transaction', 'database': 'order_db', 'operations': [
{'query': 'UPDATE orders SET status = "processing" WHERE id = 123'},
{'query': 'INSERT INTO audit_log VALUES ("order_123", "status_change", NOW())'}
]}
],
[
{'type': 'query', 'database': 'analytics_db', 'query': 'SELECT COUNT(*) FROM page_views'},
{'type': 'mixed', 'database': 'analytics_db', 'query': 'SELECT * FROM user_sessions WHERE date > TODAY'},
{'type': 'query', 'database': 'cache_db', 'query': 'SELECT * FROM cache_keys WHERE expires > NOW()'},
{'type': 'transaction', 'database': 'cache_db', 'operations': [
{'query': 'DELETE FROM cache_data WHERE expires < NOW()'},
{'query': 'INSERT INTO cleanup_log VALUES (NOW(), "cache_cleanup")'}
]}
],
[
{'type': 'query', 'database': 'config_db', 'query': 'SELECT * FROM system_config'},
{'type': 'mixed', 'database': 'log_db', 'query': 'SELECT * FROM error_logs WHERE created_at > YESTERDAY'},
{'type': 'transaction', 'database': 'log_db', 'operations': [
{'query': 'INSERT INTO archive_logs SELECT * FROM access_logs WHERE created_at < LAST_MONTH'},
{'query': 'DELETE FROM access_logs WHERE created_at < LAST_MONTH'}
]},
{'type': 'query', 'database': 'config_db', 'query': 'SELECT * FROM feature_flags'},
{'type': 'mixed', 'database': 'config_db', 'query': 'UPDATE system_config SET updated_at = NOW() WHERE active = 1'}
]
]
# 创建数据库工作线程
db_threads = []
for i, tasks in enumerate(worker_tasks):
thread = threading.Thread(
target=database_worker,
args=(i+1, tasks),
name=f"DBWorker-{i+1}"
)
db_threads.append(thread)
thread.start()
# 等待所有数据库线程完成
for thread in db_threads:
thread.join()
print("\n=== 最终全局统计 ===")
final_pool = ThreadSafeConnectionPool()
print(f"全局统计: {final_pool.get_global_stats()}")
---
b.连接生命周期管理
每个线程维护独立的连接池，自动管理连接的创建、使用、过期清理和销毁，避免线程间连接冲突。
03.多租户系统中的租户隔离
a.租户数据隔离策略
a.多租户隔离示例
---
# 多租户系统中使用线程本地存储实现租户隔离的示例
import threading
import time
import random
from typing import Dict, Any, Optional
from contextlib import contextmanager
from enum import Enum
class TenantStatus(Enum):
ACTIVE = "active"
SUSPENDED = "suspended"
TRIAL = "trial"
class TenantContext:
"""租户上下文管理器"""
def __init__(self):
self._local = threading.local()
self.tenant_configs = {} # 全局租户配置
self.config_lock = threading.Lock()
@contextmanager
def tenant_scope(self, tenant_id: str, user_id: str = None):
"""租户作用域上下文管理器"""
thread_name = threading.current_thread().name
# 设置当前线程的租户上下文
self._local.tenant_id = tenant_id
self._local.user_id = user_id
self._local.start_time = time.time()
self._local.operation_count = 0
self._local.operation_history = []
print(f"{thread_name}: 进入租户 {tenant_id} 的上下文 (用户: {user_id})")
try:
# 加载租户配置
self._load_tenant_config(tenant_id)
yield self
finally:
# 清理租户上下文
duration = time.time() - self._local.start_time
print(f"{thread_name}: 离开租户 {tenant_id} 上下文，耗时 {duration:.3f}s，操作 {self._local.operation_count} 次")
self._cleanup_tenant_context()
def _load_tenant_config(self, tenant_id: str):
"""加载租户配置"""
with self.config_lock:
if tenant_id not in self.tenant_configs:
# 模拟租户配置
self.tenant_configs[tenant_id] = {
'tenant_id': tenant_id,
'database_name': f'tenant_{tenant_id}_db',
'storage_bucket': f'tenant-{tenant_id}-storage',
'max_users': random.randint(100, 1000),
'status': random.choice(list(TenantStatus)),
'features': {
'advanced_analytics': random.choice([True, False]),
'custom_branding': random.choice([True, False]),
'api_access': random.choice([True, False])
},
'rate_limits': {
'api_requests_per_minute': random.randint(100, 1000),
'file_upload_size_mb': random.randint(10, 100)
}
}
self._local.tenant_config = self.tenant_configs[tenant_id].copy()
def _cleanup_tenant_context(self):
"""清理租户上下文"""
attributes_to_remove = [
'tenant_id', 'user_id', 'start_time', 'operation_count',
'operation_history', 'tenant_config', 'current_database'
]
for attr in attributes_to_remove:
if hasattr(self._local, attr):
delattr(self._local, attr)
def get_current_tenant_id(self) -> Optional[str]:
"""获取当前租户ID"""
return getattr(self._local, 'tenant_id', None)
def get_current_user_id(self) -> Optional[str]:
"""获取当前用户ID"""
return getattr(self._local, 'user_id', None)
def get_tenant_config(self, key: str, default=None):
"""获取租户配置"""
if hasattr(self._local, 'tenant_config'):
return self._local.tenant_config.get(key, default)
return default
def has_feature(self, feature: str) -> bool:
"""检查租户是否具有指定功能"""
features = self.get_tenant_config('features', {})
return features.get(feature, False)
def check_rate_limit(self, limit_type: str, increment: int = 1) -> bool:
"""检查速率限制"""
rate_limits = self.get_tenant_config('rate_limits', {})
limit_key = f'{limit_type}_used'
if not hasattr(self._local, 'rate_limit_usage'):
self._local.rate_limit_usage = {}
current_usage = self._local.rate_limit_usage.get(limit_key, 0)
max_limit = rate_limits.get(limit_type, float('inf'))
if current_usage + increment <= max_limit:
self._local.rate_limit_usage[limit_key] = current_usage + increment
return True
else:
return False
def get_tenant_database(self) -> str:
"""获取租户数据库名"""
return self.get_tenant_config('database_name', 'default_db')
def log_operation(self, operation: str, details: Dict[str, Any] = None):
"""记录操作日志"""
if not hasattr(self._local, 'operation_history'):
self._local.operation_history = []
log_entry = {
'timestamp': time.time(),
'operation': operation,
'details': details or {},
'tenant_id': self.get_current_tenant_id(),
'user_id': self.get_current_user_id()
}
self._local.operation_history.append(log_entry)
self._local.operation_count += 1
thread_name = threading.current_thread().name
print(f"{thread_name}: 记录操作 - {operation} (租户: {self.get_current_tenant_id()})")
def get_operation_history(self, limit: int = 10):
"""获取操作历史"""
if hasattr(self._local, 'operation_history'):
return self._local.operation_history[-limit:]
return []
def get_tenant_summary(self) -> Dict[str, Any]:
"""获取租户会话摘要"""
return {
'tenant_id': self.get_current_tenant_id(),
'user_id': self.get_current_user_id(),
'duration': time.time() - getattr(self._local, 'start_time', time.time()),
'operation_count': getattr(self._local, 'operation_count', 0),
'database': self.get_tenant_database(),
'status': self.get_tenant_config('status'),
'features_enabled': list(self.get_tenant_config('features', {}).keys()),
'rate_limit_usage': getattr(self._local, 'rate_limit_usage', {})
}
class TenantDataService:
"""租户数据服务"""
def __init__(self, tenant_context: TenantContext):
self.tenant_context = tenant_context
def create_user(self, user_data: Dict[str, Any]) -> Dict[str, Any]:
"""创建用户（租户隔离）"""
if not self.tenant_context.check_rate_limit('user_creations'):
raise Exception("Rate limit exceeded for user creations")
database = self.tenant_context.get_tenant_database()
tenant_id = self.tenant_context.get_current_tenant_id()
self.tenant_context.log_operation('create_user', {
'database': database,
'user_data_keys': list(user_data.keys())
})
# 模拟创建用户
user_id = f"user_{tenant_id}_{int(time.time()*1000)}"
created_user = {
'user_id': user_id,
'tenant_id': tenant_id,
'database': database,
'created_at': time.time(),
**user_data
}
return created_user
def query_data(self, table: str, filters: Dict[str, Any] = None) -> List[Dict[str, Any]]:
"""查询数据（租户隔离）"""
if not self.tenant_context.check_rate_limit('api_requests_per_minute'):
raise Exception("API rate limit exceeded")
database = self.tenant_context.get_tenant_database()
tenant_id = self.tenant_context.get_current_tenant_id()
self.tenant_context.log_operation('query_data', {
'table': table,
'database': database,
'filters': filters or {}
})
# 模拟查询数据
result_count = random.randint(1, 50)
results = []
for i in range(result_count):
result = {
'id': f"{table}_{tenant_id}_{i}",
'tenant_id': tenant_id,
'database': database,
'table': table,
'data': f"sample_data_{i}"
}
if filters:
result.update(filters)
results.append(result)
return results
def upload_file(self, filename: str, size_mb: int) -> Dict[str, Any]:
"""上传文件（租户隔离）"""
if not self.tenant_context.check_rate_limit('file_upload_size_mb', size_mb):
raise Exception("File upload size limit exceeded")
storage_bucket = self.tenant_context.get_tenant_config('storage_bucket')
tenant_id = self.tenant_context.get_current_tenant_id()
self.tenant_context.log_operation('upload_file', {
'filename': filename,
'size_mb': size_mb,
'storage_bucket': storage_bucket
})
# 模拟文件上传
file_id = f"file_{tenant_id}_{int(time.time()*1000)}"
uploaded_file = {
'file_id': file_id,
'tenant_id': tenant_id,
'filename': filename,
'size_mb': size_mb,
'storage_bucket': storage_bucket,
'upload_path': f"{storage_bucket}/{tenant_id}/{filename}",
'uploaded_at': time.time()
}
return uploaded_file
def get_analytics(self, metric: str, period: str = 'daily') -> Dict[str, Any]:
"""获取分析数据（需要功能支持）"""
if not self.tenant_context.has_feature('advanced_analytics'):
raise Exception("Advanced analytics feature not available for this tenant")
tenant_id = self.tenant_context.get_current_tenant_id()
self.tenant_context.log_operation('get_analytics', {
'metric': metric,
'period': period
})
# 模拟分析数据
return {
'tenant_id': tenant_id,
'metric': metric,
'period': period,
'value': random.randint(1000, 10000),
'change_percent': random.uniform(-20, 50),
'generated_at': time.time()
}
def tenant_worker(worker_id: int, operations: List[Dict[str, Any]]):
"""租户工作线程"""
tenant_context = TenantContext()
data_service = TenantDataService(tenant_context)
thread_name = threading.current_thread().name
print(f"{thread_name}: 租户工作器 {worker_id} 开始处理 {len(operations)} 个操作")
results = []
for i, operation in enumerate(operations):
op_type = operation.get('type')
tenant_id = operation.get('tenant_id', f'tenant_{worker_id}')
user_id = operation.get('user_id', f'user_{worker_id}_{i}')
try:
with tenant_context.tenant_scope(tenant_id, user_id):
if op_type == 'create_user':
user_data = operation.get('user_data', {})
result = data_service.create_user(user_data)
elif op_type == 'query_data':
table = operation.get('table', 'default_table')
filters = operation.get('filters')
result = data_service.query_data(table, filters)
elif op_type == 'upload_file':
filename = operation.get('filename', f'file_{i}.txt')
size_mb = operation.get('size_mb', random.randint(1, 50))
result = data_service.upload_file(filename, size_mb)
elif op_type == 'get_analytics':
metric = operation.get('metric', 'page_views')
period = operation.get('period', 'daily')
result = data_service.get_analytics(metric, period)
else:
raise Exception(f"Unknown operation type: {op_type}")
results.append(result)
print(f"{thread_name}: 完成操作 {i+1}/{len(operations)}: {op_type}")
except Exception as e:
print(f"{thread_name}: 操作 {i+1} 失败: {e}")
continue
# 模拟操作间隔
time.sleep(random.uniform(0.05, 0.15))
return results
# 租户操作配置
tenant_operations = [
[
{'type': 'create_user', 'tenant_id': 'tenant_001', 'user_id': 'admin_001',
'user_data': {'name': 'Alice Admin', 'role': 'admin', 'email': '[email protected]'}},
{'type': 'query_data', 'tenant_id': 'tenant_001', 'table': 'products',
'filters': {'category': 'electronics'}},
{'type': 'upload_file', 'tenant_id': 'tenant_001', 'filename': 'product_catalog.pdf', 'size_mb': 25},
{'type': 'get_analytics', 'tenant_id': 'tenant_001', 'metric': 'user_registrations', 'period': 'weekly'}
],
[
{'type': 'create_user', 'tenant_id': 'tenant_002', 'user_id': 'user_002',
'user_data': {'name': 'Bob User', 'role': 'user', 'email': '[email protected]'}},
{'type': 'query_data', 'tenant_id': 'tenant_002', 'table': 'orders'},
{'type': 'upload_file', 'tenant_id': 'tenant_002', 'filename': 'order_report.xlsx', 'size_mb': 15},
{'type': 'create_user', 'tenant_id': 'tenant_002', 'user_id': 'user_003',
'user_data': {'name': 'Charlie User', 'role': 'user', 'email': '[email protected]'}}
],
[
{'type': 'query_data', 'tenant_id': 'tenant_003', 'table': 'analytics',
'filters': {'date_range': 'last_30_days'}},
{'type': 'upload_file', 'tenant_id': 'tenant_003', 'filename': 'analytics_export.csv', 'size_mb': 8},
{'type': 'get_analytics', 'tenant_id': 'tenant_003', 'metric': 'revenue', 'period': 'monthly'},
{'type': 'query_data', 'tenant_id': 'tenant_003', 'table': 'customers'},
{'type': 'create_user', 'tenant_id': 'tenant_003', 'user_id': 'manager_003',
'user_data': {'name': 'Diana Manager', 'role': 'manager', 'email': '[email protected]'}}
]
]
# 创建租户工作线程
tenant_threads = []
for i, operations in enumerate(tenant_operations):
thread = threading.Thread(
target=tenant_worker,
args=(i+1, operations),
name=f"TenantWorker-{i+1}"
)
tenant_threads.append(thread)
thread.start()
# 等待所有租户线程完成
for thread in tenant_threads:
thread.join()
print("\n=== 全局租户配置摘要 ===")
context = TenantContext()
for tenant_id, config in context.tenant_configs.items():
print(f"{tenant_id}: 状态={config['status']}, "
f"数据库={config['database_name']}, "
f"功能={list(config['features'].keys())}")
print("所有租户工作线程已完成")
---
b.资源隔离与安全性
线程本地存储确保每个租户的数据访问完全隔离，避免数据泄露和跨租户数据污染。
04.日志和监控系统中的上下文传递
a.请求追踪与日志关联
a.日志上下文传递示例
---
# 日志系统中使用线程本地存储进行上下文传递的示例
import threading
import time
import random
import uuid
from typing import Dict, Any, Optional, List
from datetime import datetime
from enum import Enum
class LogLevel(Enum):
TRACE = "TRACE"
DEBUG = "DEBUG"
INFO = "INFO"
WARNING = "WARNING"
ERROR = "ERROR"
CRITICAL = "CRITICAL"
class LoggingContext:
"""日志上下文管理器"""
def __init__(self):
self._local = threading.local()
self.global_log_store = []
self.global_store_lock = threading.Lock()
@contextmanager
def request_context(self, request_id: str = None, user_id: str = None, service: str = None):
"""请求上下文管理器"""
thread_name = threading.current_thread().name
# 生成或使用提供的请求ID
if not request_id:
request_id = str(uuid.uuid4())
# 设置线程本地上下文
self._local.request_id = request_id
self._local.user_id = user_id
self._local.service = service or thread_name
self._local.start_time = time.time()
self._local.correlation_id = str(uuid.uuid4())
self._local.trace_id = str(uuid.uuid4())
self._local.parent_span_id = None
self._local.current_span_id = None
self._local.log_buffer = []
self._local.log_count = 0
self._local.span_stack = []
print(f"{thread_name}: 开始请求上下文 - 请求ID: {request_id}, 关联ID: {self._local.correlation_id}")
try:
yield self
finally:
# 刷新所有缓冲日志
self._flush_buffered_logs()
duration = time.time() - self._local.start_time
print(f"{thread_name}: 结束请求上下文 - 耗时: {duration:.3f}s, 日志数: {self._local.log_count}")
self._cleanup_request_context()
@contextmanager
def operation_span(self, operation_name: str, tags: Dict[str, Any] = None):
"""操作跨度上下文管理器"""
thread_name = threading.current_thread().name
if not hasattr(self._local, 'current_span_id'):
# 如果没有请求上下文，创建一个临时的
self._local.current_span_id = str(uuid.uuid4())
self._local.parent_span_id = None
self._local.span_stack = []
# 创建新的跨度
parent_span_id = self._local.current_span_id
span_id = str(uuid.uuid4())
span_start_time = time.time()
# 推入跨度栈
self._local.span_stack.append(self._local.current_span_id)
self._local.current_span_id = span_id
print(f"{thread_name}: 开始操作跨度 - {operation_name} (跨度ID: {span_id})")
try:
# 记录跨度开始
self._log_span_event('span_start', operation_name, span_id, parent_span_id, tags)
yield self
finally:
# 记录跨度结束
duration = time.time() - span_start_time
self._log_span_event('span_end', operation_name, span_id, parent_span_id,
{**(tags or {}), 'duration_ms': duration * 1000})
# 恢复父跨度
if self._local.span_stack:
self._local.current_span_id = self._local.span_stack.pop()
else:
self._local.current_span_id = parent_span_id
print(f"{thread_name}: 结束操作跨度 - {operation_name} (耗时: {duration:.3f}s)")
def _log_span_event(self, event_type: str, operation_name: str, span_id: str,
parent_span_id: str, tags: Dict[str, Any] = None):
"""记录跨度事件"""
self.log(LogLevel.DEBUG, f"Span {event_type}", {
'operation_name': operation_name,
'span_id': span_id,
'parent_span_id': parent_span_id,
'tags': tags or {}
})
def _cleanup_request_context(self):
"""清理请求上下文"""
attributes_to_remove = [
'request_id', 'user_id', 'service', 'start_time',
'correlation_id', 'trace_id', 'parent_span_id',
'current_span_id', 'log_buffer', 'log_count', 'span_stack'
]
for attr in attributes_to_remove:
if hasattr(self._local, attr):
delattr(self._local, attr)
def _get_context_info(self) -> Dict[str, Any]:
"""获取当前上下文信息"""
return {
'request_id': getattr(self._local, 'request_id', None),
'user_id': getattr(self._local, 'user_id', None),
'service': getattr(self._local, 'service', None),
'correlation_id': getattr(self._local, 'correlation_id', None),
'trace_id': getattr(self._local, 'trace_id', None),
'current_span_id': getattr(self._local, 'current_span_id', None),
'thread_name': threading.current_thread().name,
'thread_id': threading.current_thread().ident
}
def log(self, level: LogLevel, message: str, extra_data: Dict[str, Any] = None):
"""记录日志"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
context_info = self._get_context_info()
log_entry = {
'timestamp': timestamp,
'level': level.value,
'message': message,
'extra_data': extra_data or {},
**context_info
}
# 添加到线程本地缓冲区
if hasattr(self._local, 'log_buffer'):
self._local.log_buffer.append(log_entry)
self._local.log_count += 1
else:
# 如果没有上下文，直接添加到全局存储
with self.global_store_lock:
self.global_log_store.append(log_entry)
# 根据日志级别决定是否立即输出
if level.value in ['ERROR', 'CRITICAL']:
self._output_log_entry(log_entry)
def _output_log_entry(self, log_entry: Dict[str, Any]):
"""输出日志条目到控制台"""
formatted_message = (
f"[{log_entry['timestamp']}] [{log_entry['level']}] "
f"[{log_entry['service'] or 'Unknown'}] "
f"[{log_entry.get('request_id', 'N/A')}] "
f"[{log_entry.get('span_id', 'N/A')}] "
f"{log_entry['message']}"
)
print(formatted_message)
def _flush_buffered_logs(self):
"""刷新缓冲的日志到全局存储"""
if hasattr(self._local, 'log_buffer') and self._local.log_buffer:
with self.global_store_lock:
self.global_log_store.extend(self._local.log_buffer)
# 输出所有缓冲的日志
for log_entry in self._local.log_buffer:
self._output_log_entry(log_entry)
self._local.log_buffer.clear()
def debug(self, message: str, **kwargs):
"""记录调试日志"""
self.log(LogLevel.DEBUG, message, kwargs)
def info(self, message: str, **kwargs):
"""记录信息日志"""
self.log(LogLevel.INFO, message, kwargs)
def warning(self, message: str, **kwargs):
"""记录警告日志"""
self.log(LogLevel.WARNING, message, kwargs)
def error(self, message: str, **kwargs):
"""记录错误日志"""
self.log(LogLevel.ERROR, message, kwargs)
def critical(self, message: str, **kwargs):
"""记录严重错误日志"""
self.log(LogLevel.CRITICAL, message, kwargs)
def get_thread_logs(self, limit: int = None) -> List[Dict[str, Any]]:
"""获取当前线程的日志"""
if hasattr(self._local, 'log_buffer'):
logs = self._local.log_buffer[:]
if limit:
logs = logs[-limit:]
return logs
return []
def get_global_logs(self, request_id: str = None, level: LogLevel = None,
limit: int = 100) -> List[Dict[str, Any]]:
"""获取全局日志"""
with self.global_store_lock:
logs = self.global_log_store[:]
# 过滤条件
if request_id:
logs = [log for log in logs if log.get('request_id') == request_id]
if level:
logs = [log for log in logs if log['level'] == level.value]
# 限制数量并按时间排序
logs = sorted(logs, key=lambda x: x['timestamp'], reverse=True)
if limit:
logs = logs[:limit]
return logs
def get_request_summary(self) -> Dict[str, Any]:
"""获取请求摘要"""
if not hasattr(self._local, 'request_id'):
return None
duration = time.time() - self._local.start_time if hasattr(self._local, 'start_time') else 0
level_counts = {}
for log in self._local.log_buffer:
level = log['level']
level_counts[level] = level_counts.get(level, 0) + 1
return {
'request_id': self._local.request_id,
'correlation_id': self._local.correlation_id,
'trace_id': self._local.trace_id,
'duration': duration,
'log_count': self._local.log_count,
'level_distribution': level_counts,
'span_count': len(self._local.span_stack) + 1 if hasattr(self._local, 'span_stack') else 0
}
class ServiceWithLogging:
"""带日志记录的服务类"""
def __init__(self, service_name: str, logging_context: LoggingContext):
self.service_name = service_name
self.logging_context = logging_context
def process_user_request(self, user_id: str, request_data: Dict[str, Any]) -> Dict[str, Any]:
"""处理用户请求"""
with self.logging_context.operation_span("process_user_request",
{"user_id": user_id, "service": self.service_name}):
self.logging_context.info(f"开始处理用户请求", user_id=user_id)
# 验证用户
with self.logging_context.operation_span("validate_user"):
time.sleep(random.uniform(0.05, 0.1))
if random.random() < 0.1:
self.logging_context.error("用户验证失败", user_id=user_id)
raise Exception("User validation failed")
self.logging_context.debug("用户验证成功", user_id=user_id)
# 处理业务逻辑
with self.logging_context.operation_span("business_logic"):
self.logging_context.info("开始业务逻辑处理")
time.sleep(random.uniform(0.1, 0.3))
# 模拟数据处理
processed_data = {
"original_data": request_data,
"processed_at": time.time(),
"service": self.service_name,
"user_id": user_id
}
if random.random() < 0.05:
self.logging_context.warning("业务逻辑处理出现警告", warning_type="slow_processing")
# 保存结果
with self.logging_context.operation_span("save_result"):
time.sleep(random.uniform(0.02, 0.08))
self.logging_context.info("结果保存成功")
self.logging_context.info("用户请求处理完成", user_id=user_id)
return processed_data
def health_check(self) -> Dict[str, Any]:
"""健康检查"""
with self.logging_context.operation_span("health_check", {"service": self.service_name}):
self.logging_context.debug("开始健康检查")
# 检查数据库连接
with self.logging_context.operation_span("check_database"):
time.sleep(random.uniform(0.01, 0.05))
db_status = "healthy" if random.random() < 0.95 else "unhealthy"
self.logging_context.debug(f"数据库状态: {db_status}")
# 检查外部服务
with self.logging_context.operation_span("check_external_services"):
time.sleep(random.uniform(0.02, 0.06))
external_status = "healthy" if random.random() < 0.9 else "degraded"
self.logging_context.debug(f"外部服务状态: {external_status}")
overall_status = "healthy" if db_status == "healthy" and external_status == "healthy" else "unhealthy"
health_info = {
"service": self.service_name,
"status": overall_status,
"database": db_status,
"external_services": external_status,
"timestamp": time.time()
}
if overall_status != "healthy":
self.logging_context.warning("健康检查发现问题", health_info=health_info)
else:
self.logging_context.debug("健康检查通过")
return health_info
def logging_worker(worker_id: int, requests: List[Dict[str, Any]]):
"""日志工作线程"""
logging_context = LoggingContext()
service = ServiceWithLogging(f"Service-{worker_id}", logging_context)
thread_name = threading.current_thread().name
print(f"{thread_name}: 日志工作器 {worker_id} 开始处理 {len(requests)} 个请求")
results = []
for i, request in enumerate(requests):
request_id = request.get('request_id', f"req_{worker_id}_{i}")
user_id = request.get('user_id', f"user_{worker_id}_{i}")
request_data = request.get('data', {})
request_type = request.get('type', 'user_request')
try:
with logging_context.request_context(request_id, user_id, service.service_name):
if request_type == 'user_request':
result = service.process_user_request(user_id, request_data)
elif request_type == 'health_check':
result = service.health_check()
else:
raise Exception(f"Unknown request type: {request_type}")
results.append(result)
# 获取请求摘要
summary = logging_context.get_request_summary()
print(f"{thread_name}: 请求摘要 - {summary}")
except Exception as e:
logging_context.error(f"请求处理失败: {str(e)}",
request_id=request_id, user_id=user_id)
continue
# 模拟请求间隔
time.sleep(random.uniform(0.1, 0.2))
return results
# 日志请求配置
logging_requests = [
[
{'type': 'user_request', 'request_id': 'req_001', 'user_id': 'alice_001',
'data': {'action': 'view_profile', 'page': 1}},
{'type': 'health_check', 'request_id': 'health_001'},
{'type': 'user_request', 'request_id': 'req_002', 'user_id': 'bob_002',
'data': {'action': 'update_settings', 'theme': 'dark'}}
],
[
{'type': 'user_request', 'request_id': 'req_003', 'user_id': 'charlie_003',
'data': {'action': 'search_products', 'query': 'laptop', 'category': 'electronics'}},
{'type': 'user_request', 'request_id': 'req_004', 'user_id': 'diana_004',
'data': {'action': 'place_order', 'items': ['item_1', 'item_2']}},
{'type': 'health_check', 'request_id': 'health_002'}
],
[
{'type': 'health_check', 'request_id': 'health_003'},
{'type': 'user_request', 'request_id': 'req_005', 'user_id': 'eve_005',
'data': {'action': 'view_analytics', 'period': 'weekly'}},
{'type': 'user_request', 'request_id': 'req_006', 'user_id': 'frank_006',
'data': {'action': 'export_data', 'format': 'csv', 'date_range': 'last_month'}}
]
]
# 创建日志工作线程
logging_threads = []
for i, requests in enumerate(logging_requests):
thread = threading.Thread(
target=logging_worker,
args=(i+1, requests),
name=f"LogWorker-{i+1}"
)
logging_threads.append(thread)
thread.start()
# 等待所有日志线程完成
for thread in logging_threads:
thread.join()
print("\n=== 全局日志分析 ===")
context = LoggingContext()
# 按请求ID分组日志
all_logs = context.get_global_logs(limit=50)
request_groups = {}
for log in all_logs:
req_id = log.get('request_id', 'unknown')
if req_id not in request_groups:
request_groups[req_id] = []
request_groups[req_id].append(log)
print(f"总日志条数: {len(all_logs)}")
print(f"涉及请求数: {len(request_groups)}")
# 显示每个请求的日志摘要
for req_id, logs in request_groups.items():
if req_id != 'unknown':
level_counts = {}
for log in logs:
level = log['level']
level_counts[level] = level_counts.get(level, 0) + 1
print(f"请求 {req_id}: {len(logs)} 条日志, 级别分布: {level_counts}")
print("所有日志工作线程已完成")
---
b.分布式追踪支持
线程本地存储支持分布式追踪系统的上下文传递，确保跨服务调用时的追踪链完整性。
4.4 与全局变量的区别
01.数据共享机制对比
a.全局变量的共享特性
a.全局变量示例
---
# 全局变量与线程本地存储的对比示例
import threading
import time
import random
# 全局变量
global_counter = 0
global_data = {"shared": "initial_value"}
# 全局锁（用于保护全局变量）
global_lock = threading.Lock()
# 线程本地存储
thread_local = threading.local()
def increment_global_counter():
"""增加全局计数器（需要锁保护）"""
global global_counter
with global_lock:
old_value = global_counter
time.sleep(0.001) # 模拟处理时间
global_counter = old_value + 1
return global_counter
def modify_global_data(key, value):
"""修改全局数据（需要锁保护）"""
with global_lock:
old_value = global_data.get(key, "not_found")
global_data[key] = value
return old_value
def work_with_global_variables(worker_id, operations):
"""使用全局变量的工作线程"""
thread_name = threading.current_thread().name
results = []
print(f"{thread_name}: 开始使用全局变量进行 {operations} 次操作")
for i in range(operations):
# 操作全局计数器
counter_value = increment_global_counter()
# 修改全局数据
data_key = f"worker_{worker_id}_op_{i}"
data_value = f"value_{worker_id}_{i}_{random.randint(1, 100)}"
old_value = modify_global_data(data_key, data_value)
results.append({
'operation': i,
'counter': counter_value,
'key': data_key,
'old_value': old_value,
'new_value': data_value
})
time.sleep(random.uniform(0.01, 0.05))
print(f"{thread_name}: 全局变量操作完成，最终计数器值: {global_counter}")
return results
def work_with_thread_local(worker_id, operations):
"""使用线程本地存储的工作线程"""
thread_name = threading.current_thread().name
# 初始化线程本地数据
thread_local.counter = 0
thread_local.data = {}
thread_local.worker_id = worker_id
results = []
print(f"{thread_name}: 开始使用线程本地存储进行 {operations} 次操作")
for i in range(operations):
# 操作本地计数器（无需锁）
thread_local.counter += 1
# 修改本地数据（无需锁）
data_key = f"op_{i}"
data_value = f"local_value_{worker_id}_{i}_{random.randint(1, 100)}"
thread_local.data[data_key] = data_value
results.append({
'operation': i,
'counter': thread_local.counter,
'key': data_key,
'value': data_value,
'thread_id': threading.current_thread().ident
})
time.sleep(random.uniform(0.01, 0.05))
print(f"{thread_name}: 线程本地操作完成，本地计数器值: {thread_local.counter}")
return results
# 创建并启动使用全局变量的线程
print("=== 使用全局变量的线程 ===")
global_threads = []
for i in range(3):
thread = threading.Thread(
target=work_with_global_variables,
args=(i+1, 5),
name=f"GlobalWorker-{i+1}"
)
global_threads.append(thread)
thread.start()
# 等待全局变量线程完成
for thread in global_threads:
thread.join()
print(f"\n全局变量最终状态:")
print(f" 计数器: {global_counter}")
print(f" 数据键数量: {len(global_data)}")
# 重置全局状态
global_counter = 0
global_data.clear()
# 创建并启动使用线程本地存储的线程
print("\n=== 使用线程本地存储的线程 ===")
local_threads = []
for i in range(3):
thread = threading.Thread(
target=work_with_thread_local,
args=(i+1, 5),
name=f"LocalWorker-{i+1}"
)
local_threads.append(thread)
thread.start()
# 等待线程本地存储线程完成
for thread in local_threads:
thread.join()
print(f"\n线程本地存储最终状态:")
print(f" 全局计数器: {global_counter} (未被修改)")
print(f" 全局数据键数量: {len(global_data)} (未被修改)")
---
b.共享访问问题
全局变量在所有线程间共享，需要使用锁等同步机制来避免数据竞争，而线程本地存储每个线程都有独立副本。
b.线程本地存储的隔离特性
a.数据隔离验证
通过实际代码验证线程本地存储的数据隔离性，确保不同线程间的数据互不影响。
02.性能特性对比
a.访问速度差异
a.性能测试示例
---
# 全局变量与线程本地存储的性能对比示例
import threading
import time
from contextlib import contextmanager
# 性能测试配置
ITERATIONS = 1000000
THREAD_COUNT = 4
# 全局变量
global_var = 0
global_lock = threading.RLock() # 使用可重入锁
# 线程本地存储
thread_local = threading.local()
@contextmanager
def timer(description):
"""简单的计时器上下文管理器"""
start_time = time.perf_counter()
yield
end_time = time.perf_counter()
print(f"{description}: {end_time - start_time:.6f} 秒")
def benchmark_global_access():
"""全局变量访问性能测试"""
global global_var
for i in range(ITERATIONS):
with global_lock:
global_var += 1
temp = global_var
global_var = temp * 2
global_var = temp
def benchmark_local_access():
"""线程本地存储访问性能测试"""
if not hasattr(thread_local, 'counter'):
thread_local.counter = 0
for i in range(ITERATIONS):
thread_local.counter += 1
temp = thread_local.counter
thread_local.counter = temp * 2
thread_local.counter = temp
def benchmark_global_read_only():
"""全局变量只读访问性能测试"""
global global_var
for i in range(ITERATIONS):
temp = global_var # 只读访问，不需要锁
_ = temp * 2
def benchmark_local_read_only():
"""线程本地存储只读访问性能测试"""
if not hasattr(thread_local, 'counter'):
thread_local.counter = 0
for i in range(ITERATIONS):
temp = thread_local.counter
_ = temp * 2
def run_performance_test(test_name, target_function, thread_count=THREAD_COUNT):
"""运行性能测试"""
print(f"\n=== {test_name} ===")
print(f"迭代次数: {ITERATIONS:,}")
print(f"线程数量: {thread_count}")
# 单线程测试
with timer(f"{test_name} (单线程)"):
target_function()
# 多线程测试
threads = []
with timer(f"{test_name} ({thread_count}线程)"):
for _ in range(thread_count):
thread = threading.Thread(target=target_function)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
# 运行各种性能测试
print("开始性能基准测试...")
# 1. 全局变量写操作（需要锁）
run_performance_test("全局变量写操作", benchmark_global_access)
# 2. 线程本地存储写操作
run_performance_test("线程本地存储写操作", benchmark_local_access)
# 3. 全局变量只读操作
run_performance_test("全局变量只读操作", benchmark_global_read_only)
# 4. 线程本地存储只读操作
run_performance_test("线程本地存储只读操作", benchmark_local_read_only)
# 5. 更复杂的操作性能测试
def complex_global_operation():
"""复杂的全局变量操作"""
global global_var
with global_lock:
global_var = (global_var + 1) % 1000
temp = global_var
result = temp * temp + temp
global_var = result % 1000
def complex_local_operation():
"""复杂的线程本地存储操作"""
if not hasattr(thread_local, 'value'):
thread_local.value = 0
thread_local.value = (thread_local.value + 1) % 1000
temp = thread_local.value
result = temp * temp + temp
thread_local.value = result % 1000
print("\n=== 复杂操作性能对比 ===")
run_performance_test("复杂全局变量操作", complex_global_operation)
run_performance_test("复杂线程本地存储操作", complex_local_operation)
print("\n性能测试完成")
---
b.锁开销影响
全局变量需要同步机制保护，锁的获取和释放会带来额外开销，而线程本地存储无需同步，访问效率更高。
03.内存使用对比
a.内存分配策略
a.内存使用分析示例
---
# 内存使用对比分析示例
import threading
import time
import gc
import sys
from typing import Dict, List, Any
class MemoryAnalyzer:
"""内存分析器"""
@staticmethod
def get_object_size(obj):
"""获取对象大小（简化版本）"""
return sys.getsizeof(obj)
@staticmethod
def analyze_global_memory():
"""分析全局变量内存使用"""
global_data = {
'counters': [0] * 1000,
'strings': [f"string_{i}" for i in range(500)],
'nested_data': {
'level1': {
'level2': {
'data': list(range(100))
}
}
}
}
total_size = MemoryAnalyzer.get_object_size(global_data)
counter_size = MemoryAnalyzer.get_object_size(global_data['counters'])
string_size = MemoryAnalyzer.get_object_size(global_data['strings'])
return {
'total_size': total_size,
'counters_size': counter_size,
'strings_size': string_size,
'nested_size': total_size - counter_size - string_size
}
@staticmethod
def analyze_thread_local_memory(thread_count=4):
"""分析线程本地存储内存使用"""
thread_locals = []
total_sizes = []
def create_local_data():
"""为每个线程创建本地数据"""
local = threading.local()
local.counters = [0] * 1000
local.strings = [f"thread_{threading.current_thread().ident}_string_{i}"
for i in range(500)]
local.nested_data = {
'level1': {
'level2': {
'data': list(range(100))
}
}
}
local.thread_id = threading.current_thread().ident
thread_locals.append(local)
# 计算本地数据大小
total_size = (MemoryAnalyzer.get_object_size(local.counters) +
MemoryAnalyzer.get_object_size(local.strings) +
MemoryAnalyzer.get_object_size(local.nested_data))
total_sizes.append(total_size)
# 创建多个线程，每个都有自己的本地数据
threads = []
for i in range(thread_count):
thread = threading.Thread(target=create_local_data)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
return {
'thread_count': thread_count,
'per_thread_sizes': total_sizes,
'total_size': sum(total_sizes),
'average_size': sum(total_sizes) / len(total_sizes) if total_sizes else 0
}
def memory_usage_comparison():
"""内存使用对比测试"""
print("=== 内存使用对比分析 ===")
# 强制垃圾回收
gc.collect()
# 分析全局变量内存使用
print("\n1. 全局变量内存分析:")
global_memory = MemoryAnalyzer.analyze_global_memory()
print(f" 总大小: {global_memory['total_size']:,} 字节")
print(f" 计数器大小: {global_memory['counters_size']:,} 字节")
print(f" 字符串大小: {global_memory['strings_size']:,} 字节")
print(f" 嵌套数据大小: {global_memory['nested_size']:,} 字节")
# 分析线程本地存储内存使用
print("\n2. 线程本地存储内存分析:")
for thread_count in [2, 4, 8]:
local_memory = MemoryAnalyzer.analyze_thread_local_memory(thread_count)
print(f" 线程数: {thread_count}")
print(f" 总大小: {local_memory['total_size']:,} 字节")
print(f" 平均每线程: {local_memory['average_size']:,} 字节")
print(f" 各线程大小: {[f'{size:,}' for size in local_memory['per_thread_sizes']]}")
# 内存效率对比
print("\n3. 内存效率对比:")
single_global_size = global_memory['total_size']
multi_local_size_4 = MemoryAnalyzer.analyze_thread_local_memory(4)['total_size']
print(f" 单个全局实例: {single_global_size:,} 字节")
print(f" 4个线程本地实例: {multi_local_size_4:,} 字节")
print(f" 内存开销比: {multi_local_size_4 / single_global_size:.2f}x")
# 内存访问性能测试
def test_memory_access_pattern():
"""测试内存访问模式对性能的影响"""
print("\n4. 内存访问性能测试:")
# 全局变量访问测试
global_data = list(range(10000))
def global_access_test():
total = 0
for i in range(1000):
total += sum(global_data[:100]) # 访问全局数据的一部分
return total
# 线程本地存储访问测试
def local_access_test():
if not hasattr(threading.local(), 'data'):
threading.local().data = list(range(10000))
total = 0
for i in range(1000):
total += sum(threading.local().data[:100]) # 访问本地数据的一部分
return total
# 性能测试
start_time = time.perf_counter()
global_result = global_access_test()
global_time = time.perf_counter() - start_time
start_time = time.perf_counter()
local_result = local_access_test()
local_time = time.perf_counter() - start_time
print(f" 全局变量访问时间: {global_time:.6f} 秒")
print(f" 线程本地存储访问时间: {local_time:.6f} 秒")
print(f" 性能比: {local_time / global_time:.2f}x")
test_memory_access_pattern()
# 运行内存对比分析
memory_usage_comparison()
---
b.内存开销分析
线程本地存储为每个线程维护独立的数据副本，总体内存使用量会随线程数量增加而线性增长。
04.应用场景适应性分析
a.同步需求场景
a.同步复杂度对比
---
# 同步需求场景的对比分析示例
import threading
import time
import random
from collections import defaultdict
from typing import Dict, List
class SynchronizationComplexityDemo:
"""同步复杂度演示"""
def __init__(self):
# 全局共享资源
self.global_counters = defaultdict(int)
self.global_data = {}
self.global_lock = threading.RLock() # 可重入锁
self.condition_var = threading.Condition(self.global_lock)
# 线程本地存储
self.thread_local = threading.local()
def complex_global_operation(self, worker_id: int, operation_type: str):
"""复杂的全局变量操作（需要复杂同步）"""
thread_name = threading.current_thread().name
if operation_type == "counter":
# 复杂的计数器操作
with self.global_lock:
# 获取多个计数器的值
old_counter = self.global_counters[f"worker_{worker_id}"]
total_count = sum(self.global_counters.values())
# 模拟复杂的计算
time.sleep(random.uniform(0.001, 0.005))
# 更新计数器
self.global_counters[f"worker_{worker_id}"] = old_counter + 1
new_total = sum(self.global_counters.values())
return {
'worker_id': worker_id,
'old_counter': old_counter,
'new_counter': old_counter + 1,
'old_total': total_count,
'new_total': new_total
}
elif operation_type == "data":
# 复杂的数据操作
with self.global_lock:
key = f"data_{worker_id}_{time.time()}"
value = {
'worker_id': worker_id,
'timestamp': time.time(),
'random_data': [random.randint(1, 1000) for _ in range(10)]
}
# 检查是否有足够的空间
if len(self.global_data) >= 1000:
# 清理旧数据
oldest_keys = sorted(self.global_data.keys())[:10]
for old_key in oldest_keys:
del self.global_data[old_key]
self.global_data[key] = value
return {'key': key, 'total_data_size': len(self.global_data)}
elif operation_type == "condition":
# 条件等待操作
with self.condition_var:
# 等待特定条件
while sum(self.global_counters.values()) < worker_id * 5:
self.condition_var.wait(timeout=0.1)
# 执行操作
self.global_counters[f"condition_worker_{worker_id}"] += 1
self.condition_var.notify_all()
return {
'worker_id': worker_id,
'condition_met': True,
'total_count': sum(self.global_counters.values())
}
def simple_local_operation(self, worker_id: int, operation_type: str):
"""简单的线程本地存储操作（无需同步）"""
thread_name = threading.current_thread().name
# 初始化本地数据
if not hasattr(self.thread_local, 'counters'):
self.thread_local.counters = defaultdict(int)
self.thread_local.data = {}
self.thread_local.operation_log = []
if operation_type == "counter":
# 简单的计数器操作
old_counter = self.thread_local.counters[f"op_{operation_type}"]
self.thread_local.counters[f"op_{operation_type}"] += 1
# 记录操作
self.thread_local.operation_log.append({
'timestamp': time.time(),
'operation': 'counter',
'worker_id': worker_id,
'old_value': old_counter,
'new_value': old_counter + 1
})
return {
'worker_id': worker_id,
'old_counter': old_counter,
'new_counter': old_counter + 1,
'thread_id': threading.current_thread().ident
}
elif operation_type == "data":
# 简单的数据操作
key = f"local_data_{worker_id}_{len(self.thread_local.data)}"
value = {
'worker_id': worker_id,
'thread_id': threading.current_thread().ident,
'timestamp': time.time(),
'random_data': [random.randint(1, 1000) for _ in range(10)]
}
self.thread_local.data[key] = value
return {'key': key, 'local_data_size': len(self.thread_local.data)}
elif operation_type == "condition":
# 模拟条件操作（无需真正的等待）
counter_key = f"condition_worker_{worker_id}"
target_count = worker_id * 5
# 直接操作本地数据
self.thread_local.counters[counter_key] += 1
# 模拟条件检查
condition_met = sum(self.thread_local.counters.values()) >= target_count
self.thread_local.operation_log.append({
'timestamp': time.time(),
'operation': 'condition',
'worker_id': worker_id,
'condition_met': condition_met
})
return {
'worker_id': worker_id,
'condition_met': condition_met,
'local_count': sum(self.thread_local.counters.values())
}
def get_local_summary(self):
"""获取线程本地数据摘要"""
if hasattr(self.thread_local, 'counters'):
return {
'counters': dict(self.thread_local.counters),
'data_count': len(getattr(self.thread_local, 'data', {})),
'operation_log_size': len(getattr(self.thread_local, 'operation_log', [])),
'total_operations': sum(self.thread_local.counters.values())
}
return None
def synchronization_comparison_demo():
"""同步复杂度对比演示"""
print("=== 同步复杂度对比演示 ===")
demo = SynchronizationComplexityDemo()
# 全局变量同步操作测试
print("\n1. 全局变量同步操作:")
global_threads = []
global_results = []
def global_worker(worker_id):
thread_results = []
for i in range(10):
op_type = random.choice(["counter", "data", "condition"])
try:
result = demo.complex_global_operation(worker_id, op_type)
thread_results.append(result)
except Exception as e:
print(f"全局工作器 {worker_id} 操作失败: {e}")
time.sleep(random.uniform(0.01, 0.05))
global_results.extend(thread_results)
start_time = time.perf_counter()
for i in range(4):
thread = threading.Thread(target=global_worker, args=(i+1,))
global_threads.append(thread)
thread.start()
for thread in global_threads:
thread.join()
global_time = time.perf_counter() - start_time
print(f" 执行时间: {global_time:.3f} 秒")
print(f" 操作总数: {len(global_results)}")
print(f" 全局计数器状态: {dict(demo.global_counters)}")
print(f" 全局数据大小: {len(demo.global_data)}")
# 线程本地存储操作测试
print("\n2. 线程本地存储操作:")
local_threads = []
local_summaries = []
def local_worker(worker_id):
thread_results = []
for i in range(10):
op_type = random.choice(["counter", "data", "condition"])
result = demo.simple_local_operation(worker_id, op_type)
thread_results.append(result)
time.sleep(random.uniform(0.01, 0.05))
summary = demo.get_local_summary()
local_summaries.append(summary)
return thread_results
start_time = time.perf_counter()
for i in range(4):
thread = threading.Thread(target=local_worker, args=(i+1,))
local_threads.append(thread)
thread.start()
for thread in local_threads:
thread.join()
local_time = time.perf_counter() - start_time
print(f" 执行时间: {local_time:.3f} 秒")
print(f" 线程摘要数量: {len(local_summaries)}")
for i, summary in enumerate(local_summaries):
if summary:
print(f" 线程 {i+1} 摘要: 计数器={len(summary['counters'])}, "
f"数据项={summary['data_count']}, "
f"总操作={summary['total_operations']}")
# 性能和复杂度对比
print("\n3. 复杂度对比:")
print(f" 全局变量方法:")
print(f" - 需要复杂的锁管理")
print(f" - 需要条件变量处理")
print(f" - 容易出现死锁风险")
print(f" - 代码复杂度高")
print(f" - 执行时间: {global_time:.3f}秒")
print(f" 线程本地存储方法:")
print(f" - 无需锁和同步")
print(f" - 代码简洁明了")
print(f" - 无死锁风险")
print(f" - 执行时间: {local_time:.3f}秒")
print(f" 性能提升: {((global_time - local_time) / global_time * 100):.1f}%")
# 运行同步复杂度对比演示
synchronization_comparison_demo()
---
b.死锁风险对比
全局变量操作需要复杂的同步机制，容易出现死锁问题，而线程本地存储天然避免了死锁风险。
05.最佳实践选择指南
a.选择决策树
a.决策因素分析
---
# 线程本地存储vs全局变量选择指南示例
import threading
import time
from typing import Dict, List, Any
from enum import Enum
class DataType(Enum):
COUNTER = "counter"
CONFIGURATION = "configuration"
SESSION_DATA = "session_data"
CACHE = "cache"
LOG_BUFFER = "log_buffer"
class AccessPattern(Enum):
READ_HEAVY = "read_heavy"
WRITE_HEAVY = "write_heavy"
MIXED = "mixed"
SEQUENTIAL = "sequential"
class ConsistencyRequirement(Enum):
STRONG = "strong"
EVENTUAL = "eventual"
NONE = "none"
class StorageDecisionEngine:
"""存储方案决策引擎"""
def __init__(self):
self.decision_factors = {
'thread_isolation': 0.3, # 线程隔离需求权重
'performance': 0.25, # 性能需求权重
'memory_efficiency': 0.2, # 内存效率权重
'code_simplicity': 0.15, # 代码简洁性权重
'scalability': 0.1 # 可扩展性权重
}
def analyze_use_case(self, use_case: Dict[str, Any]) -> Dict[str, Any]:
"""分析用例特征"""
analysis = {}
# 线程隔离需求分析
isolation_score = self._analyze_isolation_need(use_case)
analysis['isolation_score'] = isolation_score
# 性能需求分析
performance_score = self._analyze_performance_need(use_case)
analysis['performance_score'] = performance_score
# 内存效率分析
memory_score = self._analyze_memory_efficiency(use_case)
analysis['memory_score'] = memory_score
# 代码简洁性分析
simplicity_score = self._analyze_code_simplicity(use_case)
analysis['simplicity_score'] = simplicity_score
# 可扩展性分析
scalability_score = self._analyze_scalability(use_case)
analysis['scalability_score'] = scalability_score
return analysis
def _analyze_isolation_need(self, use_case: Dict[str, Any]) -> float:
"""分析线程隔离需求 (0-1分)"""
score = 0.0
# 数据类型影响隔离需求
data_type = use_case.get('data_type')
if data_type in [DataType.SESSION_DATA, DataType.LOG_BUFFER]:
score += 0.4 # 高隔离需求
elif data_type in [DataType.CONFIGURATION, DataType.CACHE]:
score += 0.2 # 中等隔离需求
# 访问模式影响
access_pattern = use_case.get('access_pattern')
if access_pattern == AccessPattern.WRITE_HEAVY:
score += 0.3 # 写密集需要隔离
elif access_pattern == AccessPattern.MIXED:
score += 0.2
# 一致性需求影响
consistency = use_case.get('consistency_requirement')
if consistency == ConsistencyRequirement.NONE:
score += 0.3 # 无一致性要求适合隔离
return min(score, 1.0)
def _analyze_performance_need(self, use_case: Dict[str, Any]) -> float:
"""分析性能需求 (0-1分)"""
score = 0.0
# 访问频率
access_frequency = use_case.get('access_frequency', 'medium')
if access_frequency == 'high':
score += 0.4
elif access_frequency == 'medium':
score += 0.2
# 并发级别
concurrency_level = use_case.get('concurrency_level', 'medium')
if concurrency_level == 'high':
score += 0.3 # 高并发时线程本地存储性能更好
elif concurrency_level == 'medium':
score += 0.15
# 延迟要求
latency_requirement = use_case.get('latency_requirement', 'medium')
if latency_requirement == 'low':
score += 0.3 # 低延迟要求适合线程本地存储
return min(score, 1.0)
def _analyze_memory_efficiency(self, use_case: Dict[str, Any]) -> float:
"""分析内存效率需求 (0-1分)"""
score = 0.0
# 数据大小
data_size = use_case.get('data_size', 'medium')
if data_size == 'small':
score += 0.4 # 小数据适合线程本地存储
elif data_size == 'large':
score -= 0.2 # 大数据可能更适合全局变量
# 线程数量
thread_count = use_case.get('expected_thread_count', 4)
if thread_count <= 4:
score += 0.2 # 少量线程时内存开销可接受
elif thread_count > 20:
score -= 0.3 # 大量线程时内存开销显著
# 内存限制
memory_constraint = use_case.get('memory_constraint', 'medium')
if memory_constraint == 'tight':
score -= 0.2 # 内存紧张时谨慎使用线程本地存储
return max(0.0, min(score, 1.0))
def _analyze_code_simplicity(self, use_case: Dict[str, Any]) -> float:
"""分析代码简洁性需求 (0-1分)"""
score = 0.0
# 同步复杂性
sync_complexity = use_case.get('synchronization_complexity', 'medium')
if sync_complexity == 'high':
score += 0.4 # 复杂同步时线程本地存储更简单
elif sync_complexity == 'low':
score -= 0.1 # 简单同步时全局变量也OK
# 开发团队经验
team_experience = use_case.get('team_concurrency_experience', 'medium')
if team_experience == 'low':
score += 0.3 # 经验不足时线程本地存储更安全
return min(score, 1.0)
def _analyze_scalability(self, use_case: Dict[str, Any]) -> float:
"""分析可扩展性需求 (0-1分)"""
score = 0.0
# 预期增长
growth_expectation = use_case.get('expected_growth', 'stable')
if growth_expectation == 'high':
score += 0.3 # 高增长时线程本地存储扩展性好
# 动态线程
dynamic_threads = use_case.get('dynamic_thread_creation', False)
if dynamic_threads:
score += 0.4 # 动态线程创建时线程本地存储更合适
return min(score, 1.0)
def make_recommendation(self, use_case: Dict[str, Any]) -> Dict[str, Any]:
"""做出存储方案推荐"""
analysis = self.analyze_use_case(use_case)
# 计算综合得分
thread_local_score = (
analysis['isolation_score'] * self.decision_factors['thread_isolation'] +
analysis['performance_score'] * self.decision_factors['performance'] +
analysis['memory_score'] * self.decision_factors['memory_efficiency'] +
analysis['simplicity_score'] * self.decision_factors['code_simplicity'] +
analysis['scalability_score'] * self.decision_factors['scalability']
)
# 全局变量得分（某些因素取反）
global_score = (
(1 - analysis['isolation_score']) * self.decision_factors['thread_isolation'] +
(1 - analysis['performance_score'] * 0.5) * self.decision_factors['performance'] +
(1 - analysis['memory_score'] * 0.3) * self.decision_factors['memory_efficiency'] +
(1 - analysis['simplicity_score'] * 0.7) * self.decision_factors['code_simplicity'] +
analysis['scalability_score'] * self.decision_factors['scalability'] * 0.5
)
recommendation = "thread_local" if thread_local_score > global_score else "global"
confidence = abs(thread_local_score - global_score)
return {
'recommendation': recommendation,
'confidence': confidence,
'thread_local_score': thread_local_score,
'global_score': global_score,
'analysis': analysis,
'reasoning': self._generate_reasoning(analysis, recommendation)
}
def _generate_reasoning(self, analysis: Dict[str, Any], recommendation: str) -> List[str]:
"""生成推荐理由"""
reasons = []
if recommendation == "thread_local":
if analysis['isolation_score'] > 0.6:
reasons.append("高线程隔离需求")
if analysis['performance_score'] > 0.6:
reasons.append("性能要求较高")
if analysis['simplicity_score'] > 0.6:
reasons.append("需要简化同步逻辑")
if analysis['scalability_score'] > 0.6:
reasons.append("良好的可扩展性")
else:
if analysis['memory_score'] > 0.6:
reasons.append("内存效率考虑")
if analysis['isolation_score'] < 0.4:
reasons.append("数据共享需求")
if 1 - analysis['simplicity_score'] > 0.6:
reasons.append("同步逻辑相对简单")
return reasons if reasons else ["综合评估结果"]
def demonstrate_decision_engine():
"""演示决策引擎"""
print("=== 存储方案决策指南演示 ===")
engine = StorageDecisionEngine()
# 定义不同的用例
use_cases = [
{
'name': 'Web会话管理',
'data_type': DataType.SESSION_DATA,
'access_pattern': AccessPattern.MIXED,
'consistency_requirement': ConsistencyRequirement.NONE,
'access_frequency': 'high',
'concurrency_level': 'high',
'data_size': 'small',
'expected_thread_count': 50,
'synchronization_complexity': 'high',
'dynamic_thread_creation': True
},
{
'name': '全局计数器',
'data_type': DataType.COUNTER,
'access_pattern': AccessPattern.WRITE_HEAVY,
'consistency_requirement': ConsistencyRequirement.STRONG,
'access_frequency': 'high',
'concurrency_level': 'high',
'data_size': 'small',
'expected_thread_count': 10,
'synchronization_complexity': 'medium',
'dynamic_thread_creation': False
},
{
'name': '应用配置',
'data_type': DataType.CONFIGURATION,
'access_pattern': AccessPattern.READ_HEAVY,
'consistency_requirement': ConsistencyRequirement.EVENTUAL,
'access_frequency': 'medium',
'concurrency_level': 'medium',
'data_size': 'medium',
'expected_thread_count': 8,
'synchronization_complexity': 'low',
'dynamic_thread_creation': False
},
{
'name': '线程日志缓冲',
'data_type': DataType.LOG_BUFFER,
'access_pattern': AccessPattern.WRITE_HEAVY,
'consistency_requirement': ConsistencyRequirement.NONE,
'access_frequency': 'high',
'concurrency_level': 'high',
'data_size': 'medium',
'expected_thread_count': 20,
'synchronization_complexity': 'high',
'dynamic_thread_creation': True
}
]
for use_case in use_cases:
print(f"\n用例: {use_case['name']}")
print("-" * 50)
recommendation = engine.make_recommendation(use_case)
print(f"推荐方案: {'线程本地存储' if recommendation['recommendation'] == 'thread_local' else '全局变量'}")
print(f"置信度: {recommendation['confidence']:.2f}")
print(f"线程本地存储得分: {recommendation['thread_local_score']:.3f}")
print(f"全局变量得分: {recommendation['global_score']:.3f}")
print(f"推荐理由:")
for reason in recommendation['reasoning']:
print(f" - {reason}")
print(f"详细分析:")
analysis = recommendation['analysis']
print(f" 隔离需求: {analysis['isolation_score']:.2f}")
print(f" 性能需求: {analysis['performance_score']:.2f}")
print(f" 内存效率: {analysis['memory_score']:.2f}")
print(f" 代码简洁性: {analysis['simplicity_score']:.2f}")
print(f" 可扩展性: {analysis['scalability_score']:.2f}")
# 运行决策引擎演示
demonstrate_decision_engine()
---
b.实际应用建议
根据具体的应用场景、性能需求、内存限制和团队经验来选择合适的存储方案，线程本地存储适合需要数据隔离和高性能的场景。
06.总结与展望
a.技术特点总结
a.核心优势对比
线程本地存储在数据隔离、性能优化和代码简化方面具有明显优势，特别适合多线程环境下的独立数据处理。
b.未来发展趋势
a.技术演进方向
随着多核处理器和并发编程的普及，线程本地存储技术将继续发展，提供更高效的内存管理和更丰富的功能特性。
5 线程通信
5.1 Queue队列
01.Queue队列概述
a.基本概念
Queue是Python标准库queue模块提供的线程安全队列，专为多线程环境设计，解决了线程间安全数据交换的问题。与普通列表不同，Queue内部实现了必要的线程锁机制，确保多个线程同时操作时不会出现数据竞争和不一致状态。Queue支持先进先出（FIFO）、后进先出（LIFO）和优先级队列三种模式，是Python多线程编程中最常用的线程通信工具之一。
b.主要特性
a.线程安全特性
Queue内部自动处理线程同步问题，使用锁机制确保多线程环境下的数据安全，开发者无需手动添加锁。
b.阻塞操作
当队列为空时get操作会阻塞，当队列满时put操作会阻塞，提供了timeout参数实现非阻塞或超时等待。
c.灵活队列类型
支持不同类型的队列：Queue（FIFO）、LifoQueue（LIFO）、PriorityQueue（优先级队列）。
02.Queue基础操作
a.创建队列
a.队列初始化方法
可以通过maxsize参数设置队列最大容量，maxsize=0表示无限大小队列。
b.代码示例
---
# Queue基本创建示例
import queue
import threading
import time
# 创建不同类型的队列
# FIFO队列（先进先出）
fifo_queue = queue.Queue(maxsize=5) # 最多容纳5个元素
# LIFO队列（后进先出）
lifo_queue = queue.LifoQueue(maxsize=3)
# 优先级队列（按优先级排序）
priority_queue = queue.PriorityQueue(maxsize=10)
print(f"FIFO队列最大容量: {fifo_queue.maxsize}")
print(f"LIFO队列最大容量: {lifo_queue.maxsize}")
print(f"优先级队列最大容量: {priority_queue.maxsize}")
---
b.入队操作（put）
a.put方法详解
put(item, block=True, timeout=None)：将item放入队列，block参数控制是否阻塞，timeout设置阻塞超时时间。
b.代码示例
---
# put方法使用示例
import queue
import threading
import time
def producer(queue_data):
"""生产者线程：向队列中添加数据"""
items = ["任务1", "任务2", "任务3", "任务4"]
for item in items:
try:
# 普通put操作（阻塞直到有空间）
queue_data.put(item)
print(f"生产者：放入 {item}，队列大小：{queue_data.qsize()}")
time.sleep(0.5)
except queue.Full:
print("队列已满，无法放入新数据")
break
# 创建有限大小的队列
task_queue = queue.Queue(maxsize=3)
# 启动生产者线程
producer_thread = threading.Thread(
target=producer,
args=(task_queue,),
name="Producer"
)
producer_thread.start()
producer_thread.join()
print(f"队列最终内容：{list(task_queue.queue)}")
---
c.非阻塞put操作
a.block=False的使用场景
当需要立即知道队列是否已满，而不愿等待时，使用block=False。
b.代码示例
---
# 非阻塞put操作示例
import queue
small_queue = queue.Queue(maxsize=2)
# 先放入两个元素填满队列
small_queue.put("第一项")
small_queue.put("第二项")
print(f"队列已满：{small_queue.full()}")
# 尝试非阻塞put
try:
small_queue.put("第三项", block=False)
except queue.Full:
print("非阻塞put失败：队列已满")
# 使用超时的put操作
try:
small_queue.put("第三项", timeout=2.0)
except queue.Full:
print("超时put失败：2秒内队列仍有空间")
---
03.出队操作（get）
a.get方法详解
get(block=True, timeout=None)：从队列中取出并移除元素，支持阻塞和非阻塞模式，timeout设置等待超时时间。
b.代码示例
---
# get方法使用示例
import queue
import threading
import time
def consumer(queue_data):
"""消费者线程：从队列中取出数据"""
while True:
try:
# 阻塞式get操作（等待队列中有数据）
item = queue_data.get(timeout=3.0) # 最多等待3秒
print(f"消费者：取出 {item}，队列剩余：{queue_data.qsize()}")
# 模拟处理任务
time.sleep(1)
# 通知队列任务处理完成
queue_data.task_done()
# 终止条件
if item == "终止任务":
break
except queue.Empty:
print("消费者：队列为空，等待超时")
break
# 创建队列并预填充数据
work_queue = queue.Queue()
work_items = ["数据处理", "文件写入", "网络请求", "数据库操作", "终止任务"]
for item in work_items:
work_queue.put(item)
print(f"初始队列大小：{work_queue.qsize()}")
# 启动消费者线程
consumer_thread = threading.Thread(
target=consumer,
args=(work_queue,),
name="Consumer"
)
consumer_thread.start()
consumer_thread.join()
print("消费者线程结束")
---
c.非阻塞get操作
a立即获取或失败策略
使用block=False可以立即尝试获取元素，如果队列为空则抛出Empty异常，不会阻塞线程。
b.代码示例
---
# 非阻塞get操作示例
import queue
empty_queue = queue.Queue()
# 尝试非阻塞get操作
try:
item = empty_queue.get(block=False)
print(f"获取到：{item}")
except queue.Empty:
print("非阻塞get失败：队列为空")
# 带超时的get操作
try:
# 在另一个线程中延迟放入数据
def delayed_put():
import time
time.sleep(2)
empty_queue.put("延迟数据")
import threading
timer_thread = threading.Thread(target=delayed_put)
timer_thread.start()
# 等待最多3秒获取数据
item = empty_queue.get(timeout=3.0)
print(f"超时get成功：{item}")
except queue.Empty:
print("超时get失败：3秒内没有数据")
---
04.Queue高级功能
a.队列状态查询
a.qsize()方法
返回队列中当前元素数量，注意这个值在不加锁的情况下可能不完全准确。
b.full()和empty()方法
分别检查队列是否已满或为空，同样存在线程安全问题。
c.代码示例
---
# 队列状态查询示例
import queue
import threading
import time
def monitor_queue(q):
"""监控队列状态"""
for i in range(10):
print(f"队列大小：{q.qsize()}, 是否为空：{q.empty()}, 是否已满：{q.full()}")
time.sleep(0.5)
# 创建有限大小的队列
monitor_q = queue.Queue(maxsize=5)
# 启动监控线程
monitor_thread = threading.Thread(target=monitor_queue, args=(monitor_q,))
monitor_thread.start()
# 逐步填充队列
for i in range(6):
try:
monitor_q.put(f"项目{i}")
time.sleep(0.3)
except queue.Full:
print(f"队列已满，无法放入项目{i}")
# 逐步清空队列
while not monitor_q.empty():
item = monitor_q.get()
print(f"取出：{item}")
time.sleep(0.3)
monitor_thread.join()
---
b.task_done()和join()方法
a任务完成机制
task_done()通知队列一个任务已完成，join()会阻塞直到所有任务都被处理完毕。
b.代码示例
---
# task_done和join使用示例
import queue
import threading
import time
import random
def worker(worker_id, task_queue):
"""工作线程处理任务"""
print(f"工作线程{worker_id}开始工作")
while True:
try:
# 获取任务
task = task_queue.get(timeout=2.0)
print(f"工作线程{worker_id}处理任务：{task}")
# 模拟任务处理
processing_time = random.uniform(0.5, 2.0)
time.sleep(processing_time)
# 标记任务完成
task_queue.task_done()
print(f"工作线程{worker_id}完成任务：{task}，耗时{processing_time:.2f}秒")
except queue.Empty:
print(f"工作线程{worker_id}：没有更多任务")
break
# 创建任务队列
tasks = [
"下载文件A", "处理图片B", "发送邮件C", "更新数据库D",
"生成报告E", "清理缓存F", "备份数据G", "发送通知H"
]
task_queue = queue.Queue()
# 添加任务到队列
for task in tasks:
task_queue.put(task)
print(f"添加了{len(tasks)}个任务到队列")
# 启动3个工作线程
workers = []
for i in range(3):
worker_thread = threading.Thread(
target=worker,
args=(i+1, task_queue),
name=f"Worker-{i+1}"
)
workers.append(worker_thread)
worker_thread.start()
# 等待所有任务完成
print("主线程等待所有任务完成...")
task_queue.join() # 阻塞直到所有任务被处理
print("所有任务已完成！")
# 等待工作线程结束
for worker_thread in workers:
worker_thread.join()
print("所有工作线程已结束")
---
5.2 线程安全的数据结构
01.线程安全数据结构概述
a.基本概念
线程安全的数据结构是指能够在多线程环境下安全操作的数据结构，内部已经实现了必要的同步机制（如锁、信号量等），确保多个线程同时访问时不会产生数据竞争和不一致状态。Python提供了多种内置的线程安全数据结构，如queue模块中的各种队列、threading模块中的Event、Condition等，这些工具大大简化了多线程编程的复杂性。
b.为什么需要线程安全
a.数据竞争问题
当多个线程同时读写共享数据时，如果没有适当的同步机制，可能导致数据损坏、程序崩溃或产生不可预测的结果。
b.原子操作需求
某些操作需要作为一个不可分割的整体执行，线程安全数据结构确保这些操作的原子性。
02.锁机制基础
a.Lock锁
a.Lock基本概念
threading.Lock是最基本的同步原语，同一时间只允许一个线程持有锁，其他试图获取锁的线程会被阻塞直到锁被释放。
b.Lock使用方法
acquire()获取锁，release()释放锁，建议使用with语句确保锁的正确释放。
c.代码示例
---
# threading.Lock使用示例
import threading
import time
import random
class BankAccount:
"""银行账户类 - 使用Lock保护余额操作"""
def __init__(self, initial_balance=0):
self.balance = initial_balance
self.lock = threading.Lock()
print(f"账户创建，初始余额：{self.balance}")
def deposit(self, amount):
"""存款方法"""
with self.lock: # 使用with语句自动管理锁
old_balance = self.balance
time.sleep(0.001) # 模拟处理延迟
self.balance = old_balance + amount
print(f"存款：{amount}，余额：{self.balance}（线程：{threading.current_thread().name}）")
def withdraw(self, amount):
"""取款方法"""
with self.lock:
if self.balance >= amount:
old_balance = self.balance
time.sleep(0.001)
self.balance = old_balance - amount
print(f"取款：{amount}，余额：{self.balance}（线程：{threading.current_thread().name}）")
return True
else:
print(f"取款失败：余额不足（线程：{threading.current_thread().name}）")
return False
def get_balance(self):
"""获取余额"""
with self.lock:
return self.balance
# 创建银行账户
account = BankAccount(1000)
def deposit_worker():
"""存款工作线程"""
for _ in range(5):
amount = random.randint(10, 100)
account.deposit(amount)
time.sleep(random.uniform(0.01, 0.1))
def withdraw_worker():
"""取款工作线程"""
for _ in range(5):
amount = random.randint(10, 50)
account.withdraw(amount)
time.sleep(random.uniform(0.01, 0.1))
# 启动多个存款和取款线程
threads = []
for i in range(3):
t1 = threading.Thread(target=deposit_worker, name=f"存款线程{i+1}")
t2 = threading.Thread(target=withdraw_worker, name=f"取款线程{i+1}")
threads.extend([t1, t2])
for thread in threads:
thread.start()
for thread in threads:
thread.join()
print(f"最终余额：{account.get_balance()}")
---
b.RLock可重入锁
a.RLock特性说明
threading.RLock允许同一线程多次获取同一个锁，避免了死锁问题，适用于递归函数或嵌套调用的情况。
b.RLock使用场景
当一个方法需要调用另一个也需要相同锁的方法时，使用RLock可以避免死锁。
c.代码示例
---
# threading.RLock使用示例
import threading
import time
class RecursiveResource:
"""需要递归访问的资源类"""
def __init__(self, name):
self.name = name
self.lock = threading.RLock() # 使用可重入锁
self.access_count = 0
def access_level1(self):
"""第一级访问"""
with self.lock:
print(f"访问第一级：{self.name}")
self.access_count += 1
time.sleep(0.1)
# 递归调用第二级方法
self.access_level2()
def access_level2(self):
"""第二级访问"""
with self.lock:
print(f"访问第二级：{self.name}")
self.access_count += 1
time.sleep(0.1)
# 再次递归调用
self.access_level3()
def access_level3(self):
"""第三级访问"""
with self.lock:
print(f"访问第三级：{self.name}")
self.access_count += 1
time.sleep(0.1)
def get_access_count(self):
"""获取访问次数"""
with self.lock:
return self.access_count
# 测试可重入锁
resource = RecursiveResource("测试资源")
def worker():
"""工作线程"""
for i in range(2):
print(f"\n线程 {threading.current_thread().name} 开始第{i+1}次访问")
resource.access_level1()
time.sleep(0.2)
# 启动多个线程测试
threads = []
for i in range(3):
t = threading.Thread(target=worker, name=f"工作线程{i+1}")
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"\n总访问次数：{resource.get_access_count()}")
---
d.死锁演示与预防
a死锁产生条件
多个线程相互等待对方持有的锁，导致程序永久阻塞。死锁产生的四个必要条件：互斥、请求与保持、不剥夺、循环等待。
b.预防死锁策略
按固定顺序获取锁、使用超时机制、使用可重入锁、避免嵌套锁。
c.代码示例
---
# 死锁演示与预防示例
import threading
import time
class DeadlockDemo:
"""死锁演示类"""
def __init__(self):
self.lock1 = threading.Lock()
self.lock2 = threading.Lock()
def method1(self):
"""方法1：先获取lock1再获取lock2"""
with self.lock1:
print(f"{threading.current_thread().name} 获取了lock1")
time.sleep(0.1)
print(f"{threading.current_thread().name} 等待lock2")
with self.lock2:
print(f"{threading.current_thread().name} 获取了lock2，执行方法1")
def method2(self):
"""方法2：先获取lock2再获取lock1（可能导致死锁）"""
with self.lock2:
print(f"{threading.current_thread().name} 获取了lock2")
time.sleep(0.1)
print(f"{threading.current_thread().name} 等待lock1")
with self.lock1:
print(f"{threading.current_thread().name} 获取了lock1，执行方法2")
def safe_method1(self):
"""安全方法1：按固定顺序获取锁"""
# 始终先获取lock1，再获取lock2
with self.lock1:
print(f"{threading.current_thread().name} 获取了lock1")
time.sleep(0.1)
print(f"{threading.current_thread().name} 等待lock2")
with self.lock2:
print(f"{threading.current_thread().name} 获取了lock2，执行安全方法1")
def safe_method2(self):
"""安全方法2：按固定顺序获取锁"""
# 同样先获取lock1，再获取lock2
with self.lock1:
print(f"{threading.current_thread().name} 获取了lock1")
time.sleep(0.1)
print(f"{threading.current_thread().name} 等待lock2")
with self.lock2:
print(f"{threading.current_thread().name} 获取了lock2，执行安全方法2")
def deadlock_demo():
"""死锁演示"""
print("=== 死锁演示 ===")
demo = DeadlockDemo()
def worker1():
demo.method1()
def worker2():
demo.method2()
t1 = threading.Thread(target=worker1, name="线程1")
t2 = threading.Thread(target=worker2, name="线程2")
t1.start()
t2.start()
# 设置超时，避免程序永远阻塞
t1.join(timeout=3)
t2.join(timeout=3)
if t1.is_alive() or t2.is_alive():
print("检测到死锁！程序被阻塞。")
# 强制结束演示
return
def safe_demo():
"""安全演示（预防死锁）"""
print("\n=== 安全演示（预防死锁）===")
demo = DeadlockDemo()
def worker1():
demo.safe_method1()
def worker2():
demo.safe_method2()
t1 = threading.Thread(target=worker1, name="线程1")
t2 = threading.Thread(target=worker2, name="线程2")
t1.start()
t2.start()
t1.join()
t2.join()
print("安全演示完成，没有死锁！")
# 运行演示
try:
deadlock_demo()
except:
print("死锁演示异常（预期行为）")
safe_demo()
---
03.条件变量
a.Condition条件变量
a.Condition基本概念
threading.Condition结合了锁和等待/通知机制，允许线程等待特定条件满足后才继续执行，常用于生产者-消费者模式。
b.Condition主要方法
wait()释放锁并等待，notify()唤醒一个等待线程，notify_all()唤醒所有等待线程。
c.代码示例
---
# threading.Condition使用示例
import threading
import time
import random
class BoundedBuffer:
"""有界缓冲区 - 使用Condition实现生产者消费者"""
def __init__(self, max_size=10):
self.max_size = max_size
self.buffer = []
self.condition = threading.Condition()
self.producer_count = 0
self.consumer_count = 0
def produce(self, item):
"""生产方法"""
with self.condition:
# 等待缓冲区有空余空间
while len(self.buffer) >= self.max_size:
print(f"生产者{threading.current_thread().name}：缓冲区已满，等待...")
self.condition.wait()
# 生产物品
self.buffer.append(item)
self.producer_count += 1
print(f"生产者{threading.current_thread().name}：生产了 {item}，缓冲区大小：{len(self.buffer)}")
# 通知消费者
self.condition.notify()
def consume(self):
"""消费方法"""
with self.condition:
# 等待缓冲区有物品
while len(self.buffer) == 0:
print(f"消费者{threading.current_thread().name}：缓冲区为空，等待...")
self.condition.wait()
# 消费物品
item = self.buffer.pop(0)
self.consumer_count += 1
print(f"消费者{threading.current_thread().name}：消费了 {item}，缓冲区大小：{len(self.buffer)}")
# 通知生产者
self.condition.notify()
return item
def get_stats(self):
"""获取统计信息"""
with self.condition:
return {
'buffer_size': len(self.buffer),
'produced': self.producer_count,
'consumed': self.consumer_count
}
# 创建有界缓冲区
buffer = BoundedBuffer(max_size=5)
def producer_worker(producer_id):
"""生产者工作线程"""
for i in range(8):
item = f"物品-{producer_id}-{i+1}"
buffer.produce(item)
time.sleep(random.uniform(0.1, 0.5))
print(f"生产者{producer_id}完成生产")
def consumer_worker(consumer_id):
"""消费者工作线程"""
for i in range(6):
item = buffer.consume()
time.sleep(random.uniform(0.2, 0.6))
print(f"消费者{consumer_id}完成消费")
# 启动生产者和消费者线程
threads = []
# 启动3个生产者
for i in range(3):
t = threading.Thread(target=producer_worker, args=(i+1,), name=f"生产者{i+1}")
threads.append(t)
t.start()
# 启动2个消费者
for i in range(2):
t = threading.Thread(target=consumer_worker, args=(i+1,), name=f"消费者{i+1}")
threads.append(t)
t.start()
# 等待所有线程完成
for t in threads:
t.join()
# 显示最终统计
stats = buffer.get_stats()
print(f"\n最终统计：{stats}")
---
b.Event事件
a.Event基本概念
threading.Event是最简单的线程同步机制，包含一个内部标志，可以设置(set)或清除(clear)，线程可以等待(wait)标志被设置。
b.Event主要用途
用于线程间的简单信号通知，一个或多个线程等待某个事件发生。
c.代码示例
---
# threading.Event使用示例
import threading
import time
class TrafficLight:
"""交通灯系统 - 使用Event同步"""
def __init__(self):
self.red_light = threading.Event()
self.green_light = threading.Event()
self.yellow_light = threading.Event()
# 初始状态：红灯
self.red_light.set()
self.green_light.clear()
self.yellow_light.clear()
self.running = True
def red_light_controller(self):
"""红灯控制器"""
while self.running:
print("🔴 红灯亮起，车辆停止")
self.red_light.set() # 设置红灯标志
time.sleep(5) # 红灯持续5秒
self.red_light.clear() # 清除红灯标志
def yellow_light_controller(self):
"""黄灯控制器"""
while self.running:
self.red_light.wait() # 等待红灯结束
time.sleep(5) # 红灯持续时间
print("🟡 黄灯亮起，准备变换")
self.yellow_light.set() # 设置黄灯标志
time.sleep(2) # 黄灯持续2秒
self.yellow_light.clear() # 清除黄灯标志
def green_light_controller(self):
"""绿灯控制器"""
while self.running:
self.yellow_light.wait() # 等待黄灯结束
time.sleep(2) # 黄灯持续时间
print("🟢 绿灯亮起，车辆通行")
self.green_light.set() # 设置绿灯标志
time.sleep(5) # 绿灯持续5秒
self.green_light.clear() # 清除绿灯标志
def car(self, car_id):
"""车辆线程"""
while self.running:
# 等待绿灯
self.green_light.wait()
print(f"🚗 车辆{car_id}通过路口")
time.sleep(0.5)
# 检查是否应该停止
if not self.green_light.is_set():
print(f"🚗 车辆{car_id}等待红灯")
continue
def pedestrian(self, ped_id):
"""行人线程"""
while self.running:
# 行人在绿灯时过马路
self.green_light.wait()
print(f"🚶 行人{ped_id}正在过马路")
time.sleep(2)
def stop(self):
"""停止交通灯系统"""
self.running = False
def run_simulation(self, duration=30):
"""运行交通灯模拟"""
print("开始交通灯模拟...\n")
# 启动交通灯控制器
controllers = []
controller_funcs = [
self.red_light_controller,
self.yellow_light_controller,
self.green_light_controller
]
for func in controller_funcs:
t = threading.Thread(target=func)
controllers.append(t)
t.start()
# 启动车辆和行人
participants = []
# 启动5辆车
for i in range(5):
t = threading.Thread(target=self.car, args=(i+1,))
participants.append(t)
t.start()
# 启动3个行人
for i in range(3):
t = threading.Thread(target=self.pedestrian, args=(i+1,))
participants.append(t)
t.start()
# 运行指定时间
time.sleep(duration)
self.stop()
# 等待所有线程结束
for t in controllers + participants:
t.join(timeout=1)
print("\n交通灯模拟结束")
# 运行交通灯模拟
traffic_system = TrafficLight()
traffic_system.run_simulation(duration=20)
---
04.线程安全的集合类
a.使用Queue实现的线程安全列表
a.Queue作为列表的替代
虽然Queue主要是队列，但可以通过适当的使用方式实现线程安全的列表操作。
b.代码示例
---
# 使用Queue实现线程安全列表操作
import queue
import threading
import time
class ThreadSafeList:
"""基于Queue的线程安全列表"""
def __init__(self):
self.queue = queue.Queue()
self.size = 0
def append(self, item):
"""添加元素到列表末尾"""
self.queue.put(('append', item))
self.size += 1
def insert(self, index, item):
"""在指定位置插入元素"""
self.queue.put(('insert', (index, item)))
self.size += 1
def remove(self, item):
"""移除指定元素"""
self.queue.put(('remove', item))
if self.size > 0:
self.size -= 1
def pop(self, index=-1):
"""移除并返回指定位置的元素"""
result_queue = queue.Queue()
self.queue.put(('pop', index, result_queue))
if self.size > 0:
self.size -= 1
return result_queue.get()
def get_item(self, index):
"""获取指定位置的元素"""
result_queue = queue.Queue()
self.queue.put(('get', index, result_queue))
return result_queue.get()
def get_all(self):
"""获取所有元素"""
result_queue = queue.Queue()
self.queue.put(('get_all', result_queue))
return result_queue.get()
def length(self):
"""获取列表长度"""
return self.size
def _process_operations(self):
"""处理队列中的操作"""
data = []
while True:
try:
operation = self.queue.get(timeout=0.1)
op_type = operation[0]
if op_type == 'append':
_, item = operation
data.append(item)
elif op_type == 'insert':
_, (index, item) = operation
data.insert(index, item)
elif op_type == 'remove':
_, item = operation
if item in data:
data.remove(item)
elif op_type == 'pop':
_, index, result_queue = operation
if data:
result = data.pop(index)
else:
result = None
result_queue.put(result)
elif op_type == 'get':
_, index, result_queue = operation
if 0 <= index < len(data):
result = data[index]
else:
result = None
result_queue.put(result)
elif op_type == 'get_all':
_, result_queue = operation
result_queue.put(data.copy())
self.queue.task_done()
except queue.Empty:
continue
def start_processor(self):
"""启动操作处理线程"""
processor = threading.Thread(target=self._process_operations, daemon=True)
processor.start()
return processor
# 测试线程安全列表
safe_list = ThreadSafeList()
processor = safe_list.start_processor()
def worker_append(worker_id):
"""工作线程：添加元素"""
for i in range(10):
safe_list.append(f"item-{worker_id}-{i}")
time.sleep(0.01)
def worker_read(worker_id):
"""工作线程：读取元素"""
for i in range(5):
all_items = safe_list.get_all()
print(f"读者{worker_id}看到：{len(all_items)}个元素")
time.sleep(0.02)
# 启动多个工作线程
threads = []
for i in range(3):
t1 = threading.Thread(target=worker_append, args=(i+1,))
t2 = threading.Thread(target=worker_read, args=(i+1,))
threads.extend([t1, t2])
for t in threads:
t.start()
for t in threads:
t.join()
# 显示最终结果
final_items = safe_list.get_all()
print(f"\n最终列表内容：{final_items}")
print(f"最终列表长度：{safe_list.length()}")
---
b.线程安全的字典实现
a字典线程安全的挑战
字典操作通常不是原子的，需要额外的同步机制来保证线程安全。
b.代码示例
---
# 线程安全字典实现
import threading
import time
from contextlib import contextmanager
class ThreadSafeDict:
"""线程安全字典实现"""
def __init__(self, initial_dict=None):
self._dict = dict(initial_dict) if initial_dict else {}
self._lock = threading.RLock() # 使用可重入锁
self._access_count = 0
@contextmanager
def _locked(self):
"""锁上下文管理器"""
self._lock.acquire()
try:
yield
finally:
self._lock.release()
def __setitem__(self, key, value):
"""设置键值"""
with self._locked:
self._dict[key] = value
self._access_count += 1
def __getitem__(self, key):
"""获取值"""
with self._locked:
self._access_count += 1
return self._dict[key]
def __delitem__(self, key):
"""删除键"""
with self._locked:
del self._dict[key]
self._access_count += 1
def get(self, key, default=None):
"""安全获取值"""
with self._locked:
self._access_count += 1
return self._dict.get(key, default)
def items(self):
"""获取所有键值对"""
with self._locked:
self._access_count += 1
return list(self._dict.items())
def keys(self):
"""获取所有键"""
with self._locked:
self._access_count += 1
return list(self._dict.keys())
def values(self):
"""获取所有值"""
with self._locked:
self._access_count += 1
return list(self._dict.values())
def update(self, other_dict):
"""更新字典"""
with self._locked:
self._dict.update(other_dict)
self._access_count += 1
def clear(self):
"""清空字典"""
with self._locked:
self._dict.clear()
self._access_count += 1
def copy(self):
"""复制字典"""
with self._locked:
self._access_count += 1
return self._dict.copy()
def size(self):
"""获取字典大小"""
with self._locked:
return len(self._dict)
def get_access_count(self):
"""获取访问次数"""
with self._locked:
return self._access_count
# 线程安全缓存系统示例
class ThreadSafeCache:
"""线程安全缓存系统"""
def __init__(self, max_size=1000):
self.cache = ThreadSafeDict()
self.max_size = max_size
self.hit_count = 0
self.miss_count = 0
self.stats_lock = threading.Lock()
def get(self, key):
"""获取缓存值"""
value = self.cache.get(key)
if value is not None:
with self.stats_lock:
self.hit_count += 1
print(f"缓存命中：{key} = {value}")
return value
else:
with self.stats_lock:
self.miss_count += 1
print(f"缓存未命中：{key}")
return None
def put(self, key, value):
"""设置缓存值"""
if self.cache.size() >= self.max_size:
# 简单的LRU：删除第一个元素
keys = self.cache.keys()
if keys:
del self.cache[keys[0]]
print(f"缓存已满，删除最旧项：{keys[0]}")
self.cache[key] = value
print(f"缓存设置：{key} = {value}")
def get_stats(self):
"""获取缓存统计"""
with self.stats_lock:
total_requests = self.hit_count + self.miss_count
hit_rate = (self.hit_count / total_requests * 100) if total_requests > 0 else 0
return {
'size': self.cache.size(),
'hits': self.hit_count,
'misses': self.miss_count,
'hit_rate': f"{hit_rate:.2f}%"
}
# 测试线程安全缓存
cache = ThreadSafeCache(max_size=5)
def cache_reader(reader_id):
"""缓存读取线程"""
keys = ['user1', 'user2', 'user3', 'user4', 'user5']
for i in range(8):
key = keys[i % len(keys)]
value = cache.get(key)
if value is None:
# 模拟从数据库加载
simulated_value = f"data_for_{key}"
cache.put(key, simulated_value)
time.sleep(0.1)
def cache_writer(writer_id):
"""缓存写入线程"""
for i in range(4):
key = f"new_data_{writer_id}_{i}"
value = f"value_{i}_by_writer_{writer_id}"
cache.put(key, value)
time.sleep(0.15)
# 启动多个线程测试缓存
threads = []
for i in range(3):
t1 = threading.Thread(target=cache_reader, args=(i+1,))
t2 = threading.Thread(target=cache_writer, args=(i+1,))
threads.extend([t1, t2])
for t in threads:
t.start()
for t in threads:
t.join()
# 显示缓存统计
stats = cache.get_stats()
print(f"\n缓存统计：{stats}")
print(f"缓存内容：{dict(cache.cache.items())}")
---
5.3 管道Pipe
01.Pipe管道概述
a.基本概念
Pipe（管道）是multiprocessing模块提供的进程间通信机制，用于在父子进程之间或任意两个进程之间建立双向数据通道。虽然在threading模块中没有直接的Pipe实现，但我们可以通过queue.Queue模拟管道通信机制。Pipe提供了连接对象(conn1, conn2)，每个连接对象都可以发送和接收数据，是实现进程间数据传输的高效方式。
b.Pipe的主要特点
a.双向通信能力
管道连接的两个端点都可以发送和接收数据，支持全双工通信模式。
b.高效的数据传输
相比共享内存和文件，管道提供了更简洁的数据传输接口，自动处理同步问题。
c.灵活的通信模式
可以实现一对一通信，也可以构建更复杂的通信网络拓扑。
02.基于Queue的线程管道实现
a.模拟管道通信机制
虽然Python的threading模块没有直接提供Pipe，但我们可以使用Queue来实现类似的管道通信功能，在线程间建立高效的数据通道。
b.单向管道实现
a基础实现原理
使用两个Queue分别实现两个方向的数据传输，每个Queue充当一个单向管道。
b.代码示例
---
# 基于Queue的单向管道实现
import queue
import threading
import time
class UnidirectionalPipe:
"""单向管道类"""
def __init__(self):
self.data_queue = queue.Queue()
self.closed = False
self.lock = threading.Lock()
def send(self, data):
"""发送数据"""
with self.lock:
if self.closed:
raise ConnectionError("管道已关闭")
self.data_queue.put(data)
def receive(self, timeout=None):
"""接收数据"""
try:
data = self.data_queue.get(timeout=timeout)
return data
except queue.Empty:
return None
def close(self):
"""关闭管道"""
with self.lock:
self.closed = True
def is_closed(self):
"""检查管道是否关闭"""
with self.lock:
return self.closed
class BidirectionalPipe:
"""双向管道类 - 使用两个单向管道"""
def __init__(self):
# 创建两个单向管道，分别用于两个方向的数据传输
self.pipe1_to_2 = UnidirectionalPipe() # 端点1到端点2的管道
self.pipe2_to_1 = UnidirectionalPipe() # 端点2到端点1的管道
def get_endpoint1(self):
"""获取第一个端点"""
return PipeEndpoint(self.pipe1_to_2, self.pipe2_to_1)
def get_endpoint2(self):
"""获取第二个端点"""
return PipeEndpoint(self.pipe2_to_1, self.pipe1_to_2)
def close(self):
"""关闭整个管道"""
self.pipe1_to_2.close()
self.pipe2_to_1.close()
class PipeEndpoint:
"""管道端点类"""
def __init__(self, send_pipe, receive_pipe):
self.send_pipe = send_pipe
self.receive_pipe = receive_pipe
self.endpoint_id = id(self)
def send(self, data):
"""发送数据"""
self.send_pipe.send(data)
def receive(self, timeout=None):
"""接收数据"""
return self.receive_pipe.receive(timeout)
def close(self):
"""关闭端点"""
self.send_pipe.close()
def __str__(self):
return f"PipeEndpoint-{self.endpoint_id}"
# 线程间通信示例
def message_sender(endpoint, message_count):
"""消息发送者线程"""
for i in range(message_count):
message = f"消息 {i+1} 来自 {endpoint}"
try:
endpoint.send(message)
print(f"{endpoint} 发送：{message}")
time.sleep(0.1)
except ConnectionError:
print(f"{endpoint} 发送失败：管道已关闭")
break
def message_receiver(endpoint, message_count):
"""消息接收者线程"""
received_count = 0
while received_count < message_count:
try:
message = endpoint.receive(timeout=2.0)
if message is not None:
print(f"{endpoint} 接收：{message}")
received_count += 1
except:
print(f"{endpoint} 接收超时")
break
# 创建双向管道
bidirectional_pipe = BidirectionalPipe()
endpoint1 = bidirectional_pipe.get_endpoint1()
endpoint2 = bidirectional_pipe.get_endpoint2()
# 启动通信线程
threads = []
# 端点1发送，端点2接收
t1 = threading.Thread(target=message_sender, args=(endpoint1, 5))
t2 = threading.Thread(target=message_receiver, args=(endpoint2, 5))
threads.extend([t1, t2])
# 端点2发送，端点1接收
t3 = threading.Thread(target=message_sender, args=(endpoint2, 3))
t4 = threading.Thread(target=message_receiver, args=(endpoint1, 3))
threads.extend([t3, t4])
for t in threads:
t.start()
for t in threads:
t.join()
bidirectional_pipe.close()
print("双向管道通信完成")
---
03.高级管道通信模式
a.多对多通信网络
a星型网络拓扑
使用中心节点作为消息路由器，实现多个线程间的间接通信。
b.代码示例
---
# 基于Queue的多对多通信网络
import queue
import threading
import time
from collections import defaultdict
class MessageRouter:
"""消息路由器 - 中心节点"""
def __init__(self):
self.queues = {} # 每个客户端的接收队列
self.registrations = queue.Queue()
self.messages = queue.Queue()
self.running = False
self.router_thread = None
def register_client(self, client_id):
"""注册客户端"""
client_queue = queue.Queue()
self.queues[client_id] = client_queue
print(f"路由器：客户端 {client_id} 已注册")
return client_queue
def unregister_client(self, client_id):
"""注销客户端"""
if client_id in self.queues:
del self.queues[client_id]
print(f"路由器：客户端 {client_id} 已注销")
def send_message(self, sender_id, receiver_id, message):
"""发送消息"""
msg = {
'sender': sender_id,
'receiver': receiver_id,
'message': message,
'timestamp': time.time()
}
self.messages.put(msg)
def broadcast_message(self, sender_id, message):
"""广播消息"""
msg = {
'sender': sender_id,
'receiver': 'broadcast',
'message': message,
'timestamp': time.time()
}
self.messages.put(msg)
def _router_worker(self):
"""路由器工作线程"""
print("消息路由器启动")
while self.running:
try:
# 处理消息
message = self.messages.get(timeout=1.0)
receiver_id = message['receiver']
if receiver_id == 'broadcast':
# 广播消息
for client_id, client_queue in self.queues.items():
if client_id != message['sender']: # 不发送给发送者
client_queue.put(message)
print(f"路由器：广播消息来自 {message['sender']}")
elif receiver_id in self.queues:
# 点对点消息
self.queues[receiver_id].put(message)
print(f"路由器：转发消息 {message['sender']} -> {receiver_id}")
else:
print(f"路由器：接收者 {receiver_id} 不存在")
except queue.Empty:
continue
print("消息路由器停止")
def start(self):
"""启动路由器"""
self.running = True
self.router_thread = threading.Thread(target=self._router_worker)
self.router_thread.start()
def stop(self):
"""停止路由器"""
self.running = False
if self.router_thread:
self.router_thread.join()
print("消息路由器已停止")
class NetworkClient:
"""网络客户端"""
def __init__(self, client_id, router):
self.client_id = client_id
self.router = router
self.receive_queue = router.register_client(client_id)
self.running = False
self.client_thread = None
def send_to(self, receiver_id, message):
"""发送消息给指定客户端"""
full_message = f"[{self.client_id}] {message}"
self.router.send_message(self.client_id, receiver_id, full_message)
def broadcast(self, message):
"""广播消息"""
full_message = f"[{self.client_id}] {message}"
self.router.broadcast_message(self.client_id, full_message)
def receive_messages(self):
"""接收所有消息"""
messages = []
while not self.receive_queue.empty():
try:
msg = self.receive_queue.get_nowait()
messages.append(msg)
except queue.Empty:
break
return messages
def _message_listener(self):
"""消息监听线程"""
print(f"客户端 {self.client_id} 消息监听器启动")
while self.running:
try:
message = self.receive_queue.get(timeout=1.0)
print(f"客户端 {self.client_id} 收到：来自 {message['sender']} - {message['message']}")
except queue.Empty:
continue
print(f"客户端 {self.client_id} 消息监听器停止")
def start(self):
"""启动客户端"""
self.running = True
self.client_thread = threading.Thread(target=self._message_listener)
self.client_thread.start()
def stop(self):
"""停止客户端"""
self.running = False
self.router.unregister_client(self.client_id)
if self.client_thread:
self.client_thread.join()
print(f"客户端 {self.client_id} 已停止")
# 多对多通信网络测试
def client_chat_simulation(client_id, router):
"""客户端聊天模拟"""
client = NetworkClient(client_id, router)
client.start()
# 等待所有客户端启动
time.sleep(1)
# 发送一些消息
if client_id == 'client1':
client.send_to('client2', '你好，Client2！')
time.sleep(0.5)
client.send_to('client3', '你好，Client3！')
time.sleep(0.5)
client.broadcast('大家好，我是Client1')
elif client_id == 'client2':
client.send_to('client1', '你好，Client1！很高兴收到你的消息')
time.sleep(0.5)
client.broadcast('Client2也在这里！')
elif client_id == 'client3':
client.send_to('client1', '你好Client1！')
time.sleep(0.5)
client.send_to('client2', '你好Client2！')
# 运行一段时间后停止
time.sleep(3)
client.stop()
# 创建消息路由器
router = MessageRouter()
router.start()
# 创建多个客户端
clients = ['client1', 'client2', 'client3']
client_threads = []
for client_id in clients:
t = threading.Thread(target=client_chat_simulation, args=(client_id, router))
client_threads.append(t)
t.start()
# 等待所有客户端完成
for t in client_threads:
t.join()
# 停止路由器
router.stop()
print("多对多通信网络测试完成")
---
b.请求-响应模式
a同步通信机制
实现基于管道的同步请求-响应通信，客户端发送请求后等待服务器响应。
b.代码示例
---
# 请求-响应模式的管道通信
import queue
import threading
import time
import uuid
from dataclasses import dataclass
from typing import Any, Optional
@dataclass
class RPCMessage:
"""RPC消息类"""
message_id: str
method: str
params: dict
result: Optional[Any] = None
error: Optional[str] = None
is_request: bool = True
class RPCPipe:
"""RPC管道通信"""
def __init__(self):
self.request_queue = queue.Queue()
self.response_queues = {} # message_id -> response_queue
self.lock = threading.Lock()
self.server_thread = None
self.running = False
def register_method(self, method_name, method_func):
"""注册RPC方法"""
if not hasattr(self, 'methods'):
self.methods = {}
self.methods[method_name] = method_func
def _server_worker(self):
"""RPC服务器工作线程"""
print("RPC服务器启动")
while self.running:
try:
# 接收请求
message = self.request_queue.get(timeout=1.0)
if not message.is_request:
continue
print(f"RPC服务器：收到请求 {message.method}")
# 处理请求
try:
if hasattr(self, 'methods') and message.method in self.methods:
result = self.methods[message.method](**message.params)
response = RPCMessage(
message_id=message.message_id,
method=message.method,
params={},
result=result,
is_request=False
)
else:
response = RPCMessage(
message_id=message.message_id,
method=message.method,
params={},
error=f"未知方法：{message.method}",
is_request=False
)
except Exception as e:
response = RPCMessage(
message_id=message.message_id,
method=message.method,
params={},
error=str(e),
is_request=False
)
# 发送响应
with self.lock:
if message.message_id in self.response_queues:
self.response_queues[message.message_id].put(response)
del self.response_queues[message.message_id]
except queue.Empty:
continue
print("RPC服务器停止")
def call(self, method, params=None, timeout=5.0):
"""同步RPC调用"""
if params is None:
params = {}
message_id = str(uuid.uuid4())
request = RPCMessage(
message_id=message_id,
method=method,
params=params
)
# 创建响应队列
response_queue = queue.Queue()
with self.lock:
self.response_queues[message_id] = response_queue
# 发送请求
self.request_queue.put(request)
# 等待响应
try:
response = response_queue.get(timeout=timeout)
if response.error:
raise Exception(f"RPC错误：{response.error}")
return response.result
except queue.Empty:
raise TimeoutError(f"RPC调用超时：{method}")
def start_server(self):
"""启动RPC服务器"""
self.running = True
self.server_thread = threading.Thread(target=self._server_worker)
self.server_thread.start()
def stop_server(self):
"""停止RPC服务器"""
self.running = False
if self.server_thread:
self.server_thread.join()
# RPC服务定义
class CalculatorService:
"""计算服务"""
@staticmethod
def add(a, b):
return a + b
@staticmethod
def subtract(a, b):
return a - b
@staticmethod
def multiply(a, b):
return a * b
@staticmethod
def divide(a, b):
if b == 0:
raise ValueError("除数不能为零")
return a / b
@staticmethod
def factorial(n):
"""计算阶乘（耗时操作示例）"""
if n < 0:
raise ValueError("阶乘只支持非负整数")
if n == 0 or n == 1:
return 1
result = 1
for i in range(2, n + 1):
result *= i
time.sleep(0.1) # 模拟耗时操作
return result
class StringService:
"""字符串服务"""
@staticmethod
def reverse(text):
return text[::-1]
@staticmethod
def uppercase(text):
return text.upper()
@staticmethod
def lowercase(text):
return text.lower()
@staticmethod
def length(text):
return len(text)
# 测试RPC通信
def rpc_client_worker(client_id, rpc_pipe):
"""RPC客户端工作线程"""
print(f"RPC客户端 {client_id} 启动")
# 测试计算服务
try:
result = rpc_pipe.call("add", {"a": 10, "b": 5})
print(f"客户端 {client_id}：10 + 5 = {result}")
result = rpc_pipe.call("multiply", {"a": 6, "b": 7})
print(f"客户端 {client_id}：6 × 7 = {result}")
result = rpc_pipe.call("divide", {"a": 20, "b": 4})
print(f"客户端 {client_id}：20 ÷ 4 = {result}")
# 测试字符串服务
result = rpc_pipe.call("reverse", {"text": "Hello World"})
print(f"客户端 {client_id}：reverse('Hello World') = '{result}'")
result = rpc_pipe.call("uppercase", {"text": "hello"})
print(f"客户端 {client_id}：uppercase('hello') = '{result}'")
# 测试耗时操作
result = rpc_pipe.call("factorial", {"n": 5})
print(f"客户端 {client_id}：factorial(5) = {result}")
except Exception as e:
print(f"客户端 {client_id} RPC调用失败：{e}")
print(f"RPC客户端 {client_id} 完成")
# 创建RPC管道并注册服务
rpc_pipe = RPCPipe()
# 注册计算服务
calculator = CalculatorService()
rpc_pipe.register_method("add", calculator.add)
rpc_pipe.register_method("subtract", calculator.subtract)
rpc_pipe.register_method("multiply", calculator.multiply)
rpc_pipe.register_method("divide", calculator.divide)
rpc_pipe.register_method("factorial", calculator.factorial)
# 注册字符串服务
string_service = StringService()
rpc_pipe.register_method("reverse", string_service.reverse)
rpc_pipe.register_method("uppercase", string_service.uppercase)
rpc_pipe.register_method("lowercase", string_service.lowercase)
rpc_pipe.register_method("length", string_service.length)
# 启动RPC服务器
rpc_pipe.start_server()
# 启动多个客户端
client_threads = []
for i in range(3):
t = threading.Thread(target=rpc_client_worker, args=(f"client{i+1}", rpc_pipe))
client_threads.append(t)
t.start()
# 等待所有客户端完成
for t in client_threads:
t.join()
# 停止RPC服务器
rpc_pipe.stop_server()
print("RPC通信测试完成")
---
04.管道性能优化与错误处理
a.缓冲区管理策略
a动态缓冲区调整
根据网络负载和数据传输速度动态调整缓冲区大小，优化内存使用和传输效率。
b.代码示例
---
# 智能缓冲区管理的高性能管道
import queue
import threading
import time
from collections import deque
class SmartBuffer:
"""智能缓冲区"""
def __init__(self, initial_size=100):
self.buffer = deque(maxlen=initial_size)
self.max_size = initial_size
self.access_times = deque(maxlen=50) # 最近的访问时间
self.adjustment_threshold = 10 # 调整阈值
self.lock = threading.Lock()
self.stats = {
'put_count': 0,
'get_count': 0,
'adjustments': 0,
'overflows': 0
}
def put(self, item):
"""放入项目"""
with self.lock:
self.access_times.append(time.time())
self.stats['put_count'] += 1
try:
self.buffer.append(item)
except IndexError:
# 缓冲区满，触发调整
self._adjust_size(increase=True)
self.stats['overflows'] += 1
self.buffer.append(item)
def get(self):
"""获取项目"""
with self.lock:
self.access_times.append(time.time())
self.stats['get_count'] += 1
if self.buffer:
return self.buffer.popleft()
return None
def _adjust_size(self, increase=True):
"""调整缓冲区大小"""
old_size = self.max_size
if increase:
self.max_size = int(self.max_size * 1.5)
else:
self.max_size = max(10, int(self.max_size * 0.7))
# 创建新的缓冲区
new_buffer = deque(self.buffer, maxlen=self.max_size)
self.buffer = new_buffer
self.stats['adjustments'] += 1
print(f"缓冲区大小调整：{old_size} -> {self.max_size}")
def auto_adjust(self):
"""自动调整缓冲区大小"""
with self.lock:
if len(self.access_times) < 2:
return
# 计算访问频率
current_time = time.time()
recent_accesses = [t for t in self.access_times
if current_time - t < 5.0] # 最近5秒
access_rate = len(recent_accesses) / 5.0
# 根据访问率调整大小
current_usage = len(self.buffer) / self.max_size
if access_rate > self.adjustment_threshold and current_usage > 0.8:
# 高访问率且缓冲区快满了，增大缓冲区
self._adjust_size(increase=True)
elif access_rate < self.adjustment_threshold / 2 and current_usage < 0.3:
# 低访问率且缓冲区很空，减小缓冲区
self._adjust_size(increase=False)
def get_stats(self):
"""获取统计信息"""
with self.lock:
return {
'buffer_size': len(self.buffer),
'max_size': self.max_size,
'usage_rate': len(self.buffer) / self.max_size,
**self.stats
}
class HighPerformancePipe:
"""高性能管道实现"""
def __init__(self, buffer_size=100):
self.send_buffer = SmartBuffer(buffer_size)
self.receive_buffer = SmartBuffer(buffer_size)
self.closed = False
self.lock = threading.Lock()
self.transfer_thread = None
self.stats = {
'bytes_transferred': 0,
'messages_transferred': 0,
'errors': 0
}
def send(self, data):
"""发送数据"""
with self.lock:
if self.closed:
raise ConnectionError("管道已关闭")
try:
# 模拟序列化
serialized_data = str(data)
message = {
'data': serialized_data,
'timestamp': time.time(),
'size': len(serialized_data)
}
self.send_buffer.put(message)
return True
except Exception as e:
self.stats['errors'] += 1
print(f"发送错误：{e}")
return False
def receive(self, timeout=None):
"""接收数据"""
start_time = time.time()
while True:
with self.lock:
if self.closed:
return None
message = self.receive_buffer.get()
if message:
self.stats['bytes_transferred'] += message['size']
self.stats['messages_transferred'] += 1
return eval(message['data']) # 模拟反序列化
# 检查超时
if timeout and (time.time() - start_time) > timeout:
return None
time.sleep(0.01) # 短暂休眠
def _transfer_worker(self):
"""数据传输工作线程"""
print("高性能管道传输线程启动")
while not self.closed:
try:
# 从发送缓冲区获取数据
message = self.send_buffer.get()
if message:
# 放入接收缓冲区
self.receive_buffer.put(message)
# 定期调整缓冲区大小
self.send_buffer.auto_adjust()
self.receive_buffer.auto_adjust()
time.sleep(0.001) # 高频检查
except Exception as e:
self.stats['errors'] += 1
print(f"传输错误：{e}")
print("高性能管道传输线程停止")
def start(self):
"""启动管道"""
if not self.transfer_thread:
self.transfer_thread = threading.Thread(target=self._transfer_worker)
self.transfer_thread.start()
def stop(self):
"""停止管道"""
with self.lock:
self.closed = True
if self.transfer_thread:
self.transfer_thread.join()
def get_stats(self):
"""获取性能统计"""
return {
**self.stats,
'send_buffer': self.send_buffer.get_stats(),
'receive_buffer': self.receive_buffer.get_stats()
}
# 高性能管道测试
def performance_test():
"""性能测试"""
print("开始高性能管道测试...")
# 创建管道
pipe = HighPerformancePipe(buffer_size=50)
pipe.start()
# 启动发送者线程
def sender_worker(sender_id, message_count):
"""发送者工作线程"""
sent_count = 0
for i in range(message_count):
message = f"消息 {i+1} 来自发送者{sender_id}"
if pipe.send(message):
sent_count += 1
time.sleep(0.01) # 控制发送速率
print(f"发送者{sender_id} 发送了 {sent_count} 条消息")
def receiver_worker(receiver_id, expected_count):
"""接收者工作线程"""
received_count = 0
start_time = time.time()
while received_count < expected_count:
message = pipe.receive(timeout=0.1)
if message:
received_count += 1
if received_count % 100 == 0:
print(f"接收者{receiver_id} 已接收 {received_count} 条消息")
end_time = time.time()
print(f"接收者{receiver_id} 接收了 {received_count} 条消息，耗时 {end_time - start_time:.2f} 秒")
def monitor_worker():
"""监控工作线程"""
for i in range(20):
time.sleep(1)
stats = pipe.get_stats()
print(f"监控第{i+1}秒："
f" 传输消息数 {stats['messages_transferred']}, "
f" 传输字节数 {stats['bytes_transferred']}, "
f" 错误数 {stats['errors']}")
# 启动多个发送者和接收者
threads = []
# 3个发送者
for i in range(3):
t = threading.Thread(target=sender_worker, args=(i+1, 300))
threads.append(t)
t.start()
# 2个接收者
total_expected = 900 # 3个发送者 × 300条消息
for i in range(2):
t = threading.Thread(target=receiver_worker, args=(i+1, total_expected // 2))
threads.append(t)
t.start()
# 监控线程
monitor_t = threading.Thread(target=monitor_worker)
threads.append(monitor_t)
monitor_t.start()
# 等待所有线程完成
for t in threads[:-1]: # 除了监控线程
t.join()
# 停止管道
time.sleep(2)
pipe.stop()
# 显示最终统计
final_stats = pipe.get_stats()
print(f"\n最终性能统计：{final_stats}")
# 运行性能测试
performance_test()
---
b.错误处理和恢复机制
a连接异常处理
实现管道断开检测、自动重连和数据完整性验证。
b.代码示例
---
# 带错误处理的健壮管道实现
import queue
import threading
import time
import hashlib
import pickle
class PipeError(Exception):
"""管道错误基类"""
pass
class ConnectionBrokenError(PipeError):
"""连接断开错误"""
pass
class DataCorruptionError(PipeError):
"""数据损坏错误"""
pass
class RobustPipe:
"""健壮的管道实现"""
def __init__(self, max_retries=3, retry_delay=0.1):
self.send_queue = queue.Queue()
self.receive_queue = queue.Queue()
self.closed = False
self.lock = threading.Lock()
self.max_retries = max_retries
self.retry_delay = retry_delay
self.stats = {
'messages_sent': 0,
'messages_received': 0,
'retries': 0,
'errors': 0,
'corruption_detected': 0
}
# 传输线程
self.transfer_thread = None
self.monitor_thread = None
def _calculate_checksum(self, data):
"""计算数据校验和"""
serialized = pickle.dumps(data)
return hashlib.md5(serialized).hexdigest()
def _wrap_message(self, data):
"""包装消息（添加校验和）"""
checksum = self._calculate_checksum(data)
return {
'data': data,
'checksum': checksum,
'timestamp': time.time(),
'message_id': id(data)
}
def _unwrap_message(self, wrapped_message):
"""解包消息（验证校验和）"""
try:
data = wrapped_message['data']
received_checksum = wrapped_message['checksum']
calculated_checksum = self._calculate_checksum(data)
if received_checksum != calculated_checksum:
raise DataCorruptionError("数据校验失败，可能已损坏")
return data
except (KeyError, pickle.PickleError) as e:
raise DataCorruptionError(f"消息格式错误：{e}")
def send(self, data):
"""发送数据（带重试机制）"""
for attempt in range(self.max_retries + 1):
try:
with self.lock:
if self.closed:
raise ConnectionBrokenError("管道已关闭")
wrapped_message = self._wrap_message(data)
self.send_queue.put(wrapped_message, timeout=1.0)
self.stats['messages_sent'] += 1
return True
except queue.Full:
if attempt < self.max_retries:
self.stats['retries'] += 1
print(f"发送队列满，重试 {attempt + 1}/{self.max_retries}")
time.sleep(self.retry_delay * (2 ** attempt)) # 指数退避
else:
self.stats['errors'] += 1
raise PipeError("发送失败：队列已满")
except Exception as e:
self.stats['errors'] += 1
raise PipeError(f"发送失败：{e}")
return False
def receive(self, timeout=None):
"""接收数据（带超时和验证）"""
start_time = time.time()
while True:
try:
with self.lock:
if self.closed:
raise ConnectionBrokenError("管道已关闭")
# 尝试接收消息
wrapped_message = self.receive_queue.get(timeout=0.1)
if wrapped_message:
# 验证消息完整性
data = self._unwrap_message(wrapped_message)
self.stats['messages_received'] += 1
return data
except queue.Empty:
# 检查超时
if timeout and (time.time() - start_time) > timeout:
raise TimeoutError("接收超时")
continue
except DataCorruptionError as e:
self.stats['corruption_detected'] += 1
print(f"数据损坏检测：{e}")
# 继续尝试接收下一条消息
continue
except Exception as e:
self.stats['errors'] += 1
raise PipeError(f"接收失败：{e}")
def _transfer_worker(self):
"""传输工作线程"""
print("健壮管道传输线程启动")
consecutive_errors = 0
max_consecutive_errors = 5
while not self.closed:
try:
# 从发送队列获取消息
wrapped_message = self.send_queue.get(timeout=1.0)
# 模拟网络传输（可能失败）
if self._simulate_transmission_failure():
raise ConnectionBrokenError("模拟传输失败")
# 放入接收队列
self.receive_queue.put(wrapped_message)
consecutive_errors = 0 # 重置错误计数
except queue.Empty:
continue
except Exception as e:
consecutive_errors += 1
self.stats['errors'] += 1
print(f"传输错误（连续第{consecutive_errors}次）：{e}")
# 连续错误过多，暂停传输
if consecutive_errors >= max_consecutive_errors:
print("连续错误过多，暂停传输")
time.sleep(1.0)
consecutive_errors = 0
print("健壮管道传输线程停止")
def _simulate_transmission_failure(self):
"""模拟传输失败（1%概率）"""
import random
return random.random() < 0.01
def _monitor_worker(self):
"""监控工作线程"""
print("健壮管道监控线程启动")
last_stats_time = time.time()
while not self.closed:
time.sleep(5.0) # 每5秒报告一次统计
with self.lock:
current_time = time.time()
time_interval = current_time - last_stats_time
# 计算性能指标
send_rate = self.stats['messages_sent'] / time_interval if time_interval > 0 else 0
receive_rate = self.stats['messages_received'] / time_interval if time_interval > 0 else 0
error_rate = self.stats['errors'] / (self.stats['messages_sent'] + 1)
print(f"管道统计（过去{time_interval:.1f}秒）："
f" 发送率 {send_rate:.1f} msg/s, "
f" 接收率 {receive_rate:.1f} msg/s, "
f" 错误率 {error_rate:.2%}, "
f" 重试次数 {self.stats['retries']}, "
f" 数据损坏 {self.stats['corruption_detected']}")
last_stats_time = current_time
def start(self):
"""启动管道"""
with self.lock:
if self.closed:
return False
self.transfer_thread = threading.Thread(target=self._transfer_worker)
self.monitor_thread = threading.Thread(target=self._monitor_worker)
self.transfer_thread.start()
self.monitor_thread.start()
return True
def stop(self):
"""停止管道"""
with self.lock:
self.closed = True
if self.transfer_thread:
self.transfer_thread.join()
if self.monitor_thread:
self.monitor_thread.join()
print("健壮管道已停止")
def get_stats(self):
"""获取统计信息"""
with self.lock:
return self.stats.copy()
# 健壮管道测试
def robust_pipe_test():
"""健壮管道测试"""
print("开始健壮管道测试...")
# 创建管道
pipe = RobustPipe(max_retries=5, retry_delay=0.05)
pipe.start()
# 测试数据
test_data = [
"简单字符串",
{"key": "value", "number": 42},
[1, 2, 3, 4, 5],
("tuple", "data"),
{"complex": {"nested": {"structure": True}}}
]
# 发送者线程
def sender():
"""发送者线程"""
for i, data in enumerate(test_data * 10): # 重复发送
try:
success = pipe.send(data)
if success:
print(f"发送成功：{type(data).__name__} - 第{i+1}次")
else:
print(f"发送失败：{type(data).__name__} - 第{i+1}次")
time.sleep(0.02)
except PipeError as e:
print(f"发送错误：{e}")
print("发送者完成")
# 接收者线程
def receiver():
"""接收者线程"""
received_count = 0
start_time = time.time()
while received_count < len(test_data) * 10:
try:
data = pipe.receive(timeout=1.0)
if data is not None:
received_count += 1
if received_count % 10 == 0:
print(f"接收成功：{type(data).__name__} - 第{received_count}次")
except (TimeoutError, PipeError) as e:
print(f"接收错误：{e}")
end_time = time.time()
print(f"接收者完成，接收了 {received_count} 条消息，耗时 {end_time - start_time:.2f} 秒")
# 启动发送者和接收者
sender_thread = threading.Thread(target=sender)
receiver_thread = threading.Thread(target=receiver)
sender_thread.start()
receiver_thread.start()
# 等待完成
sender_thread.join()
receiver_thread.join()
# 停止管道
time.sleep(2)
pipe.stop()
# 显示最终统计
final_stats = pipe.get_stats()
print(f"\n最终统计：{final_stats}")
# 运行健壮管道测试
robust_pipe_test()
---
5.4 共享变量与锁
01.共享变量基础概念
a.共享变量的定义与风险
共享变量是指多个线程可以同时访问和修改的全局变量或对象属性。在多线程环境中，如果不使用适当的同步机制，共享变量的并发访问会导致数据竞争、竞态条件和不一致状态。Python中的全局变量、类属性、模块级变量等都可能成为共享变量，需要特别注意线程安全问题。
b.数据竞争与竞态条件
a数据竞争产生原因
当多个线程同时读写同一个变量，且至少有一个线程在写入数据时，就会产生数据竞争。由于Python的解释器执行机制（GIL的存在使得单个字节码操作是原子的，但复合操作不是），即使看起来简单的操作也可能不是原子的。
b竞态条件识别
竞态条件是指程序的正确性依赖于事件发生的相对时间顺序。在多线程环境中，由于线程调度的不可预测性，竞态条件可能导致程序在不同运行时产生不同结果。
02.全局变量保护策略
a.全局变量的线程安全问题
全局变量在多线程环境中是最容易出问题的共享资源，因为所有线程都可以直接访问和修改。即使是简单的自增操作（如 counter += 1）在多线程环境中也不是原子的，可能导致数据丢失。
b.代码示例
---
# 全局变量线程安全问题演示
import threading
import time
# 全局计数器（线程不安全）
unsafe_counter = 0
safe_counter = 0
counter_lock = threading.Lock()
def unsafe_increment():
"""线程不安全的计数器增加"""
global unsafe_counter
# 这个操作看起来是原子的，但实际上不是
# 它包含：读取unsafe_counter -> 加1 -> 写回unsafe_counter
for _ in range(1000):
unsafe_counter += 1
def safe_increment():
"""线程安全的计数器增加"""
global safe_counter
for _ in range(1000):
with counter_lock:
safe_counter += 1
def test_unsafe_counter():
"""测试不安全的计数器"""
global unsafe_counter
unsafe_counter = 0
threads = []
for i in range(10):
t = threading.Thread(target=unsafe_increment)
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"不安全计数器结果：{unsafe_counter}（期望值：10000）")
def test_safe_counter():
"""测试安全的计数器"""
global safe_counter
safe_counter = 0
threads = []
for i in range(10):
t = threading.Thread(target=safe_increment)
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"安全计数器结果：{safe_counter}（期望值：10000）")
# 运行测试
print("=== 全局变量线程安全测试 ===")
test_unsafe_counter()
test_safe_counter()
---
c.全局变量最佳实践
a.减少全局变量使用
尽量使用局部变量和参数传递，减少对全局变量的依赖。如果必须使用全局变量，确保使用适当的锁机制保护。
b.代码示例
---
# 全局变量最佳实践示例
import threading
import time
class ThreadSafeGlobalState:
"""线程安全的全局状态管理器"""
def __init__(self):
self._data = {}
self._lock = threading.RLock() # 使用可重入锁
self._access_count = 0
def get(self, key, default=None):
"""安全获取值"""
with self._lock:
self._access_count += 1
return self._data.get(key, default)
def set(self, key, value):
"""安全设置值"""
with self._lock:
self._access_count += 1
self._data[key] = value
def update(self, updates):
"""批量更新"""
with self._lock:
self._access_count += 1
self._data.update(updates)
def delete(self, key):
"""安全删除键"""
with self._lock:
self._access_count += 1
if key in self._data:
del self._data[key]
return True
return False
def get_all(self):
"""获取所有数据的副本"""
with self._lock:
self._access_count += 1
return self._data.copy()
def get_access_count(self):
"""获取访问次数"""
with self._lock:
return self._access_count
# 全局状态管理器实例
global_state = ThreadSafeGlobalState()
def worker_function(worker_id, operations):
"""工作函数，模拟对全局状态的访问"""
for i in range(operations):
# 执行各种操作
key = f"key_{worker_id}_{i % 5}"
value = f"value_{worker_id}_{i}"
# 随机操作类型
import random
operation = random.choice(['set', 'get', 'update', 'delete'])
if operation == 'set':
global_state.set(key, value)
print(f"工作者{worker_id}：设置 {key} = {value}")
elif operation == 'get':
result = global_state.get(key)
print(f"工作者{worker_id}：获取 {key} = {result}")
elif operation == 'update':
updates = {
f"{key}_1": f"{value}_1",
f"{key}_2": f"{value}_2"
}
global_state.update(updates)
print(f"工作者{worker_id}：更新 {key}")
elif operation == 'delete':
deleted = global_state.delete(key)
print(f"工作者{worker_id}：删除 {key} - {'成功' if deleted else '失败'}")
time.sleep(0.01) # 模拟处理时间
# 测试线程安全的全局状态管理
def test_global_state():
"""测试全局状态管理"""
print("=== 线程安全全局状态管理测试 ===")
# 启动多个工作线程
threads = []
for i in range(5):
t = threading.Thread(target=worker_function, args=(i+1, 20))
threads.append(t)
t.start()
# 等待所有线程完成
for t in threads:
t.join()
# 显示最终状态
final_state = global_state.get_all()
access_count = global_state.get_access_count()
print(f"\n最终全局状态：{final_state}")
print(f"总访问次数：{access_count}")
test_global_state()
---
03.锁的进阶使用
a.读写锁（Read-Write Lock）实现
a.读写锁概念
读写锁允许多个线程同时读取共享资源，但只允许一个线程写入资源。这种机制在读多写少的场景中能显著提高性能，因为读操作可以并发执行。
b.读写锁Python实现
Python标准库没有直接提供读写锁，但我们可以使用锁和条件变量来实现。
c.代码示例
---
# 读写锁实现与应用
import threading
import time
from contextlib import contextmanager
class ReadWriteLock:
"""读写锁实现"""
def __init__(self):
self._readers = 0 # 当前读者数量
self._writers = 0 # 当前写者数量（0或1）
self._read_ready = threading.Condition(threading.RLock())
self._write_ready = threading.Condition(threading.RLock())
self._readers_lock = threading.Lock()
@contextmanager
def reader_lock(self):
"""读锁上下文管理器"""
self.acquire_read()
try:
yield
finally:
self.release_read()
@contextmanager
def writer_lock(self):
"""写锁上下文管理器"""
self.acquire_write()
try:
yield
finally:
self.release_write()
def acquire_read(self):
"""获取读锁"""
with self._read_ready:
while self._writers > 0:
self._read_ready.wait()
self._readers += 1
def release_read(self):
"""释放读锁"""
with self._read_ready:
self._readers -= 1
if self._readers == 0:
self._read_ready.notifyAll()
def acquire_write(self):
"""获取写锁"""
with self._write_ready:
while self._writers > 0 or self._readers > 0:
self._write_ready.wait()
self._writers = 1
def release_write(self):
"""释放写锁"""
with self._write_ready:
self._writers = 0
self._write_ready.notifyAll()
with self._read_ready:
self._read_ready.notifyAll()
class ThreadSafeCache:
"""使用读写锁的线程安全缓存"""
def __init__(self):
self._cache = {}
self._lock = ReadWriteLock()
self._stats = {
'reads': 0,
'writes': 0,
'hits': 0,
'misses': 0
}
def get(self, key):
"""读取缓存值"""
with self._lock.reader_lock():
self._stats['reads'] += 1
if key in self._cache:
self._stats['hits'] += 1
return self._cache[key]
else:
self._stats['misses'] += 1
return None
def set(self, key, value):
"""设置缓存值"""
with self._lock.writer_lock():
self._stats['writes'] += 1
self._cache[key] = value
def delete(self, key):
"""删除缓存项"""
with self._lock.writer_lock():
if key in self._cache:
del self._cache[key]
return True
return False
def size(self):
"""获取缓存大小"""
with self._lock.reader_lock():
return len(self._cache)
def clear(self):
"""清空缓存"""
with self._lock.writer_lock():
self._cache.clear()
def get_stats(self):
"""获取统计信息"""
with self._lock.reader_lock():
total_accesses = self._stats['reads']
hit_rate = (self._stats['hits'] / total_accesses * 100) if total_accesses > 0 else 0
return {
**self._stats,
'hit_rate': f"{hit_rate:.2f}%",
'cache_size': len(self._cache)
}
# 测试读写锁和缓存
def cache_reader(reader_id, cache, read_count):
"""缓存读取者线程"""
print(f"读取者{reader_id}开始工作")
for i in range(read_count):
key = f"key_{i % 10}"
value = cache.get(key)
print(f"读取者{reader_id}：读取 {key} = {value}")
time.sleep(0.01)
def cache_writer(writer_id, cache, write_count):
"""缓存写入者线程"""
print(f"写入者{writer_id}开始工作")
for i in range(write_count):
key = f"key_{i % 10}"
value = f"value_by_writer_{writer_id}_{i}"
cache.set(key, value)
print(f"写入者{writer_id}：写入 {key} = {value}")
time.sleep(0.02)
def test_read_write_lock():
"""测试读写锁性能"""
print("=== 读写锁性能测试 ===")
# 创建缓存并预填充数据
cache = ThreadSafeCache()
for i in range(10):
cache.set(f"key_{i}", f"initial_value_{i}")
# 启动多个读者（并发读取）
reader_threads = []
for i in range(8):
t = threading.Thread(target=cache_reader, args=(i+1, cache, 50))
reader_threads.append(t)
t.start()
# 启动少量写入者
writer_threads = []
for i in range(2):
t = threading.Thread(target=cache_writer, args=(i+1, cache, 20))
writer_threads.append(t)
t.start()
# 等待所有线程完成
for t in reader_threads + writer_threads:
t.join()
# 显示统计信息
final_stats = cache.get_stats()
print(f"\n缓存统计：{final_stats}")
test_read_write_lock()
---
b.原子操作与锁的比较
a.原子操作的概念
原子操作是指不可分割的操作，要么完全执行，要么完全不执行。在Python中，虽然GIL保证了单个字节码的原子性，但复合操作需要额外的同步机制。
b.代码示例
---
# 原子操作与锁的比较
import threading
import time
import queue
# 方法1：使用锁的计数器
class LockCounter:
"""使用锁的计数器"""
def __init__(self):
self.value = 0
self.lock = threading.Lock()
def increment(self):
"""增加计数器"""
with self.lock:
self.value += 1
def get(self):
"""获取当前值"""
with self.lock:
return self.value
# 方法2：使用队列的原子计数器
class AtomicCounter:
"""使用队列实现的原子计数器"""
def __init__(self):
self.value = 0
self.operations = queue.Queue()
self.worker_thread = None
self.running = False
def _worker(self):
"""工作线程：串行处理所有操作"""
while self.running:
try:
operation = self.operations.get(timeout=0.1)
if operation is None: # 停止信号
break
op_type = operation[0]
if op_type == 'increment':
self.value += 1
elif op_type == 'get':
result_queue = operation[1]
result_queue.put(self.value)
self.operations.task_done()
except queue.Empty:
continue
def start(self):
"""启动工作线程"""
self.running = True
self.worker_thread = threading.Thread(target=self._worker)
self.worker_thread.start()
def stop(self):
"""停止工作线程"""
self.running = False
self.operations.put(None) # 发送停止信号
if self.worker_thread:
self.worker_thread.join()
def increment(self):
"""原子增加"""
self.operations.put(('increment',))
def get(self):
"""原子获取"""
result_queue = queue.Queue()
self.operations.put(('get', result_queue))
return result_queue.get()
# 方法3：使用threading.local避免共享状态
class LocalCounter:
"""使用线程本地存储的计数器"""
def __init__(self):
self.local_data = threading.local()
def increment(self):
"""增加当前线程的计数器"""
if not hasattr(self.local_data, 'counter'):
self.local_data.counter = 0
self.local_data.counter += 1
def get_current(self):
"""获取当前线程的计数器"""
return getattr(self.local_data, 'counter', 0)
def get_total(self, thread_count):
"""估算总计数（需要线程数量）"""
# 注意：这种方法不是真正的原子操作
total = 0
# 这里仅作演示，实际应用中需要其他机制来聚合所有线程的结果
return total
# 性能比较测试
def test_lock_counter(thread_count, operations_per_thread):
"""测试基于锁的计数器"""
print(f"\n=== 锁计数器测试（{thread_count}线程，每线程{operations_per_thread}操作）===")
counter = LockCounter()
threads = []
start_time = time.time()
for i in range(thread_count):
t = threading.Thread(target=lambda: [
counter.increment() for _ in range(operations_per_thread)
])
threads.append(t)
t.start()
for t in threads:
t.join()
end_time = time.time()
expected = thread_count * operations_per_thread
actual = counter.get()
print(f"期望值：{expected}")
print(f"实际值：{actual}")
print(f"耗时：{end_time - start_time:.3f}秒")
return end_time - start_time
def test_atomic_counter(thread_count, operations_per_thread):
"""测试基于队列的原子计数器"""
print(f"\n=== 原子计数器测试（{thread_count}线程，每线程{operations_per_thread}操作）===")
counter = AtomicCounter()
counter.start()
threads = []
start_time = time.time()
for i in range(thread_count):
t = threading.Thread(target=lambda: [
counter.increment() for _ in range(operations_per_thread)
])
threads.append(t)
t.start()
for t in threads:
t.join()
end_time = time.time()
expected = thread_count * operations_per_thread
actual = counter.get()
print(f"期望值：{expected}")
print(f"实际值：{actual}")
print(f"耗时：{end_time - start_time:.3f}秒")
counter.stop()
return end_time - start_time
# 运行性能比较
lock_time = test_lock_counter(10, 10000)
atomic_time = test_atomic_counter(10, 10000)
print(f"\n性能比较：")
print(f"锁计数器：{lock_time:.3f}秒")
print(f"原子计数器：{atomic_time:.3f}秒")
print(f"性能差异：{abs(lock_time - atomic_time):.3f}秒")
---
04.死锁预防与检测
a.死锁的四个必要条件
a.互斥条件
资源不能被多个线程同时使用。
b.请求与保持条件
线程已获得部分资源，还在等待其他资源。
c.不剥夺条件
资源不能被强制性地从持有它的线程中剥夺。
d.循环等待条件
存在一种线程资源的循环等待链。
b.死锁预防策略
a.锁排序策略
所有线程按照固定的顺序获取锁，避免循环等待。
b.代码示例
---
# 死锁预防：锁排序策略
import threading
import time
from contextlib import contextmanager
class LockOrderManager:
"""锁顺序管理器"""
def __init__(self):
self._locks = {}
self._global_order = 0
self._order_lock = threading.Lock()
def get_lock(self, lock_id):
"""获取指定ID的锁"""
if lock_id not in self._locks:
with self._order_lock:
if lock_id not in self._locks:
# 为每个锁分配全局顺序号
lock_order = self._global_order
self._global_order += 1
self._locks[lock_id] = {
'lock': threading.Lock(),
'order': lock_order
}
return self._locks[lock_id]['lock']
@contextmanager
def acquire_ordered(self, lock_ids):
"""按顺序获取多个锁"""
# 按order字段排序锁ID
sorted_locks = sorted(
[(lock_id, self.get_lock(lock_id)) for lock_id in lock_ids],
key=lambda x: self._locks[x[0]]['order'] if x[0] in self._locks else 0
)
acquired_locks = []
try:
# 按顺序获取锁
for lock_id, lock in sorted_locks:
lock.acquire()
acquired_locks.append((lock_id, lock))
print(f"获取锁：{lock_id}")
yield
finally:
# 按相反顺序释放锁
for lock_id, lock in reversed(acquired_locks):
lock.release()
print(f"释放锁：{lock_id}")
# 资源类（使用锁排序管理器）
class Resource:
"""共享资源类"""
def __init__(self, name, lock_manager):
self.name = name
self.data = {}
self.lock_manager = lock_manager
def update_with_resource(self, other_resource, key, value):
"""与另一个资源协作更新数据"""
# 按资源名称排序获取锁
lock_ids = sorted([self.name, other_resource.name])
with self.lock_manager.acquire_ordered(lock_ids):
# 现在安全地访问两个资源
print(f"线程 {threading.current_thread().name} 正在更新 {self.name} 和 {other_resource.name}")
# 模拟耗时操作
time.sleep(0.1)
# 更新数据
self.data[key] = f"{value}_in_{self.name}"
other_resource.data[key] = f"{value}_in_{other_resource.name}"
def worker_thread(resource1, resource2, worker_id):
"""工作线程"""
print(f"工作者{worker_id}开始")
for i in range(3):
key = f"key_{i}"
value = f"value_{worker_id}_{i}"
# 协作更新资源
resource1.update_with_resource(resource2, key, value)
time.sleep(0.05)
print(f"工作者{worker_id}完成")
def test_deadlock_prevention():
"""测试死锁预防"""
print("=== 死锁预防测试 ===")
lock_manager = LockOrderManager()
# 创建多个资源
resource_a = Resource("resource_a", lock_manager)
resource_b = Resource("resource_b", lock_manager)
resource_c = Resource("resource_c", lock_manager)
# 创建多个工作线程，它们可能以不同顺序访问资源
threads = []
# 线程1：按 A->B 顺序访问
t1 = threading.Thread(target=worker_thread, args=(resource_a, resource_b, 1), name="Worker-1")
# 线程2：按 B->A 顺序访问（如果不用锁排序，会死锁）
t2 = threading.Thread(target=worker_thread, args=(resource_b, resource_a, 2), name="Worker-2")
# 线程3：按 C->A->B 顺序访问
t3 = threading.Thread(target=worker_thread, args=(resource_c, resource_a, 3), name="Worker-3")
threads.extend([t1, t2, t3])
# 启动所有线程
for t in threads:
t.start()
# 等待完成
for t in threads:
t.join()
print("死锁预防测试完成 - 没有发生死锁！")
# 运行测试
test_deadlock_prevention()
---
c.超时机制与死锁检测
a.超时锁的使用
使用try-acquire-with-timeout机制，避免无限等待。
b.代码示例
---
# 超时机制与死锁检测
import threading
import time
from collections import defaultdict
class DeadlockDetector:
"""死锁检测器"""
def __init__(self):
self.lock_graph = defaultdict(set) # 等待图
self.lock_owners = {} # 锁的拥有者
self.graph_lock = threading.Lock()
self.detection_enabled = True
def request_lock(self, thread_id, lock_id):
"""记录线程请求锁"""
with self.graph_lock:
if self.lock_owners.get(lock_id) != thread_id:
# 记录等待关系：thread_id -> lock_owner
owner = self.lock_owners.get(lock_id)
if owner and owner != thread_id:
self.lock_graph[thread_id].add(owner)
return self._detect_deadlock(thread_id)
return False
def acquire_lock(self, thread_id, lock_id):
"""记录线程获得锁"""
with self.graph_lock:
# 清理等待关系
for waiting_thread in list(self.lock_graph.keys()):
self.lock_graph[waiting_thread].discard(thread_id)
# 设置锁拥有者
self.lock_owners[lock_id] = thread_id
def release_lock(self, thread_id, lock_id):
"""记录线程释放锁"""
with self.graph_lock:
if self.lock_owners.get(lock_id) == thread_id:
del self.lock_owners[lock_id]
def _detect_deadlock(self, thread_id):
"""检测死锁（DFS）"""
if not self.detection_enabled:
return False
visited = set()
rec_stack = set()
def has_cycle(node):
if node in rec_stack:
return True
if node in visited:
return False
visited.add(node)
rec_stack.add(node)
for neighbor in self.lock_graph[node]:
if has_cycle(neighbor):
return True
rec_stack.remove(node)
return False
return has_cycle(thread_id)
def get_wait_chain(self, thread_id):
"""获取等待链"""
with self.graph_lock:
chain = []
visited = set()
def build_chain(node):
if node in visited or node not in self.lock_graph:
return
visited.add(node)
chain.append(node)
for neighbor in self.lock_graph[node]:
build_chain(neighbor)
build_chain(thread_id)
return chain
class TimeoutLock:
"""带超时的锁"""
def __init__(self, lock_id, deadlock_detector=None):
self.lock_id = lock_id
self.lock = threading.Lock()
self.detector = deadlock_detector
def acquire(self, timeout=5.0):
"""获取锁（带超时和死锁检测）"""
thread_id = threading.current_thread().ident
# 记录锁请求
if self.detector:
is_deadlock = self.detector.request_lock(thread_id, self.lock_id)
if is_deadlock:
wait_chain = self.detector.get_wait_chain(thread_id)
raise RuntimeError(f"检测到死锁！等待链：{' -> '.join(map(str, wait_chain))}")
# 尝试获取锁
acquired = self.lock.acquire(timeout=timeout)
if acquired:
# 记录锁获得
if self.detector:
self.detector.acquire_lock(thread_id, self.lock_id)
return True
else:
# 超时，清理请求记录
if self.detector:
self.detector.release_lock(thread_id, self.lock_id)
raise TimeoutError(f"获取锁 {self.lock_id} 超时")
def release(self):
"""释放锁"""
thread_id = threading.current_thread().ident
# 记录锁释放
if self.detector:
self.detector.release_lock(thread_id, self.lock_id)
self.lock.release()
@contextmanager
def with_timeout(self, timeout=5.0):
"""上下文管理器形式"""
self.acquire(timeout)
try:
yield
finally:
self.release()
def test_timeout_and_deadlock_detection():
"""测试超时机制和死锁检测"""
print("=== 超时机制与死锁检测测试 ===")
# 创建死锁检测器
detector = DeadlockDetector()
# 创建多个锁
lock1 = TimeoutLock("lock1", detector)
lock2 = TimeoutLock("lock2", detector)
lock3 = TimeoutLock("lock3", detector)
def worker_with_deadlock_risk(worker_id, lock_a, lock_b):
"""可能有死锁风险的工作线程"""
thread_name = f"Worker-{worker_id}"
try:
print(f"{thread_name} 尝试获取 {lock_a.lock_id}")
lock_a.acquire(timeout=2.0)
print(f"{thread_name} 获得 {lock_a.lock_id}")
time.sleep(0.5) # 增加死锁概率
print(f"{thread_name} 尝试获取 {lock_b.lock_id}")
lock_b.acquire(timeout=2.0)
print(f"{thread_name} 获得 {lock_b.lock_id}")
# 模拟工作
time.sleep(0.5)
print(f"{thread_name} 完成工作")
except TimeoutError as e:
print(f"{thread_name} 超时：{e}")
except RuntimeError as e:
print(f"{thread_name} 死锁检测：{e}")
finally:
# 确保释放已获得的锁
try:
lock_a.release()
except:
pass
try:
lock_b.release()
except:
pass
def safe_worker(worker_id, lock_a, lock_b):
"""安全的工作线程（按顺序获取锁）"""
thread_name = f"SafeWorker-{worker_id}"
try:
# 按锁ID排序获取锁
locks = sorted([lock_a, lock_b], key=lambda x: x.lock_id)
for lock in locks:
print(f"{thread_name} 获取 {lock.lock_id}")
lock.acquire(timeout=3.0)
# 模拟工作
time.sleep(0.5)
print(f"{thread_name} 完成工作")
except Exception as e:
print(f"{thread_name} 错误：{e}")
finally:
# 按相反顺序释放锁
for lock in sorted([lock_a, lock_b], key=lambda x: x.lock_id, reverse=True):
try:
lock.release()
print(f"{thread_name} 释放 {lock.lock_id}")
except:
pass
# 测试1：可能导致死锁的场景
print("\n--- 死锁风险测试 ---")
risky_threads = []
# 线程1：lock1 -> lock2
t1 = threading.Thread(target=worker_with_deadlock_risk, args=(1, lock1, lock2))
# 线程2：lock2 -> lock1（可能死锁）
t2 = threading.Thread(target=worker_with_deadlock_risk, args=(2, lock2, lock1))
risky_threads.extend([t1, t2])
for t in risky_threads:
t.start()
time.sleep(0.1) # 错开启动时间
for t in risky_threads:
t.join(timeout=5)
if t.is_alive():
print(f"线程 {t.name} 仍在运行，可能已死锁")
# 测试2：安全的多锁获取
print("\n--- 安全多锁测试 ---")
safe_threads = []
# 创建更多线程，包括使用lock3的
for i in range(3):
locks = [lock1, lock2, lock3]
import random
selected_locks = random.sample(locks, 2)
t = threading.Thread(target=safe_worker, args=(i+1, selected_locks[0], selected_locks[1]))
safe_threads.append(t)
t.start()
for t in safe_threads:
t.join()
print("超时机制与死锁检测测试完成")
# 运行测试
test_timeout_and_deadlock_detection()
---
6 多进程
6.1 multiprocessing.Process
01.multiprocessing模块概述
a.多进程的优势与适用场景
Python的multiprocessing模块提供了一种绕过GIL（全局解释器锁）限制的方法，通过创建真正的操作系统进程来实现并行计算。每个进程都有自己独立的Python解释器和内存空间，能够充分利用多核CPU的计算能力。与多线程相比，多进程更适合CPU密集型任务，如数据处理、机器学习、图像处理等需要大量计算的场景。
b.进程与线程的区别
a内存独立性
进程拥有独立的内存空间，进程间不共享内存（除非通过特殊机制），而同一进程内的线程共享内存空间。
b并行性差异
由于GIL的存在，Python多线程在CPU密集型任务中只能实现并发而非真正的并行，而多进程可以充分利用多核CPU实现真正的并行执行。
c创建开销
进程创建开销大于线程，进程间通信也比线程间通信更复杂。
02.Process类基础使用
a.创建子进程
a基础Process用法
multiprocessing.Process类提供了创建子进程的基本方法，通过target参数指定子进程要执行的函数，args参数传递函数参数。
b.代码示例
---
# multiprocessing.Process基础使用示例
import multiprocessing
import time
import os
def simple_worker():
"""简单的工作进程函数"""
print(f"子进程启动，PID: {os.getpid()}")
print(f"父进程PID: {os.getppid()}")
# 模拟CPU密集型任务
total = 0
for i in range(10_000_000):
total += i
print(f"子进程完成，计算结果: {total:,}")
return total
def test_basic_process():
"""测试基础进程创建"""
print(f"主进程开始，PID: {os.getpid()}")
# 创建子进程
process = multiprocessing.Process(target=simple_worker)
print("启动子进程...")
process.start()
# 等待子进程完成
process.join()
print("子进程已结束")
if __name__ == '__main__':
# Windows平台需要使用if __name__ == '__main__'
test_basic_process()
---
c.带参数的进程
a参数传递机制
通过args参数传递位置参数，通过kwargs参数传递关键字参数。
b.代码示例
---
# 带参数的进程示例
import multiprocessing
import os
import time
def worker_with_args(name, age, city="北京", hobby="编程"):
"""带参数的工作进程函数"""
print(f"工作者 {name} (PID: {os.getpid()}) 开始工作")
print(f"个人信息: {age}岁，来自{city}，爱好是{hobby}")
# 模拟不同工作负载
workload = age * 1000000 # 根据年龄调整工作量
result = 0
for i in range(workload):
result += i
print(f"工作者 {name} 完成，计算了 {workload:,} 次操作")
return result, len(name)
def test_process_with_args():
"""测试带参数的进程"""
workers = [
("张三", 25, {"city": "上海", "hobby": "阅读"}),
("李四", 30, {"city": "广州", "hobby": "运动"}),
("王五", 35) # 使用默认值
]
processes = []
for worker_info in workers:
name = worker_info[0]
age = worker_info[1]
kwargs = worker_info[2] if len(worker_info) > 2 else {}
process = multiprocessing.Process(
target=worker_with_args,
args=(name, age), # 位置参数
kwargs=kwargs # 关键字参数
)
processes.append(process)
process.start()
# 等待所有进程完成
for process in processes:
process.join()
print("所有工作者进程已完成")
if __name__ == '__main__':
test_process_with_args()
---
03.进程的生命周期管理
a.进程状态与控制
a进程状态分析
进程具有创建、就绪、运行、阻塞、终止等状态。multiprocessing.Process类提供了丰富的状态查询和控制方法，如is_alive()、terminate()、join()等。
b.进程控制方法详解
start(): 启动进程
join([timeout]): 等待进程结束
is_alive(): 检查进程是否在运行
terminate(): 强制终止进程
close(): 关闭进程对象
c.代码示例
---
# 进程生命周期管理示例
import multiprocessing
import time
import os
def long_running_worker(duration, worker_id):
"""长时间运行的工作进程"""
print(f"工作者{worker_id} (PID: {os.getpid()}) 开始运行，预计运行{duration}秒")
start_time = time.time()
while time.time() - start_time < duration:
# 模拟工作
sum_val = 0
for i in range(100000):
sum_val += i
# 定期报告状态
if int(time.time() - start_time) % 2 == 0:
elapsed = int(time.time() - start_time)
print(f"工作者{worker_id} 已运行 {elapsed} 秒")
time.sleep(0.1)
print(f"工作者{worker_id} 完成")
def test_process_lifecycle():
"""测试进程生命周期管理"""
# 创建一个运行10秒的进程
process = multiprocessing.Process(
target=long_running_worker,
args=(10, 1)
)
print("=== 进程生命周期测试 ===")
print(f"进程创建后状态: alive={process.is_alive()}")
# 启动进程
print("启动进程...")
process.start()
print(f"进程启动后状态: alive={process.is_alive()}, PID={process.pid}")
# 监控进程
monitor_time = 0
while process.is_alive() and monitor_time < 5:
time.sleep(1)
monitor_time += 1
print(f"监控第{monitor_time}秒: 进程仍在运行")
# 等待进程自然完成
print("等待进程自然结束...")
process.join(timeout=15)
if process.is_alive():
print("进程超时，强制终止")
process.terminate()
process.join(timeout=2)
else:
print("进程正常结束")
# 检查退出码
print(f"进程退出码: {process.exitcode}")
# 关闭进程对象
process.close()
print("进程对象已关闭")
def test_multiple_processes():
"""测试多进程管理"""
print("\n=== 多进程管理测试 ===")
# 创建多个不同运行时间的进程
processes = []
durations = [3, 5, 7, 2, 4]
for i, duration in enumerate(durations):
process = multiprocessing.Process(
target=long_running_worker,
args=(duration, i+2) # 工作者ID从2开始
)
processes.append(process)
process.start()
print(f"启动工作者{i+2}，运行时间: {duration}秒")
# 监控所有进程
all_completed = False
start_time = time.time()
while not all_completed and (time.time() - start_time) < 20:
all_completed = True
running_count = 0
for i, process in enumerate(processes):
if process.is_alive():
all_completed = False
running_count += 1
else:
# 检查是否正常结束
if process.exitcode == 0:
print(f"工作者{i+2} 正常完成")
else:
print(f"工作者{i+2} 异常退出，退出码: {process.exitcode}")
if running_count > 0:
print(f"仍有 {running_count} 个进程在运行，继续等待...")
time.sleep(1)
# 强制结束仍在运行的进程
for i, process in enumerate(processes):
if process.is_alive():
print(f"强制终止工作者{i+2}")
process.terminate()
process.join(timeout=2)
print("所有进程管理操作完成")
if __name__ == '__main__':
test_process_lifecycle()
test_multiple_processes()
---
04.进程间数据传递
a.进程通信机制概述
aIPC方法分类
由于进程内存独立，需要专门的进程间通信(IPC)机制。Python multiprocessing提供了多种IPC方式：Queue、Pipe、Manager、Value、Array等。
b.通信方式选择指南
Queue: 适合生产者-消费者模式
Pipe: 适合双向点对点通信
Manager: 适合共享复杂对象
Value/Array: 适合共享简单数据
b.Queue进程间通信
a.Queue特点
multiprocessing.Queue是线程安全的队列，专门用于进程间通信。它实现了所有必要的锁和序列化机制，可以安全地在进程间传递Python对象。
b.代码示例
---
# 使用Queue进行进程间通信示例
import multiprocessing
import time
import random
def producer(queue, producer_id, item_count):
"""生产者进程"""
print(f"生产者{producer_id} (PID: {multiprocessing.current_process().pid}) 开始生产")
for i in range(item_count):
item = f"商品_{producer_id}_{i}"
value = random.randint(1, 100)
# 将商品放入队列
queue.put((item, value))
print(f"生产者{producer_id} 生产了: {item} (价值: {value})")
# 随机生产间隔
time.sleep(random.uniform(0.1, 0.5))
# 发送结束信号
queue.put((None, None))
print(f"生产者{producer_id} 生产完成")
def consumer(queue, consumer_id):
"""消费者进程"""
print(f"消费者{consumer_id} (PID: {multiprocessing.current_process().pid}) 开始消费")
consumed_count = 0
total_value = 0
while True:
try:
# 从队列获取商品
item, value = queue.get(timeout=2)
if item is None: # 收到结束信号
print(f"消费者{consumer_id} 收到结束信号")
break
consumed_count += 1
total_value += value
print(f"消费者{consumer_id} 消费了: {item} (价值: {value})")
# 模拟消费时间
time.sleep(random.uniform(0.2, 0.8))
except Exception as e:
print(f"消费者{consumer_id} 获取队列超时: {e}")
break
print(f"消费者{consumer_id} 完成消费，总计: {consumed_count}件商品，总价值: {total_value}")
def test_process_queue():
"""测试进程间队列通信"""
print("=== 进程间Queue通信测试 ===")
# 创建共享队列
queue = multiprocessing.Queue()
# 创建生产者和消费者进程
producers = []
consumers = []
# 2个生产者，3个消费者
for i in range(2):
producer = multiprocessing.Process(
target=producer,
args=(queue, i+1, 8) # 每个生产者生产8件商品
)
producers.append(producer)
for i in range(3):
consumer = multiprocessing.Process(
target=consumer,
args=(queue, i+1)
)
consumers.append(consumer)
# 启动所有进程
print("启动生产者和消费者进程...")
# 先启动消费者，确保它们在等待
for consumer in consumers:
consumer.start()
# 延迟启动生产者，模拟现实场景
time.sleep(0.5)
for producer in producers:
producer.start()
# 等待所有进程完成
for producer in producers:
producer.join()
for consumer in consumers:
consumer.join()
print("所有生产者和消费者进程已完成")
if __name__ == '__main__':
test_process_queue()
---
c.Pipe进程间通信
a.Pipe特点
multiprocessing.Pipe()返回一对连接对象(conn1, conn2)，可以在两个进程间建立双向通信通道。Pipe适合点对点通信，传输速度快，但不适合多对多通信。
b.代码示例
---
# 使用Pipe进行进程间通信示例
import multiprocessing
import time
import random
def worker_task(pipe, worker_id, task_count):
"""工作任务进程"""
print(f"工作者{worker_id} (PID: {multiprocessing.current_process().pid}) 开始工作")
# 关闭管道的另一端
pipe.close()
results = []
for i in range(task_count):
task = f"任务_{worker_id}_{i}"
complexity = random.randint(1, 5)
print(f"工作者{worker_id} 开始处理: {task} (复杂度: {complexity})")
# 模拟任务处理
processing_time = complexity * 0.5
time.sleep(processing_time)
# 生成任务结果
result = {
'task': task,
'complexity': complexity,
'processing_time': processing_time,
'status': 'completed',
'worker_id': worker_id
}
results.append(result)
# 发送结果到主进程
pipe.send(result)
print(f"工作者{worker_id} 完成并发送结果: {task}")
print(f"工作者{worker_id} 所有任务完成")
def task_dispatcher(task_count_per_worker, worker_count):
"""任务分发器进程"""
print("任务分发器启动")
# 创建与每个工作者的管道
pipes = []
workers = []
for i in range(worker_count):
# 创建管道
parent_conn, child_conn = multiprocessing.Pipe()
pipes.append(parent_conn)
# 创建工作者进程
worker = multiprocessing.Process(
target=worker_task,
args=(child_conn, i+1, task_count_per_worker)
)
workers.append(worker)
worker.start()
# 监控工作者进度
completed_tasks = []
total_tasks = worker_count * task_count_per_worker
print(f"等待 {total_tasks} 个任务完成...")
start_time = time.time()
while len(completed_tasks) < total_tasks:
# 检查所有管道是否有数据
for i, pipe in enumerate(pipes):
if pipe.poll(): # 检查是否有数据可读
try:
result = pipe.recv()
completed_tasks.append(result)
print(f"收到任务结果: {result['task']} "
f"(工作者{result['worker_id']}, "
f"耗时{result['processing_time']:.1f}s)")
except Exception as e:
print(f"读取管道{i}时出错: {e}")
time.sleep(0.1)
end_time = time.time()
# 等待所有工作者完成
for worker in workers:
worker.join()
# 关闭所有管道
for pipe in pipes:
pipe.close()
print(f"\n任务分发完成统计:")
print(f"总任务数: {len(completed_tasks)}")
print(f"总耗时: {end_time - start_time:.2f}秒")
print(f"平均每任务耗时: {(end_time - start_time) / len(completed_tasks):.2f}秒")
return completed_tasks
def test_process_pipe():
"""测试进程间Pipe通信"""
print("=== 进程间Pipe通信测试 ===")
# 任务参数
task_count_per_worker = 5
worker_count = 3
# 启动任务分发器
results = task_dispatcher(task_count_per_worker, worker_count)
# 分析结果
print("\n任务结果分析:")
worker_stats = {}
for result in results:
worker_id = result['worker_id']
if worker_id not in worker_stats:
worker_stats[worker_id] = {
'count': 0,
'total_time': 0,
'complexity': []
}
worker_stats[worker_id]['count'] += 1
worker_stats[worker_id]['total_time'] += result['processing_time']
worker_stats[worker_id]['complexity'].append(result['complexity'])
for worker_id, stats in worker_stats.items():
avg_time = stats['total_time'] / stats['count']
avg_complexity = sum(stats['complexity']) / len(stats['complexity'])
print(f"工作者{worker_id}: 完成任务{stats['count']}个, "
f"平均耗时{avg_time:.2f}s, 平均复杂度{avg_complexity:.1f}")
if __name__ == '__main__':
test_process_pipe()
---
d.Manager共享对象
a.Manager特点
multiprocessing.Manager创建一个特殊的进程，管理共享对象。其他进程可以通过代理对象访问这些共享对象，Manager负责协调所有访问，确保数据一致性。
b.代码示例
---
# 使用Manager共享对象示例
import multiprocessing
import time
import random
from datetime import datetime
def bank_account_worker(shared_dict, account_manager, worker_id, operations):
"""银行账户工作进程"""
print(f"银行柜员{worker_id}开始工作")
successful_ops = 0
failed_ops = 0
for i in range(operations):
operation = random.choice(['deposit', 'withdraw', 'transfer', 'balance'])
try:
if operation == 'deposit':
# 存款
amount = random.randint(100, 1000)
account = shared_dict['account1']
with account_manager.get_lock():
account['balance'] += amount
account['transactions'].append({
'type': 'deposit',
'amount': amount,
'worker': worker_id,
'time': datetime.now().strftime('%H:%M:%S')
})
print(f"柜员{worker_id}: 存款 {amount} 元，当前余额: {account['balance']}")
successful_ops += 1
elif operation == 'withdraw':
# 取款
amount = random.randint(50, 500)
account = shared_dict['account1']
with account_manager.get_lock():
if account['balance'] >= amount:
account['balance'] -= amount
account['transactions'].append({
'type': 'withdraw',
'amount': amount,
'worker': worker_id,
'time': datetime.now().strftime('%H:%M:%S')
})
print(f"柜员{worker_id}: 取款 {amount} 元，当前余额: {account['balance']}")
successful_ops += 1
else:
print(f"柜员{worker_id}: 取款失败，余额不足 (需要: {amount}, 当前: {account['balance']})")
failed_ops += 1
elif operation == 'transfer':
# 转账
amount = random.randint(100, 500)
source = shared_dict['account1']
target = shared_dict['account2']
with account_manager.get_lock():
if source['balance'] >= amount:
source['balance'] -= amount
target['balance'] += amount
source['transactions'].append({
'type': 'transfer_out',
'amount': amount,
'target': 'account2',
'worker': worker_id,
'time': datetime.now().strftime('%H:%M:%S')
})
target['transactions'].append({
'type': 'transfer_in',
'amount': amount,
'source': 'account1',
'worker': worker_id,
'time': datetime.now().strftime('%H:%M:%S')
})
print(f"柜员{worker_id}: 转账 {amount} 元 (account1->account2)")
successful_ops += 1
else:
print(f"柜员{worker_id}: 转账失败，余额不足")
failed_ops += 1
elif operation == 'balance':
# 查询余额
for account_name in ['account1', 'account2']:
account = shared_dict[account_name]
print(f"柜员{worker_id}: {account_name} 余额: {account['balance']} 元")
successful_ops += 1
except Exception as e:
print(f"柜员{worker_id} 操作失败: {e}")
failed_ops += 1
# 模拟处理时间
time.sleep(random.uniform(0.1, 0.3))
# 记录工作者统计
shared_dict['worker_stats'][worker_id] = {
'successful': successful_ops,
'failed': failed_ops,
'total': operations
}
print(f"柜员{worker_id}完成工作: 成功{successful_ops}次, 失败{failed_ops}次")
def test_shared_manager():
"""测试Manager共享对象"""
print("=== Manager共享对象测试 ===")
# 创建Manager和共享对象
with multiprocessing.Manager() as manager:
# 创建共享字典
shared_data = manager.dict()
# 初始化账户数据
shared_data['account1'] = {
'balance': 10000,
'owner': '张三',
'transactions': manager.list()
}
shared_data['account2'] = {
'balance': 5000,
'owner': '李四',
'transactions': manager.list()
}
shared_data['worker_stats'] = manager.dict()
print(f"初始账户状态:")
print(f"account1 ({shared_data['account1']['owner']}): {shared_data['account1']['balance']} 元")
print(f"account2 ({shared_data['account2']['owner']}): {shared_data['account2']['balance']} 元")
# 创建多个银行柜员进程
num_workers = 4
operations_per_worker = 10
workers = []
for i in range(num_workers):
worker = multiprocessing.Process(
target=bank_account_worker,
args=(shared_data, manager, i+1, operations_per_worker)
)
workers.append(worker)
worker.start()
# 等待所有柜员完成
for worker in workers:
worker.join()
# 显示最终结果
print(f"\n最终账户状态:")
print(f"account1 ({shared_data['account1']['owner']}): {shared_data['account1']['balance']} 元")
print(f"account2 ({shared_data['account2']['owner']}): {shared_data['account2']['balance']} 元")
print(f"\n交易记录:")
for account_name in ['account1', 'account2']:
account = shared_data[account_name]
print(f"\n{account_name} 交易记录 ({len(account['transactions'])}笔):")
for i, transaction in enumerate(account['transactions'][-5:], 1): # 显示最后5笔
print(f" {i}. {transaction['type']} {transaction['amount']}元 "
f"({transaction.get('time', 'N/A')})")
print(f"\n柜员工作统计:")
total_successful = 0
total_failed = 0
for worker_id, stats in shared_data['worker_stats'].items():
print(f" 柜员{worker_id}: 成功{stats['successful']}次, "
f"失败{stats['failed']}次, 总计{stats['total']}次")
total_successful += stats['successful']
total_failed += stats['failed']
print(f"\n总计: 成功{total_successful}次, 失败{total_failed}次")
if __name__ == '__main__':
test_shared_manager()
---
6.2 进程池ProcessPoolExecutor
01.进程池概述
a.进程池的优势
进程池通过预先创建和复用进程来避免频繁创建/销毁进程的开销，提高了程序性能。对于需要处理大量短任务的场景，进程池能够显著减少进程创建的延迟，同时有效控制系统资源的使用。concurrent.futures.ProcessPoolExecutor提供了高级的异步接口，简化了并行编程的复杂性。
b.适用场景分析
aCPU密集型任务批处理
适合需要大量计算的并行任务，如数据分析、图像处理、科学计算等。
b.IO密集型但受GIL影响的任务
虽然主要是IO操作，但由于GIL限制导致多线程性能不佳的场景。
c长期运行的服务
需要持续处理请求的服务器应用，可以使用进程池避免频繁创建进程。
02.ProcessPoolExecutor基础使用
a.基础语法与配置
a.ProcessPoolExecutor参数说明
max_workers: 最大工作进程数，默认为CPU核心数
mp_context: 多进程上下文，用于控制进程创建方式
initializer: 工作进程初始化函数
initargs: 初始化函数参数
b.代码示例
---
# ProcessPoolExecutor基础使用示例
import concurrent.futures
import time
import os
import random
def simple_task(task_id):
"""简单任务函数"""
pid = os.getpid()
print(f"任务{task_id}在进程{pid}中开始")
# 模拟CPU密集型工作
result = sum(i * i for i in range(100000))
# 随机延迟，模拟不同任务复杂度
time.sleep(random.uniform(0.1, 0.5))
print(f"任务{task_id}在进程{pid}中完成，结果: {result:,}")
return {'task_id': task_id, 'result': result, 'pid': pid}
def test_basic_executor():
"""测试基础执行器用法"""
print(f"主进程PID: {os.getpid()}")
print("=== ProcessPoolExecutor基础测试 ===")
# 创建进程池，使用CPU核心数
with concurrent.futures.ProcessPoolExecutor() as executor:
# 提交任务
task_count = 8
print(f"提交{task_count}个任务到进程池...")
# 使用submit提交单个任务
futures = []
for i in range(task_count):
future = executor.submit(simple_task, i+1)
futures.append(future)
print("所有任务已提交，等待完成...")
# 收集结果
results = []
for i, future in enumerate(concurrent.futures.as_completed(futures)):
try:
result = future.result(timeout=10)
results.append(result)
print(f"任务完成: {result['task_id']} (进程: {result['pid']})")
except Exception as e:
print(f"任务执行失败: {e}")
print(f"总完成任务数: {len(results)}")
# 进程池自动关闭
print("进程池已关闭")
if __name__ == '__main__':
test_basic_executor()
---
b.map函数使用
a.map方法特点
map方法提供了类似内置map的接口，可以批量提交任务并保持结果顺序。对于需要保持任务顺序的场景，map方法非常方便。
b.代码示例
---
# ProcessPoolExecutor.map使用示例
import concurrent.futures
import time
import os
import random
def compute_prime(n):
"""计算第n个素数"""
if n < 1:
return None
def is_prime(num):
if num < 2:
return False
for i in range(2, int(num**0.5) + 1):
if num % i == 0:
return False
return True
count = 0
num = 2
while count < n:
if is_prime(num):
count += 1
if count == n:
return num
num += 1
return None
def fibonacci(n):
"""计算斐波那契数列第n项"""
if n <= 1:
return n
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b
def compute_factorial(n):
"""计算阶乘"""
if n < 0:
return None
if n == 0 or n == 1:
return 1
result = 1
for i in range(2, n + 1):
result *= i
return result
def mathematical_task(task_spec):
"""数学计算任务"""
task_type, n = task_spec
pid = os.getpid()
start_time = time.time()
if task_type == 'prime':
result = compute_prime(n)
task_name = f"第{n}个素数"
elif task_type == 'fibonacci':
result = fibonacci(n)
task_name = f"斐波那契数列第{n}项"
elif task_type == 'factorial':
result = compute_factorial(n)
task_name = f"{n}的阶乘"
else:
return None
end_time = time.time()
duration = end_time - start_time
print(f"进程{pid}计算{task_name} = {result:,} (耗时: {duration:.3f}s)")
return {
'task_type': task_type,
'n': n,
'result': result,
'duration': duration,
'pid': pid
}
def test_map_usage():
"""测试map方法使用"""
print("=== ProcessPoolExecutor.map测试 ===")
# 准备任务数据
tasks = [
('prime', 1000),
('prime', 2000),
('prime', 1500),
('fibonacci', 35),
('fibonacci', 40),
('fibonacci', 38),
('factorial', 10),
('factorial', 12),
('factorial', 11)
]
print(f"准备执行{len(tasks)}个数学计算任务:")
# 创建进程池
with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
print(f"使用3个工作进程并行计算...")
start_time = time.time()
# 使用map方法批量提交任务
# map会保持任务的顺序，结果与输入任务一一对应
results = list(executor.map(mathematical_task, tasks))
end_time = time.time()
total_time = end_time - start_time
print(f"\n所有计算完成，总耗时: {total_time:.3f}秒")
print(f"平均每任务耗时: {total_time / len(tasks):.3f}秒")
# 按任务类型分析结果
print("\n=== 计算结果分析 ===")
task_types = {}
for result in results:
if result:
task_type = result['task_type']
if task_type not in task_types:
task_types[task_type] = []
task_types[task_type].append(result)
for task_type, type_results in task_types.items():
print(f"\n{task_type}计算结果:")
for result in type_results:
task_name = {
'prime': f"第{result['n']}个素数",
'fibonacci': f"斐波那契数列第{result['n']}项",
'factorial': f"{result['n']}的阶乘"
}[task_type]
print(f" {task_name} = {result['result']:,} "
f"(进程: {result['pid']}, 耗时: {result['duration']:.3f}s)")
def test_map_with_timeout():
"""测试带超时的map使用"""
print("\n=== 带超时的map测试 ===")
# 一些可能较长的任务
long_tasks = [
('prime', 5000), # 较长的素数计算
('fibonacci', 45), # 较大的斐波那契数
('factorial', 15), # 较大的阶乘
]
with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
try:
print("开始执行长时间任务，设置8秒超时...")
results = list(executor.map(mathematical_task, long_tasks, timeout=8))
for i, result in enumerate(results):
if result:
print(f"任务{i+1}完成: {result['task_type']} - {result['result']:,}")
except concurrent.futures.TimeoutError:
print("某些任务执行超时")
except Exception as e:
print(f"任务执行出错: {e}")
if __name__ == '__main__':
test_map_usage()
test_map_with_timeout()
---
03.异步编程与回调
a.异步任务处理
a异步编程优势
ProcessPoolExecutor提供了异步接口，允许非阻塞地提交任务并获取结果，提高了程序的响应性和资源利用率。
b.代码示例
---
# ProcessPoolExecutor异步编程示例
import concurrent.futures
import time
import os
import random
from datetime import datetime
def download_image(image_id):
"""模拟图片下载任务"""
pid = os.getpid()
print(f"[{datetime.now().strftime('%H:%M:%S')}] 进程{pid}开始下载图片{image_id}")
# 模拟下载时间（1-5秒）
download_time = random.uniform(1.0, 5.0)
time.sleep(download_time)
# 模拟图片数据
image_size = random.randint(100, 1000) # KB
image_data = {
'id': image_id,
'size': image_size,
'download_time': download_time,
'pid': pid,
'timestamp': datetime.now()
}
print(f"[{datetime.now().strftime('%H:%M:%S')}] 进程{pid}完成下载图片{image_id} "
f"({image_size}KB, 耗时{download_time:.1f}s)")
return image_data
def process_image(image_data):
"""模拟图片处理任务"""
if image_data is None:
return None
pid = os.getpid()
image_id = image_data['id']
print(f"[{datetime.now().strftime('%H:%M:%S')}] 进程{pid}开始处理图片{image_id}")
# 模拟处理时间（0.5-2秒）
process_time = random.uniform(0.5, 2.0)
time.sleep(process_time)
# 模拟处理结果
processed_data = {
'original': image_data,
'processed_size': int(image_data['size'] * 0.8), # 压缩后大小
'process_time': process_time,
'pid': pid,
'timestamp': datetime.now()
}
print(f"[{datetime.now().strftime('%H:%M:%S')}] 进程{pid}完成处理图片{image_id} "
f"({processed_data['processed_size']}KB, 耗时{process_time:.1f}s)")
return processed_data
def callback_function(future):
"""任务完成回调函数"""
try:
result = future.result()
if result:
task_type = future.task_info.get('type', 'unknown')
print(f"[{datetime.now().strftime('%H:%M:%S')}] "
f"回调: {task_type}任务完成 - {result}")
except Exception as e:
print(f"[{datetime.now().strftime('%H:%M:%S')}] 回调: 任务执行失败 - {e}")
def error_callback(future):
"""错误回调函数"""
try:
result = future.result()
except Exception as e:
print(f"[{datetime.now().strftime('%H:%M:%S')}] 错误回调: {e}")
def test_async_callbacks():
"""测试异步回调机制"""
print("=== ProcessPoolExecutor异步回调测试 ===")
# 创建进程池
with concurrent.futures.ProcessPoolExecutor(max_workers=3) as executor:
# 图片ID列表
image_ids = list(range(1, 8))
# 为Future对象添加额外信息
def add_task_info(future, task_info):
future.task_info = task_info
# 第一阶段：异步下载图片
download_futures = []
print("开始异步下载图片...")
for image_id in image_ids:
future = executor.submit(download_image, image_id)
add_task_info(future, {'type': 'download', 'image_id': image_id})
# 添加完成回调
future.add_done_callback(callback_function)
future.add_done_callback(error_callback)
download_futures.append(future)
# 第二阶段：为每个下载完成的图片提交处理任务
processed_futures = []
download_results = []
print("监控下载进度并提交处理任务...")
for future in concurrent.futures.as_completed(download_futures, timeout=30):
try:
image_data = future.result()
download_results.append(image_data)
if image_data:
# 立即提交处理任务
process_future = executor.submit(process_image, image_data)
add_task_info(process_future, {
'type': 'process',
'image_id': image_data['id']
})
# 为处理任务添加回调
process_future.add_done_callback(callback_function)
process_future.add_done_callback(error_callback)
processed_futures.append(process_future)
except Exception as e:
print(f"下载任务失败: {e}")
# 等待所有处理任务完成
print("等待所有图片处理完成...")
processed_results = []
for future in concurrent.futures.as_completed(processed_futures, timeout=30):
try:
result = future.result()
if result:
processed_results.append(result)
except Exception as e:
print(f"处理任务失败: {e}")
# 统计结果
print(f"\n=== 任务完成统计 ===")
print(f"下载任务: {len(download_results)}/{len(image_ids)} 成功")
print(f"处理任务: {len(processed_results)}/{len(download_results)} 成功")
if download_results:
total_download_time = sum(r['download_time'] for r in download_results)
avg_download_time = total_download_time / len(download_results)
total_size = sum(r['size'] for r in download_results)
print(f"下载统计: 总大小{total_size}KB, 平均下载时间{avg_download_time:.1f}s")
if processed_results:
total_process_time = sum(r['process_time'] for r in processed_results)
avg_process_time = total_process_time / len(processed_results)
total_processed_size = sum(r['processed_size'] for r in processed_results)
print(f"处理统计: 总压缩大小{total_processed_size}KB, 平均处理时间{avg_process_time:.1f}s")
def test_progressive_task_submission():
"""测试渐进式任务提交"""
print("\n=== 渐进式任务提交测试 ===")
with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
pending_futures = []
completed_count = 0
total_tasks = 10
# 逐步提交任务
def submit_next_task():
nonlocal completed_count
if completed_count < total_tasks:
task_id = completed_count + 1
print(f"提交任务 {task_id}/{total_tasks}")
future = executor.submit(download_image, task_id)
future.task_info = {'type': 'progressive', 'task_id': task_id}
future.add_done_callback(callback_function)
pending_futures.append(future)
completed_count += 1
# 初次提交几个任务
for _ in range(3):
submit_next_task()
# 当有任务完成时，提交新任务
while pending_futures:
# 等待任意任务完成
done, _ = concurrent.futures.wait(
pending_futures,
return_when=concurrent.futures.FIRST_COMPLETED
)
# 移除已完成的任务
for future in done:
pending_futures.remove(future)
# 提交新任务（如果还有）
if completed_count < total_tasks:
submit_next_task()
print("所有渐进式任务已提交并完成")
if __name__ == '__main__':
test_async_callbacks()
test_progressive_task_submission()
---
04.高级进程池配置
a.进程池性能优化
a.max_workers参数优化
max_workers应根据CPU核心数和任务特性来设置。CPU密集型任务建议设置为CPU核心数，IO密集型任务可以适当增加。
b.代码示例
---
# ProcessPoolExecutor高级配置示例
import concurrent.futures
import time
import os
import multiprocessing
import random
from threading import Lock
import threading
# 全局统计锁
stats_lock = Lock()
global_stats = {
'cpu_tasks': 0,
'io_tasks': 0,
'mixed_tasks': 0,
'start_times': {},
'process_usage': {}
}
def cpu_intensive_task(task_id, complexity):
"""CPU密集型任务"""
pid = os.getpid()
start_time = time.time()
# 模拟CPU密集型计算
result = 0
for i in range(complexity * 100000):
result += i * i
# 更新统计
end_time = time.time()
with stats_lock:
global_stats['cpu_tasks'] += 1
if pid not in global_stats['process_usage']:
global_stats['process_usage'][pid] = 0
global_stats['process_usage'][pid] += 1
print(f"CPU任务{task_id}完成，进程{pid}，复杂度{complexity}，耗时{end_time-start_time:.2f}s")
return {
'task_id': task_id,
'type': 'cpu',
'complexity': complexity,
'result': result,
'duration': end_time - start_time,
'pid': pid
}
def io_intensive_task(task_id, operations):
"""IO密集型任务（模拟）"""
pid = os.getpid()
start_time = time.time()
# 模拟IO操作（文件读写、网络请求等）
total_data = 0
for i in range(operations):
# 模拟IO延迟
time.sleep(0.01)
data = random.randint(1000, 10000)
total_data += data
end_time = time.time()
with stats_lock:
global_stats['io_tasks'] += 1
if pid not in global_stats['process_usage']:
global_stats['process_usage'][pid] = 0
global_stats['process_usage'][pid] += 1
print(f"IO任务{task_id}完成，进程{pid}，操作数{operations}，耗时{end_time-start_time:.2f}s")
return {
'task_id': task_id,
'type': 'io',
'operations': operations,
'result': total_data,
'duration': end_time - start_time,
'pid': pid
}
def mixed_task(task_id, cpu_work, io_operations):
"""混合型任务"""
pid = os.getpid()
start_time = time.time()
# 先进行CPU工作
cpu_result = sum(i * i for i in range(cpu_work * 1000))
# 再进行IO操作
for _ in range(io_operations):
time.sleep(0.005)
end_time = time.time()
with stats_lock:
global_stats['mixed_tasks'] += 1
if pid not in global_stats['process_usage']:
global_stats['process_usage'][pid] = 0
global_stats['process_usage'][pid] += 1
print(f"混合任务{task_id}完成，进程{pid}，CPU复杂度{cpu_work}，IO操作{io_operations}，耗时{end_time-start_time:.2f}s")
return {
'task_id': task_id,
'type': 'mixed',
'cpu_work': cpu_work,
'io_operations': io_operations,
'result': cpu_result,
'duration': end_time - start_time,
'pid': pid
}
def test_worker_count_optimization():
"""测试不同工作进程数的性能"""
print("=== 进程池工作进程数优化测试 ===")
cpu_count = multiprocessing.cpu_count()
print(f"系统CPU核心数: {cpu_count}")
# 测试不同的工作进程数
worker_counts = [1, 2, cpu_count//2, cpu_count, cpu_count*2]
# 准备测试任务
test_tasks = []
for i in range(20):
task_type = random.choice(['cpu', 'io', 'mixed'])
if task_type == 'cpu':
test_tasks.append(('cpu', i+1, random.randint(1, 5)))
elif task_type == 'io':
test_tasks.append(('io', i+1, random.randint(5, 15)))
else:
test_tasks.append(('mixed', i+1, random.randint(2, 4), random.randint(3, 8)))
results = {}
for worker_count in worker_counts:
print(f"\n测试工作进程数: {worker_count}")
# 重置统计
with stats_lock:
global_stats['cpu_tasks'] = 0
global_stats['io_tasks'] = 0
global_stats['mixed_tasks'] = 0
global_stats['process_usage'] = {}
start_time = time.time()
with concurrent.futures.ProcessPoolExecutor(max_workers=worker_count) as executor:
futures = []
# 提交任务
for task in test_tasks:
if task[0] == 'cpu':
future = executor.submit(cpu_intensive_task, task[1], task[2])
elif task[0] == 'io':
future = executor.submit(io_intensive_task, task[1], task[2])
else:
future = executor.submit(mixed_task, task[1], task[2], task[3])
futures.append(future)
# 等待所有任务完成
task_results = []
for future in concurrent.futures.as_completed(futures):
try:
result = future.result()
task_results.append(result)
except Exception as e:
print(f"任务执行失败: {e}")
end_time = time.time()
total_time = end_time - start_time
# 记录性能统计
results[worker_count] = {
'total_time': total_time,
'completed_tasks': len(task_results),
'avg_task_time': total_time / len(task_results) if task_results else 0,
'process_usage': len(global_stats['process_usage'])
}
print(f"完成时间: {total_time:.2f}s")
print(f"完成任务数: {len(task_results)}")
print(f"平均任务时间: {total_time / len(task_results):.2f}s")
print(f"使用进程数: {len(global_stats['process_usage'])}")
# 分析结果
print(f"\n=== 性能分析结果 ===")
best_config = min(results.items(), key=lambda x: x[1]['total_time'])
for worker_count, stats in results.items():
efficiency = best_config[1]['total_time'] / stats['total_time']
print(f"工作进程数{worker_count}: {stats['total_time']:.2f}s, "
f"效率{efficiency:.2%}")
print(f"\n最佳配置: {best_config[0]}个工作进程, 用时{best_config[1]['total_time']:.2f}s")
def initialize_worker(worker_id):
"""工作进程初始化函数"""
print(f"工作进程初始化: PID={os.getpid()}, Worker_ID={worker_id}")
# 可以在这里进行每个进程的初始化工作，如建立数据库连接等
return worker_id
def test_custom_initializer():
"""测试自定义初始化函数"""
print("\n=== 自定义初始化函数测试 ===")
def worker_with_init():
"""使用初始化数据的任务函数"""
pid = os.getpid()
# 模拟使用初始化时建立的连接
print(f"任务执行，进程{pid}使用已初始化的资源")
time.sleep(random.uniform(0.5, 1.5))
return pid
with concurrent.futures.ProcessPoolExecutor(
max_workers=3,
initializer=initialize_worker,
initargs=("custom_init",)
) as executor:
futures = []
# 提交多个任务
for i in range(6):
future = executor.submit(worker_with_init)
futures.append(future)
# 收集结果
for future in concurrent.futures.as_completed(futures):
try:
result = future.result()
print(f"任务完成，进程{result}")
except Exception as e:
print(f"任务失败: {e}")
def test_error_handling():
"""测试进程池错误处理"""
print("\n=== 进程池错误处理测试 ===")
def failing_task(task_id, should_fail=False):
"""可能失败的任务"""
pid = os.getpid()
if should_fail:
raise ValueError(f"任务{task_id}在进程{pid}中故意失败")
else:
time.sleep(0.5)
return f"任务{task_id}在进程{pid}中成功完成"
with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
futures = []
# 提交一些正常任务和一些失败任务
for i in range(6):
should_fail = i in [2, 4] # 第3和第5个任务会失败
future = executor.submit(failing_task, i+1, should_fail)
futures.append(future)
# 处理结果和错误
successful_tasks = 0
failed_tasks = 0
for i, future in enumerate(futures):
try:
result = future.result(timeout=3)
print(f"任务成功: {result}")
successful_tasks += 1
except Exception as e:
print(f"任务失败: {e}")
failed_tasks += 1
print(f"错误处理统计: 成功{successful_tasks}个, 失败{failed_tasks}个")
if __name__ == '__main__':
test_worker_count_optimization()
test_custom_initializer()
test_error_handling()
---
b.进程池监控与管理
a健康检查机制
实现对进程池的健康检查，监控进程状态、任务执行情况，及时发现和处理异常情况。
b.代码示例
---
# 进程池监控与管理示例
import concurrent.futures
import time
import os
import psutil
import threading
from queue import Queue, Empty
from collections import defaultdict, deque
class ProcessPoolMonitor:
"""进程池监控器"""
def __init__(self, executor, check_interval=1.0):
self.executor = executor
self.check_interval = check_interval
self.monitoring = False
self.monitor_thread = None
# 统计数据
self.stats = {
'submitted_tasks': 0,
'completed_tasks': 0,
'failed_tasks': 0,
'active_workers': 0,
'task_history': deque(maxlen=100),
'worker_stats': defaultdict(dict),
'system_stats': deque(maxlen=60), # 保存60秒的系统统计
}
# 任务跟踪
self.active_futures = {}
self.result_queue = Queue()
def start_monitoring(self):
"""启动监控"""
if self.monitoring:
return
self.monitoring = True
self.monitor_thread = threading.Thread(target=self._monitor_worker, daemon=True)
self.monitor_thread.start()
print("进程池监控已启动")
def stop_monitoring(self):
"""停止监控"""
self.monitoring = False
if self.monitor_thread:
self.monitor_thread.join(timeout=5)
print("进程池监控已停止")
def _monitor_worker(self):
"""监控工作线程"""
while self.monitoring:
try:
self._collect_stats()
time.sleep(self.check_interval)
except Exception as e:
print(f"监控线程异常: {e}")
def _collect_stats(self):
"""收集统计数据"""
current_time = time.time()
# 收集系统统计
system_info = {
'timestamp': current_time,
'cpu_percent': psutil.cpu_percent(),
'memory_percent': psutil.virtual_memory().percent,
'active_futures': len(self.active_futures),
'completed_tasks': self.stats['completed_tasks'],
'failed_tasks': self.stats['failed_tasks']
}
self.stats['system_stats'].append(system_info)
# 定期打印监控信息
if len(self.stats['system_stats']) % 10 == 0:
self._print_status()
def _print_status(self):
"""打印状态信息"""
recent_stats = self.stats['system_stats'][-1] if self.stats['system_stats'] else {}
print(f"\n=== 进程池状态报告 (时间: {time.strftime('%H:%M:%S')}) ===")
print(f"活跃任务: {recent_stats.get('active_futures', 0)}")
print(f"已完成任务: {self.stats['completed_tasks']}")
print(f"失败任务: {self.stats['failed_tasks']}")
print(f"系统CPU使用率: {recent_stats.get('cpu_percent', 0):.1f}%")
print(f"系统内存使用率: {recent_stats.get('memory_percent', 0):.1f}%")
if recent_stats.get('active_futures', 0) > 0:
print(f"活跃任务ID: {list(self.active_futures.keys())}")
def submit_task(self, fn, *args, **kwargs):
"""提交任务并监控"""
future = self.executor.submit(fn, *args, **kwargs)
task_id = future.task_id = id(future)
self.active_futures[task_id] = {
'future': future,
'start_time': time.time(),
'function': fn.__name__,
'args': args,
'kwargs': kwargs
}
self.stats['submitted_tasks'] += 1
# 添加完成回调
future.add_done_callback(self._task_completed_callback)
return future
def _task_completed_callback(self, future):
"""任务完成回调"""
task_id = future.task_id
if task_id in self.active_futures:
task_info = self.active_futures[task_id]
end_time = time.time()
duration = end_time - task_info['start_time']
try:
result = future.result()
self.stats['completed_tasks'] += 1
status = 'success'
print(f"任务{task_id}成功完成，耗时{duration:.2f}s")
except Exception as e:
self.stats['failed_tasks'] += 1
status = 'failed'
print(f"任务{task_id}执行失败: {e}")
# 记录任务历史
self.stats['task_history'].append({
'task_id': task_id,
'function': task_info['function'],
'duration': duration,
'status': status,
'end_time': end_time
})
# 从活跃任务中移除
del self.active_futures[task_id]
def get_performance_report(self):
"""获取性能报告"""
if not self.stats['task_history']:
return "暂无任务执行历史"
completed_tasks = [t for t in self.stats['task_history'] if t['status'] == 'success']
failed_tasks = [t for t in self.stats['task_history'] if t['status'] == 'failed']
total_tasks = len(self.stats['task_history'])
success_rate = len(completed_tasks) / total_tasks if total_tasks > 0 else 0
if completed_tasks:
avg_duration = sum(t['duration'] for t in completed_tasks) / len(completed_tasks)
max_duration = max(t['duration'] for t in completed_tasks)
min_duration = min(t['duration'] for t in completed_tasks)
else:
avg_duration = max_duration = min_duration = 0
report = f"""
=== 进程池性能报告 ===
总任务数: {total_tasks}
成功任务数: {len(completed_tasks)}
失败任务数: {len(failed_tasks)}
成功率: {success_rate:.2%}
执行时间统计:
- 平均耗时: {avg_duration:.2f}s
- 最大耗时: {max_duration:.2f}s
- 最小耗时: {min_duration:.2f}s
当前状态:
- 提交任务: {self.stats['submitted_tasks']}
- 完成任务: {self.stats['completed_tasks']}
- 失败任务: {self.stats['failed_tasks']}
- 活跃任务: {len(self.active_futures)}
"""
return report
def monitored_heavy_computation(task_id, data_size):
"""被监控的重度计算任务"""
pid = os.getpid()
print(f"重度计算任务{task_id}开始，进程{pid}，数据量{data_size}")
# 模拟重度计算
result = sum(i * i * i for i in range(data_size))
time.sleep(2) # 模拟处理时间
print(f"重度计算任务{task_id}完成，进程{pid}")
return {
'task_id': task_id,
'pid': pid,
'result': result,
'data_size': data_size
}
def monitored_io_simulation(task_id, io_count):
"""被监控的IO模拟任务"""
pid = os.getpid()
print(f"IO模拟任务{task_id}开始，进程{pid}，IO次数{io_count}")
total_data = 0
for i in range(io_count):
# 模拟IO操作
time.sleep(0.1)
data = random.randint(100, 1000)
total_data += data
print(f"IO模拟任务{task_id}完成，进程{pid}，总数据{total_data}")
return {
'task_id': task_id,
'pid': pid,
'total_data': total_data,
'io_count': io_count
}
def test_process_pool_monitoring():
"""测试进程池监控"""
print("=== 进程池监控测试 ===")
# 创建进程池
executor = concurrent.futures.ProcessPoolExecutor(max_workers=4)
# 创建监控器
monitor = ProcessPoolMonitor(executor, check_interval=2.0)
monitor.start_monitoring()
try:
# 提交各种类型的任务
futures = []
# 提交重度计算任务
for i in range(3):
future = monitor.submit_task(
monitored_heavy_computation,
f"cpu_{i+1}",
(i+1) * 50000
)
futures.append(future)
# 提交IO模拟任务
for i in range(4):
future = monitor.submit_task(
monitored_io_simulation,
f"io_{i+1}",
(i+1) * 10
)
futures.append(future)
# 提交一些可能失败的任务
def failing_task():
time.sleep(1)
raise RuntimeError("故意失败的测试任务")
for i in range(2):
future = monitor.submit_task(failing_task)
futures.append(future)
# 等待所有任务完成
print("等待所有任务完成...")
concurrent.futures.wait(futures, timeout=30)
finally:
# 停止监控
time.sleep(2) # 等待最后的统计
monitor.stop_monitoring()
# 打印性能报告
print(monitor.get_performance_report())
# 关闭进程池
executor.shutdown(wait=True)
print("进程池已关闭")
if __name__ == '__main__':
test_process_pool_monitoring()
---
6.3 进程间通信
01.进程间通信概述
a.IPC机制分类
进程间通信（IPC）是多进程编程的核心概念，不同进程间需要通过各种机制来交换数据和同步状态。Python multiprocessing模块提供了丰富的IPC机制：Queue（队列）、Pipe（管道）、Manager（管理器）、Value/Array（共享变量）、Event（事件）、Condition（条件变量）、Semaphore（信号量）等。每种机制都有其特定的适用场景和性能特征。
b.通信方式选择指南
aQueue队列通信
适合生产者-消费者模式，支持多对多通信，内置同步机制，使用简单。
b.Pipe管道通信
适合点对点双向通信，性能较高，但只适合两个进程间通信。
c.Manager管理器
适合共享复杂对象和状态，支持多进程访问，但有性能开销。
d.Value/Array共享内存
适合共享简单数据类型，性能高，但需要手动处理同步。
02.Queue高级应用
a.多级队列通信
a分层通信模式
在复杂的多进程应用中，可以使用多级队列实现分层通信架构，如主进程与工作组通信，工作组内进程间通信，不同工作组间协调等。
b.代码示例
---
# 多级队列通信架构示例
import multiprocessing
import time
import random
from collections import defaultdict, deque
from datetime import datetime
class TaskDispatcher:
"""任务分发器"""
def __init__(self):
self.main_queue = multiprocessing.Queue()
self.worker_queues = {} # worker_id -> queue
self.result_queue = multiprocessing.Queue()
self.control_queue = multiprocessing.Queue()
self.running = True
def create_worker_queue(self, worker_id):
"""为工作进程创建专用队列"""
queue = multiprocessing.Queue()
self.worker_queues[worker_id] = queue
return queue
def dispatch_task(self, task):
"""分发任务到合适的工作队列"""
worker_id = self._select_worker(task)
if worker_id in self.worker_queues:
self.worker_queues[worker_id].put(task)
return worker_id
return None
def _select_worker(self, task):
"""选择工作进程的算法"""
# 简单的轮询算法
worker_ids = list(self.worker_queues.keys())
if worker_ids:
return worker_ids[hash(str(task)) % len(worker_ids)]
return None
def stop(self):
"""停止分发器"""
self.running = False
class WorkerProcess:
"""工作进程"""
def __init__(self, worker_id, task_queue, result_queue, control_queue):
self.worker_id = worker_id
self.task_queue = task_queue
self.result_queue = result_queue
self.control_queue = control_queue
self.processed_count = 0
self.start_time = None
def run(self):
"""运行工作进程"""
self.start_time = time.time()
pid = multiprocessing.current_process().pid
print(f"工作进程{self.worker_id} (PID: {pid}) 启动")
while True:
try:
# 检查控制消息
if not self.control_queue.empty():
try:
control_msg = self.control_queue.get_nowait()
if control_msg.get('command') == 'stop':
print(f"工作进程{self.worker_id}收到停止命令")
break
elif control_msg.get('command') == 'status':
self._send_status()
except:
pass
# 获取任务
try:
task = self.task_queue.get(timeout=1.0)
result = self._process_task(task)
self.result_queue.put(result)
self.processed_count += 1
except:
continue
except Exception as e:
print(f"工作进程{self.worker_id}异常: {e}")
self._send_final_status()
print(f"工作进程{self.worker_id}结束")
def _process_task(self, task):
"""处理单个任务"""
task_type = task.get('type', 'unknown')
task_id = task.get('id', 'unknown')
start_time = time.time()
if task_type == 'computation':
result = self._handle_computation(task)
elif task_type == 'io_simulation':
result = self._handle_io_simulation(task)
elif task_type == 'mixed':
result = self._handle_mixed_task(task)
else:
result = {'error': f'未知任务类型: {task_type}'}
end_time = time.time()
result.update({
'worker_id': self.worker_id,
'task_id': task_id,
'processing_time': end_time - start_time,
'timestamp': datetime.now()
})
return result
def _handle_computation(self, task):
"""处理计算任务"""
complexity = task.get('complexity', 1000000)
result = sum(i * i for i in range(complexity))
return {'type': 'computation', 'result': result, 'complexity': complexity}
def _handle_io_simulation(self, task):
"""处理IO模拟任务"""
operations = task.get('operations', 10)
total_data = 0
for _ in range(operations):
time.sleep(0.01) # 模拟IO延迟
total_data += random.randint(100, 1000)
return {'type': 'io_simulation', 'total_data': total_data, 'operations': operations}
def _handle_mixed_task(self, task):
"""处理混合任务"""
# 先计算
computation_result = sum(i for i in range(50000))
# 再IO
time.sleep(0.1)
return {'type': 'mixed', 'computation_result': computation_result}
def _send_status(self):
"""发送状态信息"""
status = {
'worker_id': self.worker_id,
'type': 'status',
'processed_count': self.processed_count,
'uptime': time.time() - self.start_time if self.start_time else 0
}
self.result_queue.put(status)
def _send_final_status(self):
"""发送最终状态"""
final_status = {
'worker_id': self.worker_id,
'type': 'final_status',
'processed_count': self.processed_count,
'total_runtime': time.time() - self.start_time if self.start_time else 0
}
self.result_queue.put(final_status)
def test_multilevel_queues():
"""测试多级队列通信"""
print("=== 多级队列通信测试 ===")
# 创建分发器
dispatcher = TaskDispatcher()
# 创建工作进程
num_workers = 4
workers = []
worker_processes = []
for i in range(num_workers):
# 为每个工作进程创建专用队列
task_queue = dispatcher.create_worker_queue(i+1)
result_queue = dispatcher.result_queue
control_queue = dispatcher.control_queue
worker = WorkerProcess(i+1, task_queue, result_queue, control_queue)
process = multiprocessing.Process(target=worker.run)
workers.append(worker)
worker_processes.append(process)
process.start()
# 等待工作进程启动
time.sleep(1)
# 生成测试任务
task_types = ['computation', 'io_simulation', 'mixed']
tasks = []
for i in range(20):
task_type = random.choice(task_types)
if task_type == 'computation':
task = {
'type': 'computation',
'id': f'comp_{i+1}',
'complexity': random.randint(100000, 500000)
}
elif task_type == 'io_simulation':
task = {
'type': 'io_simulation',
'id': f'io_{i+1}',
'operations': random.randint(5, 15)
}
else:
task = {
'type': 'mixed',
'id': f'mixed_{i+1}'
}
tasks.append(task)
print(f"生成了{len(tasks)}个任务，开始分发...")
# 分发任务
dispatched_tasks = 0
for task in tasks:
worker_id = dispatcher.dispatch_task(task)
if worker_id:
dispatched_tasks += 1
print(f"任务{task['id']}分发到工作进程{worker_id}")
time.sleep(0.1) # 控制分发速度
print(f"成功分发{dispatched_tasks}个任务")
# 监控执行进度
completed_tasks = []
worker_stats = defaultdict(list)
start_monitor_time = time.time()
timeout = 30 # 30秒超时
while len(completed_tasks) < dispatched_tasks and (time.time() - start_monitor_time) < timeout:
try:
# 收集结果
while not dispatcher.result_queue.empty():
try:
result = dispatcher.result_queue.get_nowait()
if result['type'] in ['computation', 'io_simulation', 'mixed']:
completed_tasks.append(result)
worker_id = result['worker_id']
worker_stats[worker_id].append(result)
print(f"任务{result['task_id']}完成，工作进程{worker_id}，"
f"耗时{result['processing_time']:.3f}s")
elif result['type'] == 'status':
worker_id = result['worker_id']
print(f"工作进程{worker_id}状态：已处理{result['processed_count']}个任务")
except:
break
# 检查工作进程状态
if random.random() < 0.1: # 10%概率发送状态查询
for worker_id in range(1, num_workers + 1):
dispatcher.control_queue.put({'command': 'status'})
time.sleep(0.5)
except KeyboardInterrupt:
print("用户中断，正在停止...")
break
# 发送停止命令
print("发送停止命令给所有工作进程...")
for _ in range(num_workers):
dispatcher.control_queue.put({'command': 'stop'})
# 等待进程结束
for process in worker_processes:
process.join(timeout=5)
if process.is_alive():
process.terminate()
# 收集最终结果
while not dispatcher.result_queue.empty():
try:
result = dispatcher.result_queue.get_nowait()
if result['type'] == 'final_status':
worker_id = result['worker_id']
print(f"工作进程{worker_id}最终状态：处理{result['processed_count']}个任务，"
f"运行{result['total_runtime']:.2f}秒")
except:
break
# 分析结果
print(f"\n=== 执行结果分析 ===")
print(f"总任务数: {len(tasks)}")
print(f"分发任务数: {dispatched_tasks}")
print(f"完成任务数: {len(completed_tasks)}")
if completed_tasks:
total_time = sum(t['processing_time'] for t in completed_tasks)
avg_time = total_time / len(completed_tasks)
print(f"平均处理时间: {avg_time:.3f}秒")
# 按任务类型分析
type_stats = defaultdict(list)
for task in completed_tasks:
type_stats[task['type']].append(task['processing_time'])
for task_type, times in type_stats.items():
avg_type_time = sum(times) / len(times)
print(f"{task_type}任务: {len(times)}个，平均时间{avg_type_time:.3f}秒")
# 工作进程分析
for worker_id, worker_tasks in worker_stats.items():
if worker_tasks:
worker_time = sum(t['processing_time'] for t in worker_tasks)
print(f"工作进程{worker_id}: 处理{len(worker_tasks)}个任务，总耗时{worker_time:.3f}秒")
if __name__ == '__main__':
test_multilevel_queues()
---
b.优先级队列实现
a优先级队列概念
优先级队列允许根据任务优先级来处理任务，高优先级任务会优先执行。可以通过multiprocessing.Queue和自定义排序逻辑实现。
b.代码示例
---
# 优先级队列实现示例
import multiprocessing
import time
import random
import heapq
from datetime import datetime
class PriorityTask:
"""优先级任务类"""
def __init__(self, priority, task_id, task_data):
self.priority = priority
self.task_id = task_id
self.task_data = task_data
self.timestamp = time.time()
def __lt__(self, other):
# 优先级数字越小，优先级越高
if self.priority != other.priority:
return self.priority < other.priority
# 相同优先级按时间排序
return self.timestamp < other.timestamp
def __repr__(self):
return f"PriorityTask(id={self.task_id}, priority={self.priority})"
class PriorityQueueManager:
"""优先级队列管理器"""
def __init__(self):
self.input_queue = multiprocessing.Queue()
self.priority_queue = []
self.output_queue = multiprocessing.Queue()
self.running = True
self.lock = multiprocessing.Lock()
def add_task(self, priority, task_id, task_data):
"""添加任务到优先级队列"""
task = PriorityTask(priority, task_id, task_data)
self.input_queue.put(task)
print(f"添加任务{task_id}，优先级{priority}")
def get_next_task(self, timeout=None):
"""获取下一个最高优先级任务"""
try:
return self.output_queue.get(timeout=timeout)
except:
return None
def stop(self):
"""停止队列管理器"""
self.running = False
def priority_queue_worker(input_queue, priority_queue_list, output_queue, lock):
"""优先级队列工作线程"""
print(f"优先级队列工作线程启动，PID: {multiprocessing.current_process().pid}")
while True:
try:
# 从输入队列获取任务
task = input_queue.get(timeout=1.0)
if task is None: # 停止信号
break
with lock:
# 将任务添加到优先级队列
heapq.heappush(priority_queue_list, task)
print(f"任务{task.task_id}已加入优先级队列 (当前队列长度: {len(priority_queue_list)})")
# 如果队列中有任务，处理最高优先级的任务
if priority_queue_list:
next_task = heapq.heappop(priority_queue_list)
output_queue.put(next_task)
print(f"处理任务{next_task.task_id} (优先级: {next_task.priority})")
except:
continue
print("优先级队列工作线程结束")
def task_worker(task_queue, worker_id):
"""任务工作进程"""
print(f"任务工作进程{worker_id}启动，PID: {multiprocessing.current_process().pid}")
processed_count = 0
while True:
try:
# 从优先级队列获取任务
priority_task = task_queue.get(timeout=3.0)
if priority_task is None:
break
task_id = priority_task.task_id
priority = priority_task.priority
task_data = priority_task.task_data
print(f"工作进程{worker_id}开始处理任务{task_id} (优先级: {priority})")
# 根据任务类型处理
start_time = time.time()
if task_data['type'] == 'urgent':
# 紧急任务 - 快速处理
result = sum(i * i for i in range(100000))
processing_time = 0.1
elif task_data['type'] == 'normal':
# 普通任务 - 标准处理
result = sum(i * i for i in range(500000))
processing_time = 0.5
else:
# 低优先级任务 - 慢速处理
result = sum(i * i for i in range(1000000))
processing_time = 1.0
time.sleep(processing_time)
end_time = time.time()
processed_count += 1
total_time = end_time - start_time
print(f"工作进程{worker_id}完成任务{task_id}，结果: {result:,}, "
f"耗时: {total_time:.2f}s")
# 可以在这里发送结果到结果队列
# result_queue.put({'worker_id': worker_id, 'task_id': task_id, 'result': result})
except:
break
print(f"任务工作进程{worker_id}结束，共处理{processed_count}个任务")
def test_priority_queue():
"""测试优先级队列"""
print("=== 优先级队列测试 ===")
# 创建优先级队列管理器
manager = multiprocessing.Manager()
priority_list = manager.list()
lock = manager.Lock()
# 创建队列
input_queue = multiprocessing.Queue()
output_queue = multiprocessing.Queue()
result_queue = multiprocessing.Queue()
# 启动优先级队列管理进程
queue_manager = multiprocessing.Process(
target=priority_queue_worker,
args=(input_queue, priority_list, output_queue, lock)
)
queue_manager.start()
# 等待队列管理器启动
time.sleep(0.5)
# 启动任务工作进程
num_workers = 3
workers = []
for i in range(num_workers):
worker = multiprocessing.Process(
target=task_worker,
args=(output_queue, i+1)
)
workers.append(worker)
worker.start()
# 生成不同优先级的任务
tasks = []
task_types = ['urgent', 'normal', 'low']
# 生成20个随机任务
for i in range(20):
task_type = random.choice(task_types)
if task_type == 'urgent':
priority = random.randint(1, 3) # 高优先级 (1-3)
elif task_type == 'normal':
priority = random.randint(4, 6) # 中等优先级 (4-6)
else:
priority = random.randint(7, 10) # 低优先级 (7-10)
task = {
'type': task_type,
'data': f'task_data_{i+1}'
}
tasks.append((priority, f'task_{i+1}', task))
# 随机打乱任务顺序
random.shuffle(tasks)
print(f"生成{len(tasks)}个任务，开始按随机顺序添加到优先级队列...")
# 添加任务到优先级队列
for priority, task_id, task_data in tasks:
input_queue.put(PriorityTask(priority, task_id, task_data))
time.sleep(random.uniform(0.05, 0.15)) # 随机间隔添加任务
print("所有任务已添加，等待处理完成...")
time.sleep(5) # 等待任务处理
# 发送停止信号
input_queue.put(None)
for _ in range(num_workers):
output_queue.put(None)
# 等待进程结束
queue_manager.join(timeout=5)
for worker in workers:
worker.join(timeout=5)
if worker.is_alive():
worker.terminate()
print("优先级队列测试完成")
if __name__ == '__main__':
test_priority_queue()
---
03.Pipe通信模式
a.多路复用Pipe
a管道复用概念
使用select或poll机制可以同时监控多个管道，实现多路复用，提高通信效率和响应性。
b.代码示例
---
# 多路复用Pipe通信示例
import multiprocessing
import time
import random
import select
from datetime import datetime
class MultiplexerProcess:
"""多路复用进程"""
def __init__(self):
self.connections = {} # conn_id -> (conn, last_activity)
self.running = True
self.message_count = 0
self.start_time = time.time()
def add_connection(self, conn_id, conn):
"""添加连接"""
self.connections[conn_id] = {'conn': conn, 'last_activity': time.time(), 'message_count': 0}
print(f"多路复用器：添加连接{conn_id}")
def remove_connection(self, conn_id):
"""移除连接"""
if conn_id in self.connections:
conn = self.connections[conn_id]['conn']
try:
conn.close()
except:
pass
del self.connections[conn_id]
print(f"多路复用器：移除连接{conn_id}")
def run_multiplexer(self):
"""运行多路复用器"""
print(f"多路复用器启动，PID: {multiprocessing.current_process().pid}")
while self.running and self.connections:
try:
# 获取所有连接的管道
pipes = [conn_info['conn'] for conn_info in self.connections.values()]
if not pipes:
break
# 使用select监控管道（注意：Windows不支持select for pipe）
# 这里使用polling方式替代
readable_conns = []
for conn_id, conn_info in self.connections.items():
try:
if conn_info['conn'].poll():
readable_conns.append(conn_id)
except:
self.remove_connection(conn_id)
# 处理可读的连接
for conn_id in readable_conns:
try:
conn_info = self.connections[conn_id]
conn = conn_info['conn']
# 接收消息
message = conn.recv()
self.message_count += 1
conn_info['message_count'] += 1
conn_info['last_activity'] = time.time()
# 处理消息
response = self._handle_message(conn_id, message)
# 发送响应
if response:
conn.send(response)
except Exception as e:
print(f"多路复用器处理连接{conn_id}时出错: {e}")
self.remove_connection(conn_id)
time.sleep(0.01) # 短暂休眠，避免CPU占用过高
except Exception as e:
print(f"多路复用器异常: {e}")
break
runtime = time.time() - self.start_time
print(f"多路复用器结束，运行时间: {runtime:.2f}秒，处理消息: {self.message_count}条")
def _handle_message(self, conn_id, message):
"""处理接收到的消息"""
msg_type = message.get('type', 'unknown')
timestamp = datetime.now().strftime('%H:%M:%S')
if msg_type == 'data':
data = message.get('data', '')
print(f"[{timestamp}] 多路复用器收到来自{conn_id}的数据: {data}")
return {
'type': 'response',
'status': 'received',
'data': f"收到您的数据: {data}",
'timestamp': timestamp
}
elif msg_type == 'ping':
print(f"[{timestamp}] 多路复用器收到来自{conn_id}的ping")
return {
'type': 'pong',
'timestamp': timestamp
}
elif msg_type == 'status':
return {
'type': 'status_response',
'active_connections': len(self.connections),
'total_messages': self.message_count,
'timestamp': timestamp
}
elif msg_type == 'close':
print(f"[{timestamp}] 多路复用器收到来自{conn_id}的关闭请求")
self.remove_connection(conn_id)
return None
else:
print(f"[{timestamp}] 多路复用器收到来自{conn_id}的未知消息类型: {msg_type}")
return {
'type': 'error',
'message': f'未知消息类型: {msg_type}'
}
def client_process(client_id, parent_conn, message_count, message_interval):
"""客户端进程"""
print(f"客户端{client_id}启动，PID: {multiprocessing.current_process().pid}")
try:
for i in range(message_count):
# 发送不同类型的消息
message_types = ['data', 'ping', 'status']
msg_type = random.choice(message_types)
if msg_type == 'data':
message = {
'type': 'data',
'data': f'来自客户端{client_id}的消息{i+1}',
'sequence': i+1
}
elif msg_type == 'ping':
message = {
'type': 'ping',
'client_id': client_id,
'sequence': i+1
}
else:
message = {
'type': 'status',
'client_id': client_id
}
# 发送消息
parent_conn.send(message)
print(f"客户端{client_id}发送: {msg_type} (序列: {i+1})")
# 等待响应
if parent_conn.poll(timeout=2.0):
try:
response = parent_conn.recv()
print(f"客户端{client_id}收到响应: {response['type']}")
except Exception as e:
print(f"客户端{client_id}接收响应失败: {e}")
else:
print(f"客户端{client_id}等待响应超时")
time.sleep(message_interval)
# 发送关闭消息
parent_conn.send({'type': 'close', 'client_id': client_id})
print(f"客户端{client_id}发送关闭请求")
except Exception as e:
print(f"客户端{client_id}异常: {e}")
finally:
parent_conn.close()
print(f"客户端{client_id}结束")
def test_multiplexed_pipes():
"""测试多路复用管道"""
print("=== 多路复用管道测试 ===")
# 创建多路复用器
multiplexer = MultiplexerProcess()
# 创建多路复用器进程
multiplexer_process = multiprocessing.Process(
target=multiplexer.run_multiplexer
)
multiplexer_process.start()
# 等待多路复用器启动
time.sleep(1)
# 创建多个客户端进程
num_clients = 4
clients = []
for i in range(num_clients):
# 为每个客户端创建管道
parent_conn, child_conn = multiprocessing.Pipe()
# 将子连接添加到多路复用器
multiplexer.add_connection(f'client_{i+1}', child_conn)
# 创建客户端进程
client_process = multiprocessing.Process(
target=client_process,
args=(f'client_{i+1}', parent_conn,
random.randint(5, 10), # 消息数量
random.uniform(0.5, 1.5)) # 消息间隔
)
clients.append(client_process)
client_process.start()
print(f"启动了{num_clients}个客户端进程")
# 等待所有客户端完成
for client in clients:
client.join()
print("所有客户端已完成")
# 关闭多路复用器
multiplexer.running = False
multiplexer_process.join(timeout=5)
if multiplexer_process.is_alive():
multiplexer_process.terminate()
print("多路复用管道测试完成")
if __name__ == '__main__':
test_multiplexed_pipes()
---
04.共享内存通信
a.Value和Array高级用法
a共享内存特点
multiprocessing.Value和Array提供了在进程间共享简单数据类型的高效机制，数据直接存储在共享内存中，避免了序列化开销，性能较高。
b.代码示例
---
# 共享内存高级用法示例
import multiprocessing
import time
import random
import numpy as np
from multiprocessing import shared_memory
class SharedDataStructure:
"""共享数据结构"""
def __init__(self):
# 创建共享变量
self.counter = multiprocessing.Value('i', 0) # 整数计数器
self.flag = multiprocessing.Value('b', False) # 布尔标志
self.status = multiprocessing.Value('i', 0) # 状态码
# 创建共享数组
self.results = multiprocessing.Array('d', 100) # 双精度浮点数数组
self.timestamps = multiprocessing.Array('d', 100) # 时间戳数组
self.process_ids = multiprocessing.Array('i', 100) # 进程ID数组
# 创建共享字符串（通过Array实现）
self.shared_string = multiprocessing.Array('c', 256) # 字符数组
# 同步锁
self.lock = multiprocessing.Lock()
self.data_ready = multiprocessing.Condition(self.lock)
self.index = multiprocessing.Value('i', 0) # 当前索引
def add_result(self, value, process_id):
"""添加结果到共享数组"""
with self.lock:
current_index = self.index.value
if current_index < len(self.results):
self.results[current_index] = value
self.timestamps[current_index] = time.time()
self.process_ids[current_index] = process_id
self.index.value += 1
# 通知等待的进程
self.data_ready.notify()
return current_index
else:
return -1 # 数组已满
def get_results(self, count=None):
"""获取结果"""
with self.lock:
if count is None:
count = self.index.value
results = []
for i in range(min(count, self.index.value)):
results.append({
'value': self.results[i],
'timestamp': self.timestamps[i],
'process_id': self.process_ids[i]
})
return results
def wait_for_results(self, min_count, timeout=None):
"""等待达到最小数量的结果"""
with self.data_ready:
start_time = time.time()
while self.index.value < min_count:
if timeout and (time.time() - start_time) > timeout:
return False
self.data_ready.wait(timeout=1.0)
return self.index.value >= min_count
def update_string(self, text):
"""更新共享字符串"""
with self.lock:
text_bytes = text.encode('utf-8')[:255] # 限制长度
for i, byte in enumerate(text_bytes):
self.shared_string[i] = byte
self.shared_string[len(text_bytes)] = b'\0' # 字符串结束符
def get_string(self):
"""获取共享字符串"""
with self.lock:
text_bytes = bytearray()
for i in range(256):
byte = self.shared_string[i]
if byte == 0:
break
text_bytes.append(byte)
return text_bytes.decode('utf-8')
def producer_worker(shared_data, worker_id, production_rate):
"""生产者工作进程"""
print(f"生产者{worker_id}启动，PID: {multiprocessing.current_process().pid}")
produced_count = 0
while shared_data.status.value == 0: # status=0表示继续生产
# 产生数据
value = random.uniform(100.0, 1000.0)
process_id = multiprocessing.current_process().pid
# 添加到共享数组
index = shared_data.add_result(value, process_id)
if index >= 0:
produced_count += 1
print(f"生产者{worker_id}生产了第{produced_count}个数据: {value:.2f} (索引: {index})")
# 更新共享字符串
shared_data.update_string(f"生产者{worker_id}已生产{produced_count}个数据")
# 控制生产速率
time.sleep(production_rate)
print(f"生产者{worker_id}结束，共生产{produced_count}个数据")
def consumer_worker(shared_data, consumer_id, consumption_rate):
"""消费者工作进程"""
print(f"消费者{consumer_id}启动，PID: {multiprocessing.current_process().pid}")
consumed_count = 0
last_index = 0
while True:
# 等待新数据
if shared_data.wait_for_results(last_index + 1, timeout=2.0):
# 获取新数据
current_results = shared_data.get_results()
new_results = current_results[last_index:]
for result in new_results:
consumed_count += 1
print(f"消费者{consumer_id}消费了第{consumed_count}个数据: "
f"{result['value']:.2f} "
f"(来自进程: {result['process_id']}, "
f"时间: {time.strftime('%H:%M:%S', time.localtime(result['timestamp']))})")
last_index = len(current_results)
# 显示共享字符串
shared_string = shared_data.get_string()
print(f"消费者{consumer_id}看到共享字符串: {shared_string}")
else:
# 超时，检查是否应该退出
if shared_data.flag.value: # flag=True表示应该退出
break
# 控制消费速率
time.sleep(consumption_rate)
print(f"消费者{consumer_id}结束，共消费{consumed_count}个数据")
def monitor_worker(shared_data, monitor_interval):
"""监控工作进程"""
print(f"监控器启动，PID: {multiprocessing.current_process().pid}")
start_time = time.time()
while True:
with shared_data.lock:
current_index = shared_data.index.value
current_string = shared_data.get_string()
print(f"\n=== 监控报告 (运行时间: {time.time() - start_time:.1f}s) ===")
print(f"共享数组索引: {current_index}/{len(shared_data.results)}")
print(f"共享计数器: {shared_data.counter.value}")
print(f"共享标志: {shared_data.flag.value}")
print(f"共享字符串: {current_string}")
# 显示最近的几个结果
if current_index > 0:
recent_count = min(5, current_index)
print(f"最近的{recent_count}个结果:")
for i in range(max(0, current_index - recent_count), current_index):
print(f" [{i}] 值: {shared_data.results[i]:.2f}, "
f"进程: {shared_data.process_ids[i]}, "
f"时间: {time.strftime('%H:%M:%S', time.localtime(shared_data.timestamps[i]))}")
# 检查是否应该退出
if shared_data.flag.value and current_index == 0:
break
time.sleep(monitor_interval)
print("监控器结束")
def test_shared_memory_advanced():
"""测试高级共享内存用法"""
print("=== 高级共享内存测试 ===")
# 创建共享数据结构
manager = multiprocessing.Manager()
shared_data = SharedDataStructure()
# 设置初始状态
shared_data.status.value = 0 # 0=运行, 1=停止
shared_data.flag.value = False
shared_data.update_string("系统启动")
# 启动生产者进程
producers = []
for i in range(3):
producer = multiprocessing.Process(
target=producer_worker,
args=(shared_data, i+1, random.uniform(0.2, 0.5))
)
producers.append(producer)
producer.start()
# 启动消费者进程
consumers = []
for i in range(2):
consumer = multiprocessing.Process(
target=consumer_worker,
args=(shared_data, i+1, random.uniform(0.3, 0.6))
)
consumers.append(consumer)
consumer.start()
# 启动监控进程
monitor = multiprocessing.Process(
target=monitor_worker,
args=(shared_data, 3.0) # 每3秒报告一次
)
monitor.start()
# 让系统运行一段时间
print("系统运行中...")
time.sleep(15)
# 停止生产者
print("停止生产者...")
shared_data.status.value = 1
# 等待生产者完成
for producer in producers:
producer.join()
print("所有生产者已停止")
# 继续运行一段时间让消费者处理完剩余数据
time.sleep(5)
# 停止消费者和监控器
shared_data.flag.value = True
shared_data.update_string("系统停止")
for consumer in consumers:
consumer.join(timeout=5)
if consumer.is_alive():
consumer.terminate()
monitor.join(timeout=5)
if monitor.is_alive():
monitor.terminate()
# 最终统计
final_results = shared_data.get_results()
print(f"\n=== 最终统计 ===")
print(f"总生产数据: {len(final_results)}")
print(f"共享数组使用率: {len(final_results)}/{len(shared_data.results)} "
f"({len(final_results)/len(shared_data.results)*100:.1f}%)")
if final_results:
avg_value = sum(r['value'] for r in final_results) / len(final_results)
max_value = max(r['value'] for r in final_results)
min_value = min(r['value'] for r in final_results)
print(f"数据统计: 平均值={avg_value:.2f}, 最大值={max_value:.2f}, 最小值={min_value:.2f}")
# 按进程统计
process_stats = {}
for result in final_results:
pid = result['process_id']
if pid not in process_stats:
process_stats[pid] = 0
process_stats[pid] += 1
print("按进程统计:")
for pid, count in process_stats.items():
print(f" 进程{pid}: {count}个数据")
print("高级共享内存测试完成")
if __name__ == '__main__':
test_shared_memory_advanced()
---
b.共享内存块管理
a.SharedMemory类使用
Python 3.8+提供了multiprocessing.shared_memory.SharedMemory类，提供了更灵活的共享内存管理，支持创建、附加、分离和释放共享内存块。
b.代码示例
---
# 共享内存块管理示例
import multiprocessing
import time
import random
import struct
from multiprocessing import shared_memory
import numpy as np
class SharedMemoryManager:
"""共享内存管理器"""
def __init__(self, name="shared_data_block", size=1024*1024): # 1MB
self.name = name
self.size = size
self.shared_block = None
self.header_format = 'QQII' # magic(8) + version(8) + data_count(4) + data_offset(4)
self.header_size = struct.calcsize(self.header_format)
def create_shared_block(self):
"""创建共享内存块"""
try:
# 尝试创建新的共享内存块
self.shared_block = shared_memory.SharedMemory(
create=True,
size=self.size,
name=self.name
)
print(f"创建共享内存块: {self.name}, 大小: {self.size}字节")
# 初始化头部信息
header = struct.pack(
self.header_format,
0x1234567890ABCDEF, # magic number
1, # version
0, # data_count
self.header_size # data_offset
)
self.shared_block.buf[:self.header_size] = header
return True
except FileExistsError:
# 共享内存块已存在，附加到现有块
self.shared_block = shared_memory.SharedMemory(name=self.name)
print(f"附加到现有共享内存块: {self.name}")
return False
def write_data(self, data, data_type='float'):
"""写入数据到共享内存"""
if not self.shared_block:
raise RuntimeError("共享内存块未初始化")
# 读取当前头部信息
header_data = self.shared_block.buf[:self.header_size]
magic, version, data_count, data_offset = struct.unpack(self.header_format, header_data)
# 确定数据类型和大小
if data_type == 'float':
data_array = np.array(data, dtype=np.float64)
format_char = 'd' # double
elif data_type == 'int':
data_array = np.array(data, dtype=np.int64)
format_char = 'q' # long long
elif data_type == 'string':
# 字符串需要特殊处理
data_bytes = data.encode('utf-8')
data_array = np.frombuffer(data_bytes, dtype=np.uint8)
format_char = 'B' # unsigned char
else:
raise ValueError(f"不支持的数据类型: {data_type}")
data_bytes = data_array.tobytes()
required_space = len(data_bytes)
# 检查是否有足够空间
available_space = self.size - data_offset
if required_space > available_space:
raise RuntimeError(f"共享内存空间不足: 需要{required_space}字节, 可用{available_space}字节")
# 写入数据
self.shared_block.buf[data_offset:data_offset + required_space] = data_bytes
# 更新头部信息
new_header = struct.pack(
self.header_format,
magic, version, data_count + 1, data_offset + required_space
)
self.shared_block.buf[:self.header_size] = new_header
print(f"写入{data_type}数据: {len(data)}项, 大小: {required_space}字节")
return data_offset
def read_data(self, max_items=10):
"""从共享内存读取数据"""
if not self.shared_block:
raise RuntimeError("共享内存块未初始化")
# 读取头部信息
header_data = self.shared_block.buf[:self.header_size]
magic, version, data_count, data_offset = struct.unpack(self.header_format, header_data)
if magic != 0x1234567890ABCDEF:
raise RuntimeError("无效的共享内存块")
print(f"读取共享内存: 版本{version}, 数据项数{data_count}, 数据偏移{data_offset}")
# 简化处理：读取最后几项作为示例
if data_count > 0:
# 读取最后几项的double值
item_size = struct.calcsize('d') # 8字节
num_items = min(max_items, (self.size - data_offset) // item_size)
if num_items > 0:
data_bytes = self.shared_block.buf[data_offset:data_offset + num_items * item_size]
values = list(struct.unpack(f'{num_items}d', data_bytes))
return values
return []
def close(self):
"""关闭共享内存块"""
if self.shared_block:
self.shared_block.close()
print(f"关闭共享内存块: {self.name}")
self.shared_block = None
def unlink(self):
"""删除共享内存块"""
if self.shared_block:
self.shared_block.close()
try:
self.shared_block.unlink()
print(f"删除共享内存块: {self.name}")
except FileNotFoundError:
print(f"共享内存块{self.name}已不存在")
self.shared_block = None
def shared_memory_writer(shared_mem_name, writer_id, data_count):
"""共享内存写入进程"""
print(f"写入进程{writer_id}启动，PID: {multiprocessing.current_process().pid}")
manager = SharedMemoryManager(shared_mem_name)
manager.create_shared_block()
try:
for i in range(data_count):
# 生成随机数据
data = [random.uniform(100.0, 1000.0) for _ in range(random.randint(1, 10))]
timestamp = time.time()
# 写入数据
data_with_timestamp = [timestamp] + data
manager.write_data(data_with_timestamp, 'float')
print(f"写入进程{writer_id}写入第{i+1}批数据: {len(data)}个浮点数")
time.sleep(random.uniform(0.5, 1.5))
except Exception as e:
print(f"写入进程{writer_id}异常: {e}")
finally:
manager.close()
def shared_memory_reader(shared_mem_name, reader_id, read_interval, duration):
"""共享内存读取进程"""
print(f"读取进程{reader_id}启动，PID: {multiprocessing.current_process().pid}")
manager = SharedMemoryManager(shared_mem_name)
try:
start_time = time.time()
read_count = 0
while time.time() - start_time < duration:
try:
# 附加到共享内存
manager.create_shared_block()
# 读取数据
data = manager.read_data(max_items=20)
if data:
read_count += 1
print(f"读取进程{reader_id}第{read_count}次读取: {len(data)}个数据值")
print(f" 数据范围: {min(data):.2f} - {max(data):.2f}")
time.sleep(read_interval)
except Exception as e:
print(f"读取进程{reader_id}读取异常: {e}")
time.sleep(read_interval)
print(f"读取进程{reader_id}结束，共读取{read_count}次")
except Exception as e:
print(f"读取进程{reader_id}异常: {e}")
finally:
manager.close()
def test_shared_memory_blocks():
"""测试共享内存块管理"""
print("=== 共享内存块管理测试 ===")
shared_mem_name = f"test_shared_mem_{int(time.time())}"
try:
# 启动写入进程
writer = multiprocessing.Process(
target=shared_memory_writer,
args=(shared_mem_name, 'writer1', 10)
)
writer.start()
# 等待写入进程开始
time.sleep(1)
# 启动多个读取进程
readers = []
for i in range(3):
reader = multiprocessing.Process(
target=shared_memory_reader,
args=(shared_mem_name, f'reader{i+1}', random.uniform(1.0, 2.0), 15)
)
readers.append(reader)
reader.start()
# 等待所有进程完成
writer.join()
for reader in readers:
reader.join()
# 最终读取所有数据
print("\n=== 最终数据读取 ===")
final_manager = SharedMemoryManager(shared_mem_name)
final_manager.create_shared_block()
all_data = final_manager.read_data(max_items=100)
if all_data:
print(f"总共读取到{len(all_data)}个数据值")
print(f"数据统计: 平均值={np.mean(all_data):.2f}, "
f"标准差={np.std(all_data):.2f}")
print(f"范围: {min(all_data):.2f} - {max(all_data):.2f}")
final_manager.close()
finally:
# 清理共享内存
cleanup_manager = SharedMemoryManager(shared_mem_name)
try:
cleanup_manager.create_shared_block()
cleanup_manager.unlink()
except:
pass
print("共享内存块管理测试完成")
if __name__ == '__main__':
test_shared_memory_blocks()
---
6.4 共享内存
01.共享内存基础概念
a.共享内存的特点与优势
共享内存是进程间通信（IPC）中最快的方式，它允许多个进程直接访问同一块物理内存空间，避免了数据在不同进程间的复制开销。在Python中，multiprocessing.shared_memory模块提供了高效的共享内存实现，特别适合需要共享大量数据的高性能场景，如机器学习、图像处理、科学计算等。
b.共享内存与数据同步
a数据一致性挑战
虽然共享内存提供了高性能的数据访问，但多进程同时访问共享数据时会产生数据竞争问题，需要使用适当的同步机制（如锁、信号量）来保证数据一致性。
b.内存管理注意事项
共享内存需要手动管理生命周期，确保在使用完毕后正确释放，避免内存泄漏。同时要注意进程崩溃时共享内存的清理问题。
02.multiprocessing.shared_memory基础
a.SharedMemory类使用
a基本创建和使用
SharedMemory类提供了创建和管理共享内存块的基础功能，支持create=True创建新块，或者attach到现有块。
b.代码示例
---
# SharedMemory基础使用示例
import multiprocessing
import time
import numpy as np
from multiprocessing import shared_memory
def shared_memory_writer(shared_mem_name, data_size):
"""共享内存写入进程"""
print(f"写入进程启动，PID: {multiprocessing.current_process().pid}")
try:
# 创建或连接到共享内存块
existing_shm = shared_memory.SharedMemory(name=shared_mem_name)
# 生成随机数据
data = np.random.random(data_size).astype(np.float64)
print(f"写入进程: 生成了{data_size}个随机数据点")
print(f"写入进程: 数据范围: {data.min():.4f} - {data.max():.4f}")
# 将数据写入共享内存
# 需要确保共享内存足够大
required_bytes = data.nbytes
if required_bytes <= len(existing_shm.buf):
# 将numpy数组直接写入共享内存缓冲区
existing_shm.buf[:required_bytes] = data.tobytes()
print(f"写入进程: 成功写入{required_bytes}字节数据到共享内存")
# 写入元数据（数据大小和数据类型）
metadata = {
'size': data_size,
'dtype': str(data.dtype),
'shape': data.shape,
'nbytes': required_bytes,
'timestamp': time.time()
}
print(f"写入进程: 数据元数据: {metadata}")
else:
print(f"写入进程: 共享内存不足，需要{required_bytes}字节，可用{len(existing_shm.buf)}字节")
except Exception as e:
print(f"写入进程异常: {e}")
finally:
if 'existing_shm' in locals():
existing_shm.close()
print("写入进程结束")
def shared_memory_reader(shared_mem_name, read_count, read_interval):
"""共享内存读取进程"""
print(f"读取进程启动，PID: {multiprocessing.current_process().pid}")
for i in range(read_count):
try:
# 连接到共享内存块
existing_shm = shared_memory.SharedMemory(name=shared_mem_name)
# 检查共享内存中的数据
if len(existing_shm.buf) > 0:
try:
# 读取第一个8字节作为数据长度（简单的元数据）
data_size_bytes = existing_shm.buf[:8]
data_size = int.from_bytes(data_size_bytes, byteorder='little')
if data_size > 0 and data_size * 8 <= len(existing_shm.buf): # 假设double类型
# 读取实际数据
data_bytes = existing_shm.buf[8:8 + data_size * 8]
data = np.frombuffer(data_bytes, dtype=np.float64, count=data_size)
print(f"读取进程第{i+1}次读取: {len(data)}个数据点")
print(f" 数据范围: {data.min():.4f} - {data.max():.4f}")
print(f" 数据均值: {data.mean():.4f}, 标准差: {data.std():.4f}")
else:
print(f"读取进程第{i+1}次读取: 共享内存中暂无有效数据")
except Exception as e:
print(f"读取进程第{i+1}次读取异常: {e}")
else:
print(f"读取进程第{i+1}次读取: 共享内存为空")
existing_shm.close()
except FileNotFoundError:
print(f"读取进程第{i+1}次读取: 共享内存不存在，等待写入进程...")
except Exception as e:
print(f"读取进程第{i+1}次读取异常: {e}")
time.sleep(read_interval)
print("读取进程结束")
def test_basic_shared_memory():
"""测试基础共享内存功能"""
print("=== 基础共享内存测试 ===")
# 创建唯一的共享内存名称
shared_mem_name = f"test_basic_shm_{int(time.time())}"
# 创建指定大小的共享内存块
shm_size = 1024 * 1024 # 1MB
try:
# 创建共享内存块
shm = shared_memory.SharedMemory(
create=True,
size=shm_size,
name=shared_mem_name
)
print(f"创建共享内存块: {shared_mem_name}, 大小: {shm_size}字节")
# 启动写入进程
writer_process = multiprocessing.Process(
target=shared_memory_writer,
args=(shared_mem_name, 100) # 生成100个数据点
)
writer_process.start()
# 稍等后启动读取进程
time.sleep(1)
reader_process = multiprocessing.Process(
target=shared_memory_reader,
args=(shared_mem_name, 5, 1.0) # 读取5次，每次间隔1秒
)
reader_process.start()
# 等待所有进程完成
writer_process.join()
reader_process.join()
finally:
# 清理共享内存
try:
if 'shm' in locals():
shm.close()
shm.unlink()
else:
# 尝试连接并清理
cleanup_shm = shared_memory.SharedMemory(name=shared_mem_name)
cleanup_shm.close()
cleanup_shm.unlink()
print(f"清理共享内存块: {shared_mem_name}")
except Exception as e:
print(f"清理共享内存时出错: {e}")
if __name__ == '__main__':
test_basic_shared_memory()
---
b.同步机制集成
a与锁和信号量结合
共享内存通常需要与同步机制结合使用，以确保数据访问的原子性和一致性。
b.代码示例
---
# 共享内存与同步机制结合示例
import multiprocessing
import time
import numpy as np
from multiprocessing import shared_memory, Value, Array
class SynchronizedSharedMemory:
"""带同步机制的共享内存管理器"""
def __init__(self, name="sync_shm", size=1024*1024):
self.name = name
self.size = size
self.shm = None
# 同步变量
self.lock = multiprocessing.Lock()
self.data_ready = multiprocessing.Value('b', False)
self.reader_count = multiprocessing.Value('i', 0)
self.writer_count = multiprocessing.Value('i', 0)
self.total_writes = multiprocessing.Value('i', 0)
self.total_reads = multiprocessing.Value('i', 0)
def create(self):
"""创建共享内存"""
self.shm = shared_memory.SharedMemory(
create=True,
size=self.size,
name=self.name
)
print(f"创建同步共享内存: {self.name}, 大小: {self.size}字节")
return self.shm
def write_data(self, data):
"""安全写入数据"""
with self.lock:
if not self.shm:
raise RuntimeError("共享内存未初始化")
# 等待所有读取者完成
while self.reader_count.value > 0:
time.sleep(0.01)
# 将数据写入共享内存
data_bytes = data.tobytes()
required_size = len(data_bytes)
if required_size > self.size:
raise RuntimeError(f"数据太大: {required_size}字节 > {self.size}字节")
# 写入数据大小和数据
size_bytes = len(data).to_bytes(8, byteorder='little')
self.shm.buf[:8] = size_bytes
self.shm.buf[8:8+len(data_bytes)] = data_bytes
self.total_writes.value += 1
self.writer_count.value += 1
self.data_ready.value = True
return True
def read_data(self, timeout=None):
"""安全读取数据"""
start_time = time.time()
with self.lock:
# 等待数据准备好
while not self.data_ready.value:
if timeout and (time.time() - start_time) > timeout:
return None
time.sleep(0.01)
if not self.shm:
raise RuntimeError("共享内存未初始化")
# 读取数据大小
size_bytes = self.shm.buf[:8]
data_size = int.from_bytes(size_bytes, byteorder='little')
if data_size <= 0 or data_size * 8 > (self.size - 8):
return None
# 读取实际数据
data_bytes = self.shm.buf[8:8+data_size*8] # 假设double类型
data = np.frombuffer(data_bytes, dtype=np.float64, count=data_size)
self.reader_count.value += 1
self.total_reads.value += 1
# 如果没有写入者等待，清除data_ready标志
if self.writer_count.value == 0:
self.data_ready.value = False
return data.copy()
def writer_done(self):
"""写入者完成"""
with self.lock:
self.writer_count.value -= 1
def reader_done(self):
"""读取者完成"""
with self.lock:
self.reader_count.value -= 1
if self.reader_count.value == 0 and self.writer_count.value == 0:
self.data_ready.value = False
def get_stats(self):
"""获取统计信息"""
with self.lock:
return {
'total_writes': self.total_writes.value,
'total_reads': self.total_reads.value,
'data_ready': self.data_ready.value,
'readers': self.reader_count.value,
'writers': self.writer_count.value
}
def close(self):
"""关闭共享内存"""
if self.shm:
self.shm.close()
print(f"关闭共享内存: {self.name}")
self.shm = None
def unlink(self):
"""删除共享内存"""
if self.shm:
self.shm.unlink()
print(f"删除共享内存: {self.name}")
def data_generator_worker(shared_mem_manager, worker_id, batch_size):
"""数据生成工作者进程"""
print(f"数据生成器{worker_id}启动，PID: {multiprocessing.current_process().pid}")
for batch in range(batch_size):
try:
# 生成随机数据
data_size = 100 + batch * 10 # 逐渐增加数据量
data = np.random.randn(data_size) * 100 # 正态分布数据
# 添加一些特征值，便于验证
data[0] = worker_id # 第一个数据点标识工作者
data[1] = batch # 第二个数据点标识批次
print(f"生成器{worker_id}生成第{batch+1}批数据: {len(data)}个点, "
f"范围: {data.min():.2f} - {data.max():.2f}")
# 写入共享内存
shared_mem_manager.write_data(data)
# 等待一段时间再生成下一批
time.sleep(0.5)
except Exception as e:
print(f"生成器{worker_id}异常: {e}")
shared_mem_manager.writer_done()
print(f"数据生成器{worker_id}完成")
def data_processor_worker(shared_mem_manager, processor_id, read_count):
"""数据处理器进程"""
print(f"数据处理器{processor_id}启动，PID: {multiprocessing.current_process().pid}")
processed_count = 0
while processed_count < read_count:
try:
# 读取共享内存数据
data = shared_mem_manager.read_data(timeout=2.0)
if data is not None:
processed_count += 1
worker_id = int(data[0]) # 提取工作者ID
batch_id = int(data[1]) # 提取批次ID
# 处理数据（计算统计信息）
result = {
'processor_id': processor_id,
'worker_id': worker_id,
'batch_id': batch_id,
'data_size': len(data),
'mean': np.mean(data[2:]), # 排除前两个标识点
'std': np.std(data[2:]),
'min': np.min(data[2:]),
'max': np.max(data[2:]),
'sum': np.sum(data[2:]),
'timestamp': time.time()
}
print(f"处理器{processor_id}处理第{processed_count}批数据: "
f"来自工作者{worker_id}的批次{batch_id}, "
f"均值: {result['mean']:.2f}, 标准差: {result['std']:.2f}")
else:
print(f"处理器{processor_id}读取超时，继续等待...")
except Exception as e:
print(f"处理器{processor_id}异常: {e}")
shared_mem_manager.reader_done()
print(f"数据处理器{processor_id}完成，共处理{processed_count}批数据")
def test_synchronized_shared_memory():
"""测试带同步机制的共享内存"""
print("=== 同步共享内存测试 ===")
shared_mem_name = f"sync_test_shm_{int(time.time())}"
shared_manager = None
try:
# 创建同步共享内存管理器
shared_manager = SynchronizedSharedMemory(shared_mem_name, 2*1024*1024) # 2MB
shared_manager.create()
print(f"共享内存状态: {shared_manager.get_stats()}")
# 启动多个数据生成器
num_generators = 2
num_processors = 3
batches_per_generator = 5
reads_per_processor = 8
generators = []
processors = []
# 启动数据生成器
for i in range(num_generators):
generator = multiprocessing.Process(
target=data_generator_worker,
args=(shared_manager, i+1, batches_per_generator)
)
generators.append(generator)
generator.start()
# 稍等后启动数据处理器
time.sleep(0.5)
for i in range(num_processors):
processor = multiprocessing.Process(
target=data_processor_worker,
args=(shared_manager, i+1, reads_per_processor)
)
processors.append(processor)
processor.start()
# 监控共享内存状态
monitor_time = 0
max_monitor_time = 20
while monitor_time < max_monitor_time:
time.sleep(2)
monitor_time += 2
stats = shared_manager.get_stats()
print(f"监控报告({monitor_time}s): {stats}")
# 检查是否所有进程都已完成
all_generators_done = not any(g.is_alive() for g in generators)
all_processors_done = not any(p.is_alive() for p in processors)
if all_generators_done and all_processors_done:
print("所有工作进程已完成")
break
# 等待所有进程完成
for generator in generators:
generator.join(timeout=5)
if generator.is_alive():
generator.terminate()
for processor in processors:
processor.join(timeout=5)
if processor.is_alive():
processor.terminate()
# 最终统计
final_stats = shared_manager.get_stats()
print(f"\n最终统计: {final_stats}")
finally:
# 清理资源
if shared_manager:
shared_manager.close()
shared_manager.unlink()
if __name__ == '__main__':
test_synchronized_shared_memory()
---
03.高级共享内存应用
a.大型数据集共享
a科学计算场景
在机器学习和科学计算中，经常需要共享大型数据集（如训练数据、模型参数等），使用共享内存可以显著提高数据访问效率，避免数据在进程间的重复传输。
b.代码示例
---
# 大型数据集共享示例
import multiprocessing
import time
import numpy as np
from multiprocessing import shared_memory
import pickle
import struct
class LargeDatasetManager:
"""大型数据集共享管理器"""
def __init__(self, name="large_dataset", max_size=10*1024*1024): # 10MB
self.name = name
self.max_size = max_size
self.header_format = 'QQQI' # magic(8) + dataset_count(8) + dataset_size(8) + flags(4)
self.header_size = struct.calcsize(self.header_format)
self.shm = None
def create_dataset(self):
"""创建数据集共享内存"""
try:
self.shm = shared_memory.SharedMemory(
create=True,
size=self.max_size,
name=self.name
)
# 初始化头部
header = struct.pack(
self.header_format,
0x4C4F414441544153, # "LARGEDATAS" 的ASCII码
0, # dataset_count
0, # dataset_size
0 # flags
)
self.shm.buf[:self.header_size] = header
print(f"创建大型数据集共享内存: {self.name}, 大小: {self.max_size}字节")
return True
except FileExistsError:
self.shm = shared_memory.SharedMemory(name=self.name)
print(f"连接到现有数据集共享内存: {self.name}")
return False
def add_dataset(self, dataset_id, data, metadata=None):
"""添加数据集"""
if not self.shm:
raise RuntimeError("共享内存未初始化")
# 序列化数据
data_bytes = pickle.dumps(data)
metadata_bytes = pickle.dumps(metadata or {})
# 计算总大小
total_size = len(data_bytes) + len(metadata_bytes) + 16 # 16字节用于长度信息
# 读取当前头部信息
header_data = self.shm.buf[:self.header_size]
magic, dataset_count, used_size, flags = struct.unpack(self.header_format, header_data)
# 检查空间
data_offset = self.header_size + used_size
if data_offset + total_size > self.max_size:
raise RuntimeError(f"共享内存空间不足: 需要{total_size}字节")
# 写入数据集
offset = data_offset
# 写入数据长度 (8字节)
self.shm.buf[offset:offset+8] = len(data_bytes).to_bytes(8, byteorder='little')
offset += 8
# 写入元数据长度 (8字节)
self.shm.buf[offset:offset+8] = len(metadata_bytes).to_bytes(8, byteorder='little')
offset += 8
# 写入数据
self.shm.buf[offset:offset+len(data_bytes)] = data_bytes
offset += len(data_bytes)
# 写入元数据
self.shm.buf[offset:offset+len(metadata_bytes)] = metadata_bytes
# 更新头部信息
new_used_size = used_size + total_size
new_header = struct.pack(
self.header_format,
magic,
dataset_count + 1,
new_used_size,
flags
)
self.shm.buf[:self.header_size] = new_header
print(f"添加数据集{dataset_id}: 数据大小{len(data_bytes)}字节, "
f"元数据大小{len(metadata_bytes)}字节")
return True
def get_dataset(self, dataset_id):
"""获取数据集"""
if not self.shm:
raise RuntimeError("共享内存未初始化")
# 读取头部信息
header_data = self.shm.buf[:self.header_size]
magic, dataset_count, used_size, flags = struct.unpack(self.header_format, header_data)
if magic != 0x4C4F414441544153:
raise RuntimeError("无效的数据集共享内存")
# 搜索数据集
offset = self.header_size
target_dataset_id = dataset_id
while offset < self.header_size + used_size:
# 读取数据长度和元数据长度
data_len = int.from_bytes(self.shm.buf[offset:offset+8], byteorder='little')
metadata_len = int.from_bytes(self.shm.buf[offset+8:offset+16], byteorder='little')
offset += 16
# 读取元数据
metadata_bytes = self.shm.buf[offset:offset+metadata_len]
metadata = pickle.loads(metadata_bytes)
offset += metadata_len
# 检查数据集ID
if metadata.get('dataset_id') == target_dataset_id:
# 读取数据
data_bytes = self.shm.buf[offset:offset+data_len]
data = pickle.loads(data_bytes)
print(f"获取数据集{dataset_id}: 大小{len(data_bytes)}字节")
return data, metadata
offset += data_len
raise ValueError(f"数据集{dataset_id}不存在")
def list_datasets(self):
"""列出所有数据集"""
if not self.shm:
raise RuntimeError("共享内存未初始化")
# 读取头部信息
header_data = self.shm.buf[:self.header_size]
magic, dataset_count, used_size, flags = struct.unpack(self.header_format, header_data)
datasets = []
offset = self.header_size
while offset < self.header_size + used_size:
# 读取长度信息
data_len = int.from_bytes(self.shm.buf[offset:offset+8], byteorder='little')
metadata_len = int.from_bytes(self.shm.buf[offset+8:offset+16], byteorder='little')
offset += 16
# 读取元数据
metadata_bytes = self.shm.buf[offset:offset+metadata_len]
metadata = pickle.loads(metadata_bytes)
offset += metadata_len
datasets.append({
'dataset_id': metadata.get('dataset_id', 'unknown'),
'created_time': metadata.get('created_time', 'unknown'),
'data_size': data_len,
'metadata_size': metadata_len,
'description': metadata.get('description', '')
})
offset += data_len
return datasets
def close(self):
"""关闭共享内存"""
if self.shm:
self.shm.close()
self.shm = None
def unlink(self):
"""删除共享内存"""
if self.shm:
self.shm.unlink()
def generate_training_data(data_size, features, samples):
"""生成训练数据"""
print(f"生成训练数据: {features}特征 x {samples}样本")
X = np.random.randn(samples, features).astype(np.float32)
y = (X[:, 0] + X[:, 1] * 0.5 + np.random.randn(samples) * 0.1).astype(np.float32)
# 添加一些特征相关性
for i in range(2, features):
X[:, i] = X[:, 0] * (0.1 * i) + X[:, 1] * (0.05 * i) + np.random.randn(samples) * 0.1
return {
'X': X,
'y': y,
'features': features,
'samples': samples
}
def machine_learning_trainer(dataset_manager, trainer_id, epochs):
"""机器学习训练进程"""
print(f"训练器{trainer_id}启动，PID: {multiprocessing.current_process().pid}")
for epoch in range(epochs):
try:
# 获取训练数据
dataset, metadata = dataset_manager.get_dataset('training_data')
X, y = dataset['X'], dataset['y']
samples, features = X.shape
print(f"训练器{trainer_id}第{epoch+1}轮: 使用{samples}样本x{features}特征")
# 模拟训练过程
start_time = time.time()
# 简单的线性回归模拟
weights = np.random.randn(features) * 0.01
learning_rate = 0.01
for iteration in range(10): # 简化训练迭代
# 前向传播
predictions = np.dot(X, weights)
# 计算损失
loss = np.mean((predictions - y) ** 2)
# 计算梯度
gradients = 2 * np.dot(X.T, (predictions - y)) / samples
# 更新权重
weights -= learning_rate * gradients
if iteration % 3 == 0:
print(f" 迭代{iteration+1}: 损失={loss:.6f}")
training_time = time.time() - start_time
# 计算最终性能
final_predictions = np.dot(X, weights)
final_loss = np.mean((final_predictions - y) ** 2)
r2_score = 1 - np.var(final_predictions - y) / np.var(y)
print(f"训练器{trainer_id}第{epoch+1}轮完成: "
f"最终损失={final_loss:.6f}, R²分数={r2_score:.4f}, "
f"耗时{training_time:.2f}秒")
# 保存训练结果
training_result = {
'epoch': epoch + 1,
'weights': weights,
'final_loss': final_loss,
'r2_score': r2_score,
'training_time': training_time,
'trainer_id': trainer_id
}
result_metadata = {
'dataset_id': f'training_result_{trainer_id}_{epoch+1}',
'created_time': time.time(),
'description': f'训练器{trainer_id}第{epoch+1}轮结果',
'trainer_id': trainer_id,
'epoch': epoch + 1
}
dataset_manager.add_dataset(
f'training_result_{trainer_id}_{epoch+1}',
training_result,
result_metadata
)
time.sleep(1) # 模拟训练间隔
except Exception as e:
print(f"训练器{trainer_id}异常: {e}")
print(f"训练器{trainer_id}完成")
def model_evaluator(dataset_manager, evaluator_id):
"""模型评估进程"""
print(f"评估器{evaluator_id}启动，PID: {multiprocessing.current_process().pid}")
# 等待训练结果
time.sleep(3)
try:
# 列出所有数据集
datasets = dataset_manager.list_datasets()
print(f"评估器{evaluator_id}发现{len(datasets)}个数据集")
# 查找训练结果
training_results = []
for dataset in datasets:
if 'training_result' in dataset['dataset_id']:
try:
result, metadata = dataset_manager.get_dataset(dataset['dataset_id'])
training_results.append((result, metadata))
print(f"评估器{evaluator_id}加载结果: {metadata['description']}")
except Exception as e:
print(f"评估器{evaluator_id}加载{dataset['dataset_id']}失败: {e}")
if training_results:
print(f"评估器{evaluator_id}开始分析{len(training_results)}个训练结果")
# 分析训练结果
all_losses = [r[0]['final_loss'] for r in training_results]
all_r2_scores = [r[0]['r2_score'] for r in training_results]
all_times = [r[0]['training_time'] for r in training_results]
print(f"\n评估器{evaluator_id}分析结果:")
print(f" 损失统计: 平均={np.mean(all_losses):.6f}, 最小={np.min(all_losses):.6f}, 最大={np.max(all_losses):.6f}")
print(f" R²分数统计: 平均={np.mean(all_r2_scores):.4f}, 最小={np.min(all_r2_scores):.4f}, 最大={np.max(all_r2_scores):.4f}")
print(f" 训练时间统计: 平均={np.mean(all_times):.2f}s, 总计={np.sum(all_times):.2f}s")
# 找出最佳模型
best_idx = np.argmax(all_r2_scores)
best_result = training_results[best_idx]
print(f"最佳模型: {best_result[1]['description']}, R²分数={best_result[0]['r2_score']:.4f}")
else:
print(f"评估器{evaluator_id}: 未找到训练结果")
except Exception as e:
print(f"评估器{evaluator_id}异常: {e}")
print(f"评估器{evaluator_id}完成")
def test_large_dataset_sharing():
"""测试大型数据集共享"""
print("=== 大型数据集共享测试 ===")
dataset_name = f"ml_dataset_{int(time.time())}"
dataset_manager = LargeDatasetManager(dataset_name, 20*1024*1024) # 20MB
try:
# 创建数据集共享内存
dataset_manager.create_dataset()
# 生成并添加训练数据
print("生成训练数据...")
training_data = generate_training_data(data_size=10000, features=20, samples=5000)
training_metadata = {
'dataset_id': 'training_data',
'created_time': time.time(),
'description': '机器学习训练数据集',
'features': training_data['features'],
'samples': training_data['samples']
}
dataset_manager.add_dataset('training_data', training_data, training_metadata)
# 列出数据集
datasets = dataset_manager.list_datasets()
print(f"当前数据集: {len(datasets)}个")
for dataset in datasets:
print(f" {dataset['dataset_id']}: {dataset['description']} "
f"({dataset['data_size']}字节)")
# 启动多个训练器和评估器
num_trainers = 2
num_evaluators = 1
epochs_per_trainer = 3
trainers = []
evaluators = []
# 启动训练器
for i in range(num_trainers):
trainer = multiprocessing.Process(
target=machine_learning_trainer,
args=(dataset_manager, i+1, epochs_per_trainer)
)
trainers.append(trainer)
trainer.start()
# 启动评估器
for i in range(num_evaluators):
evaluator = multiprocessing.Process(
target=model_evaluator,
args=(dataset_manager, i+1)
)
evaluators.append(evaluator)
time.sleep(1) # 错开启动时间
evaluator.start()
# 等待所有进程完成
for trainer in trainers:
trainer.join()
for evaluator in evaluators:
evaluator.join()
# 最终数据集统计
final_datasets = dataset_manager.list_datasets()
print(f"\n最终数据集统计: {len(final_datasets)}个数据集")
training_results_count = sum(1 for d in final_datasets if 'training_result' in d['dataset_id'])
print(f"训练结果数据集: {training_results_count}个")
total_size = sum(d['data_size'] for d in final_datasets)
print(f"总数据大小: {total_size}字节 ({total_size/1024/1024:.2f}MB)")
finally:
# 清理资源
dataset_manager.close()
dataset_manager.unlink()
print(f"清理数据集共享内存: {dataset_name}")
if __name__ == '__main__':
test_large_dataset_sharing()
---
04.性能优化与最佳实践
a.共享内存性能优化
a内存对齐与访问优化
通过适当的内存对齐和数据布局，可以提高共享内存的访问效率。使用numpy数组时，选择合适的数据类型和内存连续性可以显著提升性能。
b.代码示例
---
# 共享内存性能优化示例
import multiprocessing
import time
import numpy as np
from multiprocessing import shared_memory
import psutil
import gc
class PerformanceOptimizedSharedMemory:
"""性能优化的共享内存管理器"""
def __init__(self, name="perf_shm", size=100*1024*1024): # 100MB
self.name = name
self.size = size
self.shm = None
self.buffer_info = {}
def create_optimized(self):
"""创建优化的共享内存布局"""
self.shm = shared_memory.SharedMemory(
create=True,
size=self.size,
name=self.name
)
# 定义优化的内存布局
layout = {
'header': {'offset': 0, 'size': 1024}, # 1KB头部
'control': {'offset': 1024, 'size': 1024}, # 1KB控制信息
'data_pool': {'offset': 2048, 'size': self.size - 2048} # 数据池
}
# 存储布局信息
layout_bytes = pickle.dumps(layout)
self.shm.buf[0:len(layout_bytes)] = layout_bytes
self.buffer_info = layout
print(f"创建优化共享内存: {self.name}, 布局: {layout}")
return self.shm
def allocate_block(self, block_type, block_size):
"""分配内存块"""
if block_type not in self.buffer_info:
raise ValueError(f"未知块类型: {block_type}")
layout = self.buffer_info[block_type]
offset = layout['offset']
available_size = layout['size']
if block_size > available_size:
raise ValueError(f"块太大: {block_size} > {available_size}")
return offset, block_size
def write_numpy_array(self, array, block_type='data_pool'):
"""高效写入numpy数组"""
if not array.flags.c_contiguous:
# 确保数组是C连续的，提高内存访问效率
array = np.ascontiguousarray(array)
offset, block_size = self.allocate_block(block_type, array.nbytes)
# 使用内存视图直接复制，避免额外分配
memory_view = memoryview(self.shm.buf)
array_view = memoryview(offset:offset+array.nbytes)
# 高效复制数据
array_view[:] = array
return {
'offset': offset,
'size': array.nbytes,
'shape': array.shape,
'dtype': str(array.dtype)
}
def read_numpy_array(self, block_info, shape, dtype):
"""高效读取numpy数组"""
offset = block_info['offset']
size = block_info['size']
# 使用内存视图创建numpy数组视图
memory_view = memoryview(self.shm.buf)
array_view = memory_view[offset:offset+size]
# 直接创建numpy数组视图，避免数据复制
array = np.frombuffer(array_view, dtype=dtype).reshape(shape)
return array
def benchmark_operations(self, operations=1000):
"""性能基准测试"""
print(f"开始性能基准测试: {operations}次操作")
# 测试数据大小
test_sizes = [100, 1000, 10000, 100000]
results = {}
for size in test_sizes:
print(f"\n测试数据大小: {size}")
# 生成测试数据
test_array = np.random.randn(size).astype(np.float64)
# 测试写入性能
start_time = time.time()
for i in range(operations):
block_info = self.write_numpy_array(test_array)
write_time = time.time() - start_time
# 测试读取性能
start_time = time.time()
for i in range(operations):
read_array = self.read_numpy_array(
block_info,
test_array.shape,
test_array.dtype
)
read_time = time.time() - start_time
# 计算性能指标
write_mbps = (test_array.nbytes * operations / write_time) / (1024*1024)
read_mbps = (test_array.nbytes * operations / read_time) / (1024*1024)
write_ops_per_sec = operations / write_time
read_ops_per_sec = operations / read_time
results[size] = {
'write_time': write_time,
'read_time': read_time,
'write_mbps': write_mbps,
'read_mbps': read_mbps,
'write_ops_per_sec': write_ops_per_sec,
'read_ops_per_sec': read_ops_per_sec,
'data_size_mb': test_array.nbytes / (1024*1024)
}
print(f" 写入: {write_time:.3f}s, {write_mbps:.1f}MB/s, {write_ops_per_sec:.0f} ops/s")
print(f" 读取: {read_time:.3f}s, {read_mbps:.1f}MB/s, {read_ops_per_sec:.0f} ops/s")
return results
def close(self):
"""关闭共享内存"""
if self.shm:
self.shm.close()
self.shm = None
def unlink(self):
"""删除共享内存"""
if self.shm:
self.shm.unlink()
def performance_test_worker(shared_mem_name, worker_id, test_iterations):
"""性能测试工作进程"""
print(f"性能测试工作者{worker_id}启动，PID: {multiprocessing.current_process().pid}")
try:
# 连接到共享内存
shm = shared_memory.SharedMemory(name=shared_mem_name)
print(f"工作者{worker_id}连接到共享内存: {len(shm.buf)}字节")
# 性能测试
test_size = 50000
test_array = np.random.randn(test_size).astype(np.float64)
write_times = []
read_times = []
for i in range(test_iterations):
# 测试写入
start_time = time.perf_counter()
offset = (i * test_array.nbytes) % (len(shm.buf) - test_array.nbytes)
shm.buf[offset:offset+test_array.nbytes] = test_array.tobytes()
write_time = time.perf_counter() - start_time
write_times.append(write_time)
# 测试读取
start_time = time.perf_counter()
read_data = np.frombuffer(shm.buf[offset:offset+test_array.nbytes], dtype=np.float64)
read_time = time.perf_counter() - start_time
read_times.append(read_time)
if i % 100 == 0:
print(f"工作者{worker_id}完成第{i+1}/{test_iterations}次操作")
# 计算统计信息
avg_write_time = np.mean(write_times)
avg_read_time = np.mean(read_times)
max_write_time = np.max(write_times)
max_read_time = np.max(read_times)
data_mb = test_array.nbytes / (1024*1024)
write_mbps = data_mb / avg_write_time
read_mbps = data_mb / avg_read_time
print(f"\n工作者{worker_id}性能结果:")
print(f" 数据大小: {data_mb:.3f}MB")
print(f" 写入: 平均{avg_write_time*1000:.3f}ms, 最大{max_write_time*1000:.3f}ms, {write_mbps:.1f}MB/s")
print(f" 读取: 平均{avg_read_time*1000:.3f}ms, 最大{max_read_time*1000:.3f}ms, {read_mbps:.1f}MB/s")
shm.close()
except Exception as e:
print(f"工作者{worker_id}异常: {e}")
print(f"性能测试工作者{worker_id}完成")
def system_monitor():
"""系统资源监控"""
print("\n=== 系统资源监控 ===")
try:
# 获取CPU和内存信息
cpu_percent = psutil.cpu_percent(interval=1)
memory = psutil.virtual_memory()
process = psutil.Process()
print(f"系统CPU使用率: {cpu_percent:.1f}%")
print(f"系统内存使用: {memory.percent:.1f}% ({memory.used/1024/1024/1024:.1f}GB/{memory.total/1024/1024/1024:.1f}GB)")
print(f"当前进程PID: {process.pid}")
print(f"进程内存使用: {process.memory_info().rss/1024/1024:.1f}MB")
# 获取进程数量
process_count = len(psutil.pids())
print(f"系统进程数: {process_count}")
except Exception as e:
print(f"系统监控异常: {e}")
def test_shared_memory_performance():
"""测试共享内存性能"""
print("=== 共享内存性能测试 ===")
shared_mem_name = f"perf_test_shm_{int(time.time())}"
shm_manager = None
try:
# 系统监控
system_monitor()
# 创建优化的共享内存
shm_manager = PerformanceOptimizedSharedMemory(shared_mem_name, 50*1024*1024) # 50MB
shm_manager.create_optimized()
# 单进程基准测试
print("\n=== 单进程基准测试 ===")
benchmark_results = shm_manager.benchmark_operations(operations=500)
# 多进程并发测试
print("\n=== 多进程并发测试 ===")
num_workers = 4
iterations_per_worker = 200
workers = []
start_time = time.time()
for i in range(num_workers):
worker = multiprocessing.Process(
target=performance_test_worker,
args=(shared_mem_name, i+1, iterations_per_worker)
)
workers.append(worker)
worker.start()
for worker in workers:
worker.join()
total_time = time.time() - start_time
total_operations = num_workers * iterations_per_worker
print(f"\n多进程测试完成:")
print(f" 总时间: {total_time:.2f}秒")
print(f" 总操作数: {total_operations}")
print(f" 平均操作速率: {total_operations/total_time:.0f} ops/s")
# 系统资源使用情况
system_monitor()
# 性能分析
print("\n=== 性能分析 ===")
for size, result in benchmark_results.items():
print(f"数据大小{size}:")
print(f" 写入带宽: {result['write_mbps']:.1f}MB/s")
print(f" 读取带宽: {result['read_mbps']:.1f}MB/s")
print(f" 写入延迟: {1000/result['write_ops_per_sec']:.3f}ms")
print(f" 读取延迟: {1000/result['read_ops_per_sec']:.3f}ms")
# 内存使用分析
memory = psutil.virtual_memory()
print(f"\n内存使用分析:")
print(f" 当前使用率: {memory.percent:.1f}%")
print(f" 可用内存: {memory.available/1024/1024/1024:.1f}GB")
finally:
# 清理资源
if shm_manager:
shm_manager.close()
shm_manager.unlink()
# 强制垃圾回收
gc.collect()
print("共享内存性能测试完成")
if __name__ == '__main__':
test_shared_memory_performance()
---
7 协程与异步
7.1 asyncio基础
01.asyncio模块概述
a.异步编程概念
asyncio是Python的异步I/O框架，通过事件循环机制实现单线程并发，特别适合I/O密集型任务。asyncio使用协程（coroutine）来支持并发执行，避免了多线程的锁竞争和上下文切换开销。
b.核心组件
asyncio提供了事件循环（event loop）、协程（coroutine）、任务（task）、Future等核心组件，通过这些组件可以构建高性能的异步应用程序。
02.事件循环基础
a.事件循环工作原理
事件循环是asyncio的核心，负责调度和执行异步任务。事件循环在单线程中运行，通过事件驱动的方式处理多个I/O操作。
b.代码示例
---
# asyncio事件循环基础示例
import asyncio
import time
from datetime import datetime
async def simple_coroutine(name, delay):
"""简单的协程函数"""
print(f"{datetime.now().strftime('%H:%M:%S')} - {name}: 开始执行，延迟{delay}秒")
await asyncio.sleep(delay) # 模拟异步I/O操作
print(f"{datetime.now().strftime('%H:%M:%S')} - {name}: 执行完成")
return f"{name}的结果"
async def main():
"""主协程"""
print(f"{datetime.now().strftime('%H:%M:%S')} - 主协程开始")
# 创建多个协程任务
tasks = [
asyncio.create_task(simple_coroutine("任务A", 2)),
asyncio.create_task(simple_coroutine("任务B", 1)),
asyncio.create_task(simple_coroutine("任务C", 3)),
]
print(f"{datetime.now().strftime('%H:%M:%S')} - 所有任务已创建，等待完成")
# 等待所有任务完成
results = await asyncio.gather(*tasks)
print(f"{datetime.now().strftime('%H:%M:%S')} - 所有任务完成")
print(f"任务结果: {results}")
def test_basic_event_loop():
"""测试基础事件循环"""
print("=== asyncio基础事件循环测试 ===")
start_time = time.time()
# 运行异步主函数
asyncio.run(main())
end_time = time.time()
print(f"总执行时间: {end_time - start_time:.2f}秒")
print("注意：虽然是单线程，但任务并发执行，总时间接近最长任务的3秒")
# 传统同步方式对比
def synchronous_version():
"""同步版本对比"""
print("\n=== 同步版本对比 ===")
start_time = time.time()
def sync_task(name, delay):
print(f"{datetime.now().strftime('%H:%M:%S')} - {name}: 开始执行，延迟{delay}秒")
time.sleep(delay) # 同步阻塞
print(f"{datetime.now().strftime('%H:%M:%S')} - {name}: 执行完成")
return f"{name}的结果"
# 顺序执行
results = [
sync_task("任务A", 2),
sync_task("任务B", 1),
sync_task("任务C", 3)
]
end_time = time.time()
print(f"任务结果: {results}")
print(f"总执行时间: {end_time - start_time:.2f}秒")
print("注意：同步顺序执行，总时间是所有延迟之和6秒")
if __name__ == '__main__':
test_basic_event_loop()
synchronous_version()
---
03.协程创建与管理
a.协程定义与调用
异步函数使用async def定义，使用await关键字等待其他协程完成。协程本身不会立即执行，需要通过事件循环来调度运行。
b.代码示例
---
# 协程创建与管理示例
import asyncio
import time
import random
from typing import List, Tuple
class AsyncTaskManager:
"""异步任务管理器"""
def __init__(self):
self.tasks = []
self.results = {}
async def create_task(self, coro, task_id):
"""创建并跟踪任务"""
task = asyncio.create_task(coro)
task.set_name(task_id)
self.tasks.append(task)
return task
async def wait_for_all(self):
"""等待所有任务完成"""
if not self.tasks:
return []
print(f"等待{len(self.tasks)}个任务完成...")
results = await asyncio.gather(*self.tasks, return_exceptions=True)
for i, (task, result) in enumerate(zip(self.tasks, results)):
task_id = task.get_name()
if isinstance(result, Exception):
print(f"任务{task_id}失败: {result}")
self.results[task_id] = None
else:
print(f"任务{task_id}完成: {result}")
self.results[task_id] = result
return results
async def wait_with_timeout(self, timeout):
"""带超时的等待"""
try:
return await asyncio.wait_for(
asyncio.gather(*self.tasks),
timeout=timeout
)
except asyncio.TimeoutError:
print(f"任务执行超时({timeout}秒)")
# 取消未完成的任务
for task in self.tasks:
if not task.done():
task.cancel()
return None
finally:
# 等待取消操作完成
await asyncio.gather(*self.tasks, return_exceptions=True)
async def data_processing_task(task_id, data_size, processing_time):
"""数据处理任务"""
print(f"任务{task_id}开始：处理{data_size}个数据项")
# 模拟数据加载
await asyncio.sleep(0.5)
print(f"任务{task_id}：数据加载完成")
# 模拟数据处理
for i in range(processing_time):
await asyncio.sleep(0.1)
progress = (i + 1) / processing_time * 100
print(f"任务{task_id}进度：{progress:.1f}%")
# 生成随机结果
result = sum(random.randint(1, 100) for _ in range(data_size))
print(f"任务{task_id}完成：结果={result}")
return result
async def io_simulation_task(task_id, io_operations, io_delay):
"""I/O操作模拟任务"""
print(f"IO任务{task_id}开始：{io_operations}次IO操作")
total_bytes = 0
for i in range(io_operations):
# 模拟网络请求或文件读取
await asyncio.sleep(io_delay)
bytes_read = random.randint(100, 1000)
total_bytes += bytes_read
if (i + 1) % 3 == 0:
print(f"IO任务{task_id}：已完成{i+1}次IO操作，共读取{total_bytes}字节")
print(f"IO任务{task_id}完成：总读取{total_bytes}字节")
return total_bytes
async def monitor_tasks(manager, monitor_interval=1.0):
"""任务监控协程"""
print("任务监控器启动")
start_time = time.time()
while manager.tasks:
await asyncio.sleep(monitor_interval)
current_time = time.time()
elapsed = current_time - start_time
# 统计任务状态
completed = sum(1 for task in manager.tasks if task.done())
pending = len(manager.tasks) - completed
print(f"监控报告({elapsed:.1f}s): 完成{completed}个，待处理{pending}个")
# 如果所有任务完成，退出监控
if pending == 0:
break
print("任务监控器结束")
async def test_coroutine_management():
"""测试协程管理"""
print("=== 协程创建与管理测试 ===")
manager = AsyncTaskManager()
# 创建数据处理任务
data_tasks = []
for i in range(3):
task = await manager.create_task(
data_processing_task(f"DATA_{i+1}",
random.randint(50, 200),
random.randint(3, 8)),
f"DATA_{i+1}"
)
data_tasks.append(task)
# 创建IO模拟任务
io_tasks = []
for i in range(2):
task = await manager.create_task(
io_simulation_task(f"IO_{i+1}",
random.randint(5, 10),
random.uniform(0.2, 0.5)),
f"IO_{i+1}"
)
io_tasks.append(task)
# 启动监控任务
monitor_task = asyncio.create_task(monitor_tasks(manager, 0.5))
# 等待所有任务完成
start_time = time.time()
results = await manager.wait_for_all()
end_time = time.time()
# 停止监控
monitor_task.cancel()
try:
await monitor_task
except asyncio.CancelledError:
pass
print(f"\n=== 执行结果 ===")
print(f"总任务数: {len(manager.tasks)}")
print(f"总执行时间: {end_time - start_time:.2f}秒")
print(f"任务结果: {manager.results}")
# 计算统计信息
successful_results = [r for r in manager.results.values() if r is not None]
if successful_results:
print(f"平均结果: {sum(successful_results) / len(successful_results):.2f}")
print(f"最大结果: {max(successful_results)}")
print(f"最小结果: {min(successful_results)}")
async def test_timeout_handling():
"""测试超时处理"""
print("\n=== 超时处理测试 ===")
manager = AsyncTaskManager()
# 创建一些长时间运行的任务
for i in range(3):
await manager.create_task(
data_processing_task(f"SLOW_{i+1}", 100, 15), # 长时间任务
f"SLOW_{i+1}"
)
# 设置较短的超时时间
timeout = 3.0
print(f"设置超时时间: {timeout}秒")
start_time = time.time()
results = await manager.wait_with_timeout(timeout)
end_time = time.time()
print(f"超时测试完成，实际耗时: {end_time - start_time:.2f}秒")
if __name__ == '__main__':
asyncio.run(test_coroutine_management())
asyncio.run(test_timeout_handling())
---
04.异步I/O操作
a.文件操作异步化
asyncio提供了异步文件操作接口，可以在不阻塞事件循环的情况下进行文件读写操作，提高I/O密集型应用的性能。
b.代码示例
---
# 异步I/O操作示例
import asyncio
import aiofiles
import time
import os
import random
from pathlib import Path
class AsyncFileProcessor:
"""异步文件处理器"""
def __init__(self, base_dir="async_files"):
self.base_dir = Path(base_dir)
self.base_dir.mkdir(exist_ok=True)
self.processed_files = []
async def write_data_file(self, filename, data):
"""异步写入数据文件"""
filepath = self.base_dir / filename
try:
async with aiofiles.open(filepath, 'w', encoding='utf-8') as f:
await f.write(data)
print(f"成功写入文件: {filename} (大小: {len(data)}字节)")
return True
except Exception as e:
print(f"写入文件失败 {filename}: {e}")
return False
async def read_data_file(self, filename):
"""异步读取数据文件"""
filepath = self.base_dir / filename
try:
async with aiofiles.open(filepath, 'r', encoding='utf-8') as f:
content = await f.read()
print(f"成功读取文件: {filename} (大小: {len(content)}字节)")
return content
except Exception as e:
print(f"读取文件失败 {filename}: {e}")
return None
async def process_file_content(self, content):
"""处理文件内容"""
# 模拟内容处理（统计、分析等）
await asyncio.sleep(0.1) # 模拟处理时间
lines = content.split('\n')
word_count = sum(len(line.split()) for line in lines)
char_count = len(content)
return {
'lines': len(lines),
'words': word_count,
'characters': char_count,
'avg_words_per_line': word_count / len(lines) if lines else 0
}
async def generate_test_data(file_processor, num_files):
"""生成测试数据文件"""
print(f"生成{num_files}个测试数据文件...")
tasks = []
for i in range(num_files):
# 生成随机数据
data_lines = []
line_count = random.randint(50, 200)
for j in range(line_count):
words = [f"word_{random.randint(1, 1000)}" for _ in range(random.randint(5, 15))]
line = ' '.join(words)
data_lines.append(line)
data = '\n'.join(data_lines)
filename = f"test_file_{i+1:03d}.txt"
task = file_processor.write_data_file(filename, data)
tasks.append((filename, task))
# 并发写入所有文件
results = await asyncio.gather(*[task for _, task in tasks])
successful_files = []
for (filename, _), success in zip(tasks, results):
if success:
successful_files.append(filename)
print(f"成功生成{len(successful_files)}个文件")
return successful_files
async def process_files_concurrently(file_processor, filenames):
"""并发处理多个文件"""
print(f"开始并发处理{len(filenames)}个文件...")
async def process_single_file(filename):
"""处理单个文件"""
print(f"开始处理文件: {filename}")
# 读取文件
content = await file_processor.read_data_file(filename)
if content is None:
return None
# 处理内容
stats = await file_processor.process_file_content(content)
result = {
'filename': filename,
'stats': stats,
'processed_at': time.time()
}
print(f"文件处理完成: {filename} - {stats['lines']}行, {stats['words']}词")
return result
# 创建并发任务
tasks = [process_single_file(filename) for filename in filenames]
# 限制并发数量，避免同时打开太多文件
semaphore = asyncio.Semaphore(5) # 最多同时处理5个文件
async def limited_process(filename):
async with semaphore:
return await process_single_file(filename)
limited_tasks = [limited_process(filename) for filename in filenames]
# 等待所有任务完成
start_time = time.time()
results = await asyncio.gather(*limited_tasks, return_exceptions=True)
end_time = time.time()
# 统计结果
successful_results = [r for r in results if isinstance(r, dict)]
failed_count = len(results) - len(successful_results)
print(f"\n=== 文件处理结果 ===")
print(f"总文件数: {len(filenames)}")
print(f"成功处理: {len(successful_results)}")
print(f"处理失败: {failed_count}")
print(f"总处理时间: {end_time - start_time:.2f}秒")
# 汇总统计
if successful_results:
total_lines = sum(r['stats']['lines'] for r in successful_results)
total_words = sum(r['stats']['words'] for r in successful_results)
total_chars = sum(r['stats']['characters'] for r in successful_results)
print(f"总计: {total_lines}行, {total_words}词, {total_chars}字符")
print(f"平均每文件: {total_lines/len(successful_results):.1f}行")
return successful_results
async def batch_file_operations(file_processor, filenames, batch_size=3):
"""批量文件操作"""
print(f"批量处理文件，批次大小: {batch_size}")
all_results = []
for i in range(0, len(filenames), batch_size):
batch = filenames[i:i + batch_size]
print(f"\n处理批次 {i//batch_size + 1}: {batch}")
# 处理当前批次
batch_results = await process_files_concurrently(file_processor, batch)
all_results.extend(batch_results)
# 批次间短暂休息
if i + batch_size < len(filenames):
await asyncio.sleep(0.5)
return all_results
async def cleanup_test_files(file_processor):
"""清理测试文件"""
print("\n清理测试文件...")
files = list(file_processor.base_dir.glob("*.txt"))
delete_tasks = []
for filepath in files:
try:
os.remove(filepath)
print(f"删除文件: {filepath.name}")
except Exception as e:
print(f"删除文件失败 {filepath.name}: {e}")
async def test_async_io_operations():
"""测试异步I/O操作"""
print("=== 异步I/O操作测试 ===")
file_processor = AsyncFileProcessor("async_test_files")
try:
# 生成测试数据
num_files = 10
test_files = await generate_test_data(file_processor, num_files)
if not test_files:
print("没有生成测试文件，跳过处理")
return
# 并发处理文件
results = await process_files_concurrently(file_processor, test_files)
# 批量处理演示
batch_results = await batch_file_operations(
file_processor, test_files[:6], batch_size=2
)
print(f"\n=== 测试总结 ===")
print(f"总处理文件数: {len(results)}")
print(f"批量处理文件数: {len(batch_results)}")
finally:
# 清理测试文件
await cleanup_test_files(file_processor)
# 同步版本对比
def sync_file_processing_test():
"""同步文件处理对比"""
print("\n=== 同步文件处理对比 ===")
import time
# 创建测试目录
sync_dir = Path("sync_test_files")
sync_dir.mkdir(exist_ok=True)
try:
# 同步生成测试文件
start_time = time.time()
test_files = []
for i in range(5): # 减少文件数量以便对比
data_lines = []
for j in range(100):
words = [f"word_{random.randint(1, 100)}" for _ in range(10)]
data_lines.append(' '.join(words))
data = '\n'.join(data_lines)
filename = f"sync_file_{i+1}.txt"
filepath = sync_dir / filename
with open(filepath, 'w', encoding='utf-8') as f:
f.write(data)
test_files.append(filename)
gen_time = time.time() - start_time
print(f"同步生成{len(test_files)}个文件耗时: {gen_time:.2f}秒")
# 同步处理文件
start_time = time.time()
results = []
for filename in test_files:
filepath = sync_dir / filename
with open(filepath, 'r', encoding='utf-8') as f:
content = f.read()
# 简单统计
lines = content.split('\n')
word_count = sum(len(line.split()) for line in lines)
results.append({
'filename': filename,
'lines': len(lines),
'words': word_count
})
process_time = time.time() - start_time
print(f"同步处理{len(test_files)}个文件耗时: {process_time:.2f}秒")
print(f"同步总耗时: {gen_time + process_time:.2f}秒")
# 清理同步文件
for filename in test_files:
filepath = sync_dir / filename
os.remove(filepath)
finally:
if sync_dir.exists():
sync_dir.rmdir()
if __name__ == '__main__':
# 安装aiofiles（如果需要）
try:
import aiofiles
except ImportError:
print("请先安装aiofiles: pip install aiofiles")
exit(1)
asyncio.run(test_async_io_operations())
sync_file_processing_test()
---
05.异步网络编程
a.异步HTTP客户端
使用aiohttp等异步HTTP客户端库可以实现高并发的网络请求，避免阻塞事件循环，大幅提升网络密集型应用的性能。
b.代码示例
---
# 异步网络编程示例
import asyncio
import aiohttp
import time
import json
from typing import List, Dict, Any
from urllib.parse import urljoin
class AsyncHttpClient:
"""异步HTTP客户端"""
def __init__(self, base_url=None, timeout=30):
self.base_url = base_url
self.timeout = aiohttp.ClientTimeout(total=timeout)
self.session = None
async def __aenter__(self):
connector = aiohttp.TCPConnector(
limit=100, # 最大连接数
limit_per_host=20, # 每个主机最大连接数
ttl_dns_cache=300, # DNS缓存时间
use_dns_cache=True,
)
self.session = aiohttp.ClientSession(
connector=connector,
timeout=self.timeout
)
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
if self.session:
await self.session.close()
async def get(self, url, params=None, headers=None):
"""异步GET请求"""
if self.base_url:
url = urljoin(self.base_url, url)
try:
async with self.session.get(url, params=params, headers=headers) as response:
response.raise_for_status()
return await response.json()
except Exception as e:
print(f"GET请求失败 {url}: {e}")
return None
async def post(self, url, data=None, json=None, headers=None):
"""异步POST请求"""
if self.base_url:
url = urljoin(self.base_url, url)
try:
async with self.session.post(
url, data=data, json=json, headers=headers
) as response:
response.raise_for_status()
return await response.json()
except Exception as e:
print(f"POST请求失败 {url}: {e}")
return None
# 模拟API服务（用于测试）
class MockApiService:
"""模拟API服务"""
def __init__(self):
self.users = [
{"id": i, "name": f"User{i}", "email": f"user{i}@example.com",
"age": 20 + i % 30, "score": 60 + i % 40}
for i in range(1, 101) # 100个用户
]
self.posts = [
{"id": i, "user_id": (i % 100) + 1, "title": f"Post {i}",
"content": f"This is the content of post {i}", "likes": i % 50}
for i in range(1, 501) # 500个帖子
]
async def get_user(self, user_id):
"""获取用户信息"""
await asyncio.sleep(0.1) # 模拟网络延迟
for user in self.users:
if user["id"] == user_id:
return user
return None
async def get_user_posts(self, user_id):
"""获取用户的帖子"""
await asyncio.sleep(0.15) # 模拟网络延迟
user_posts = [post for post in self.posts if post["user_id"] == user_id]
return user_posts
async def search_posts(self, keyword):
"""搜索帖子"""
await asyncio.sleep(0.2) # 模拟网络延迟
matching_posts = [
post for post in self.posts
if keyword.lower() in post["title"].lower() or
keyword.lower() in post["content"].lower()
]
return matching_posts
async def fetch_user_data(api_service, user_id):
"""获取单个用户数据"""
print(f"获取用户{user_id}的数据...")
# 获取用户基本信息
user = await api_service.get_user(user_id)
if not user:
return None
# 获取用户帖子
posts = await api_service.get_user_posts(user_id)
return {
"user": user,
"posts": posts,
"post_count": len(posts),
"total_likes": sum(post["likes"] for post in posts)
}
async def batch_fetch_users(api_service, user_ids, max_concurrent=10):
"""批量获取用户数据"""
print(f"批量获取{len(user_ids)}个用户的数据，最大并发: {max_concurrent}")
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_fetch(user_id):
async with semaphore:
return await fetch_user_data(api_service, user_id)
start_time = time.time()
tasks = [limited_fetch(user_id) for user_id in user_ids]
results = await asyncio.gather(*tasks, return_exceptions=True)
end_time = time.time()
# 处理结果
successful_results = []
failed_count = 0
for i, result in enumerate(results):
if isinstance(result, Exception):
print(f"获取用户{user_ids[i]}数据失败: {result}")
failed_count += 1
elif result is not None:
successful_results.append(result)
print(f"批量获取完成: 成功{len(successful_results)}个，失败{failed_count}个，耗时{end_time - start_time:.2f}秒")
return successful_results
async def analyze_user_data(user_data_list):
"""分析用户数据"""
if not user_data_list:
return {}
total_posts = sum(data["post_count"] for data in user_data_list)
total_likes = sum(data["total_likes"] for data in user_data_list)
avg_posts = total_posts / len(user_data_list)
avg_likes = total_likes / len(user_data_list)
# 找出最活跃的用户
most_active = max(user_data_list, key=lambda x: x["post_count"])
most_popular = max(user_data_list, key=lambda x: x["total_likes"])
return {
"total_users": len(user_data_list),
"total_posts": total_posts,
"total_likes": total_likes,
"avg_posts_per_user": avg_posts,
"avg_likes_per_user": avg_likes,
"most_active_user": {
"name": most_active["user"]["name"],
"post_count": most_active["post_count"]
},
"most_popular_user": {
"name": most_popular["user"]["name"],
"total_likes": most_popular["total_likes"]
}
}
async def content_search_analysis(api_service, keywords):
"""内容搜索分析"""
print(f"分析关键词: {keywords}")
search_tasks = []
for keyword in keywords:
task = api_service.search_posts(keyword)
search_tasks.append((keyword, task))
start_time = time.time()
results = await asyncio.gather(*[task for _, task in search_tasks])
end_time = time.time()
search_results = {}
for (keyword, _), posts in zip(search_tasks, results):
search_results[keyword] = {
"post_count": len(posts),
"total_likes": sum(post["likes"] for post in posts),
"posts": posts[:5] # 只保留前5个帖子作为示例
}
print(f"搜索分析完成，耗时{end_time - start_time:.2f}秒")
return search_results
async def test_async_networking():
"""测试异步网络编程"""
print("=== 异步网络编程测试 ===")
api_service = MockApiService()
# 测试用户数据获取
user_ids = list(range(1, 21)) # 获取前20个用户
user_data = await batch_fetch_users(api_service, user_ids, max_concurrent=5)
if user_data:
analysis = await analyze_user_data(user_data)
print("\n=== 用户数据分析结果 ===")
for key, value in analysis.items():
print(f"{key}: {value}")
# 测试内容搜索
keywords = ["python", "async", "programming", "data", "network"]
search_results = await content_search_analysis(api_service, keywords)
print("\n=== 搜索分析结果 ===")
for keyword, result in search_results.items():
print(f"关键词 '{keyword}': {result['post_count']}个帖子, {result['total_likes']}个赞")
# 同步版本对比
def sync_networking_test():
"""同步网络处理对比"""
print("\n=== 同步网络处理对比 ===")
import time
class SyncApiService:
def __init__(self):
self.users = [
{"id": i, "name": f"User{i}", "email": f"user{i}@example.com"}
for i in range(1, 11) # 只用10个用户便于对比
]
def get_user(self, user_id):
time.sleep(0.1) # 同步等待
for user in self.users:
if user["id"] == user_id:
return user
return None
api_service = SyncApiService()
user_ids = list(range(1, 11))
# 同步顺序获取
start_time = time.time()
results = []
for user_id in user_ids:
user = api_service.get_user(user_id)
if user:
results.append(user)
end_time = time.time()
print(f"同步获取{len(results)}个用户数据，耗时: {end_time - start_time:.2f}秒")
print("注意：同步顺序执行，总时间是各个请求延迟之和")
async def test_rate_limiting():
"""测试速率限制"""
print("\n=== 速率限制测试 ===")
class RateLimitedClient:
def __init__(self, requests_per_second=2):
self.requests_per_second = requests_per_second
self.last_request_time = 0
async def make_request(self, url):
"""带速率限制的请求"""
current_time = time.time()
time_since_last = current_time - self.last_request_time
min_interval = 1.0 / self.requests_per_second
if time_since_last < min_interval:
sleep_time = min_interval - time_since_last
await asyncio.sleep(sleep_time)
self.last_request_time = time.time()
print(f"请求: {url} (时间: {self.last_request_time:.2f})")
# 模拟请求处理
await asyncio.sleep(0.1)
return {"url": url, "status": "success"}
client = RateLimitedClient(requests_per_second=3)
urls = [f"https://api.example.com/data/{i}" for i in range(8)]
start_time = time.time()
tasks = [client.make_request(url) for url in urls]
results = await asyncio.gather(*tasks)
end_time = time.time()
print(f"速率限制测试完成: {len(results)}个请求，耗时{end_time - start_time:.2f}秒")
if __name__ == '__main__':
# 安装aiohttp（如果需要）
try:
import aiohttp
except ImportError:
print("请先安装aiohttp: pip install aiohttp")
exit(1)
asyncio.run(test_async_networking())
sync_networking_test()
asyncio.run(test_rate_limiting())
---
7.2 async与await语法
01.async/await语法基础
a.基本语法规则
async def用于定义异步函数，await用于等待异步操作完成。async/await是Python异步编程的核心语法，使异步代码看起来像同步代码一样直观。
b.语法特点
async函数总是返回协程对象，await只能在async函数内部使用，用于暂停当前协程的执行，等待其他协程完成。
02.异步函数定义
a.简单异步函数
使用async def定义的函数是协程函数，调用时不会立即执行，而是返回协程对象，需要通过事件循环来执行。
b.代码示例
---
# async/await基础语法示例
import asyncio
import time
from typing import Any, Coroutine
async def simple_greeting(name: str, delay: float) -> str:
"""简单的异步问候函数"""
print(f"开始准备问候{name}...")
await asyncio.sleep(delay) # 模拟异步操作
greeting = f"你好，{name}！"
print(f"问候{name}完成")
return greeting
async def data_processing(data: list, processing_time: float) -> dict:
"""异步数据处理"""
print(f"开始处理{len(data)}个数据项...")
# 模拟数据处理步骤
await asyncio.sleep(processing_time * 0.3)
filtered_data = [x for x in data if x > 0]
print(f"数据过滤完成: {len(filtered_data)}个有效项")
await asyncio.sleep(processing_time * 0.4)
processed_data = [x * 2 for x in filtered_data]
print(f"数据处理完成")
await asyncio.sleep(processing_time * 0.3)
result = {
'original_count': len(data),
'filtered_count': len(filtered_data),
'processed_data': processed_data,
'sum': sum(processed_data)
}
print(f"结果汇总完成")
return result
async def sequential_execution():
"""顺序执行示例"""
print("=== 顺序执行示例 ===")
start_time = time.time()
# 顺序执行多个异步操作
greeting1 = await simple_greeting("Alice", 1.0)
greeting2 = await simple_greeting("Bob", 0.8)
greeting3 = await simple_greeting("Charlie", 1.2)
data = [1, -2, 3, 4, -5, 6, 7, -8, 9, 10]
result = await data_processing(data, 2.0)
end_time = time.time()
print(f"\n顺序执行结果:")
print(f"问候: {greeting1}, {greeting2}, {greeting3}")
print(f"数据处理: {result}")
print(f"总耗时: {end_time - start_time:.2f}秒")
async def concurrent_execution():
"""并发执行示例"""
print("\n=== 并发执行示例 ===")
start_time = time.time()
# 并发执行多个异步操作
greeting_tasks = [
simple_greeting("Alice", 1.0),
simple_greeting("Bob", 0.8),
simple_greeting("Charlie", 1.2)
]
data = [1, -2, 3, 4, -5, 6, 7, -8, 9, 10]
data_task = data_processing(data, 2.0)
# 使用asyncio.gather并发执行
greetings, data_result = await asyncio.gather(
asyncio.gather(*greeting_tasks),
data_task
)
end_time = time.time()
print(f"\n并发执行结果:")
print(f"问候: {greetings}")
print(f"数据处理: {data_result}")
print(f"总耗时: {end_time - start_time:.2f}秒")
# 演示协程对象特性
def demonstrate_coroutine_objects():
"""演示协程对象"""
print("\n=== 协程对象特性演示 ===")
# 调用异步函数不会立即执行
coro1 = simple_greeting("Test", 0.5)
coro2 = data_processing([1, 2, 3], 1.0)
print(f"coro1类型: {type(coro1)}")
print(f"coro2类型: {type(coro2)}")
print(f"协程对象是可等待的: {asyncio.iscoroutine(coro1)}")
# 需要通过await或事件循环来执行
async def run_coroutines():
result1 = await coro1
result2 = await coro2
print(f"协程1结果: {result1}")
print(f"协程2结果: {result2}")
return run_coroutines()
if __name__ == '__main__':
async def main():
await sequential_execution()
await concurrent_execution()
await demonstrate_coroutine_objects()
asyncio.run(main())
---
03.await表达式详解
a.await的工作机制
await表达式会暂停当前协程的执行，等待被等待的协程完成，然后继续执行。await是异步编程中的控制流工具，用于协调多个异步操作。
b.代码示例
---
# await表达式详解示例
import asyncio
import time
import random
from typing import List, Dict, Optional
class AsyncDatabase:
"""模拟异步数据库"""
def __init__(self):
self.users = {
1: {"name": "Alice", "age": 25, "city": "Beijing"},
2: {"name": "Bob", "age": 30, "city": "Shanghai"},
3: {"name": "Charlie", "age": 35, "city": "Guangzhou"},
4: {"name": "Diana", "age": 28, "city": "Shenzhen"},
5: {"name": "Eve", "age": 22, "city": "Hangzhou"}
}
self.posts = [
{"id": i, "user_id": (i % 5) + 1, "title": f"Post {i}",
"content": f"Content for post {i}", "likes": random.randint(0, 100)}
for i in range(1, 51)
]
async def get_user(self, user_id: int) -> Optional[Dict]:
"""异步获取用户信息"""
await asyncio.sleep(0.1) # 模拟数据库查询延迟
return self.users.get(user_id)
async def get_user_posts(self, user_id: int) -> List[Dict]:
"""异步获取用户帖子"""
await asyncio.sleep(0.15) # 模拟数据库查询延迟
return [post for post in self.posts if post["user_id"] == user_id]
async def get_popular_posts(self, min_likes: int = 50) -> List[Dict]:
"""异步获取热门帖子"""
await asyncio.sleep(0.2) # 模拟数据库查询延迟
return [post for post in self.posts if post["likes"] >= min_likes]
class AsyncDataProcessor:
"""异步数据处理器"""
def __init__(self, db: AsyncDatabase):
self.db = db
async def get_user_profile(self, user_id: int) -> Optional[Dict]:
"""获取用户完整档案 - 演示顺序await"""
print(f"开始获取用户{user_id}的档案...")
# 顺序执行await：先获取用户信息，再获取帖子
user = await self.db.get_user(user_id)
if not user:
print(f"用户{user_id}不存在")
return None
print(f"获取到用户{user_id}基本信息: {user['name']}")
posts = await self.db.get_user_posts(user_id)
print(f"获取到用户{user_id}的{len(posts)}个帖子")
profile = {
"user": user,
"posts": posts,
"post_count": len(posts),
"total_likes": sum(post["likes"] for post in posts),
"avg_likes": sum(post["likes"] for post in posts) / len(posts) if posts else 0
}
print(f"用户{user_id}档案构建完成")
return profile
async def get_multiple_user_profiles(self, user_ids: List[int]) -> Dict[int, Dict]:
"""获取多个用户档案 - 演示并发await"""
print(f"开始批量获取{len(user_ids)}个用户的档案...")
# 并发执行多个get_user_profile调用
profile_tasks = [self.get_user_profile(uid) for uid in user_ids]
profiles = await asyncio.gather(*profile_tasks)
# 构建结果字典
result = {}
for uid, profile in zip(user_ids, profiles):
if profile:
result[uid] = profile
print(f"批量获取完成: {len(result)}个有效档案")
return result
async def analyze_user_activity(self, user_id: int) -> Optional[Dict]:
"""分析用户活动 - 演示复杂的await嵌套"""
print(f"开始分析用户{user_id}的活动...")
# 先获取用户档案（包含await嵌套）
profile = await self.get_user_profile(user_id)
if not profile:
return None
# 获取热门帖子进行对比
popular_posts = await self.db.get_popular_posts(min_likes=30)
# 分析用户在热门帖子中的表现
user_popular_posts = [
post for post in popular_posts
if post["user_id"] == user_id
]
# 计算活动指标
analysis = {
"user_id": user_id,
"user_name": profile["user"]["name"],
"total_posts": profile["post_count"],
"total_likes": profile["total_likes"],
"avg_likes_per_post": profile["avg_likes"],
"popular_posts_count": len(user_popular_posts),
"popular_posts_ratio": len(user_popular_posts) / profile["post_count"] if profile["post_count"] > 0 else 0,
"activity_score": self._calculate_activity_score(profile, user_popular_posts)
}
print(f"用户{user_id}活动分析完成: 活跃度{analysis['activity_score']:.1f}")
return analysis
def _calculate_activity_score(self, profile: Dict, popular_posts: List[Dict]) -> float:
"""计算用户活跃度分数"""
base_score = profile["post_count"] * 10
likes_bonus = profile["total_likes"] * 0.5
popular_bonus = len(popular_posts) * 100
return base_score + likes_bonus + popular_bonus
async def generate_user_report(self, user_id: int) -> Optional[str]:
"""生成用户报告 - 演示await在实际应用中的使用"""
print(f"开始为用户{user_id}生成报告...")
# 获取用户活动分析
analysis = await self.analyze_user_activity(user_id)
if not analysis:
return None
# 模拟报告生成
await asyncio.sleep(0.5) # 模拟报告生成时间
report = f"""
用户活动报告
================
用户名: {analysis['user_name']}
总发帖数: {analysis['total_posts']}
总获赞数: {analysis['total_likes']}
平均每帖获赞: {analysis['avg_likes_per_post']:.1f}
热门帖子数: {analysis['popular_posts_count']}
热门帖子比例: {analysis['popular_posts_ratio']:.2%}
活跃度评分: {analysis['activity_score']:.1f}
报告生成时间: {time.strftime('%Y-%m-%d %H:%M:%S')}
"""
print(f"用户{user_id}报告生成完成")
return report.strip()
async def demonstrate_await_patterns():
"""演示不同的await模式"""
print("=== await模式演示 ===")
db = AsyncDatabase()
processor = AsyncDataProcessor(db)
# 1. 简单await
print("\n1. 简单await示例:")
user = await db.get_user(1)
print(f"获取用户: {user}")
# 2. 顺序await
print("\n2. 顺序await示例:")
user = await db.get_user(2)
posts = await db.get_user_posts(2)
print(f"用户{user['name']}有{len(posts)}个帖子")
# 3. 并发await
print("\n3. 并发await示例:")
user_task = db.get_user(3)
posts_task = db.get_user_posts(3)
popular_task = db.get_popular_posts()
user, posts, popular = await asyncio.gather(user_task, posts_task, popular_task)
print(f"并发获取: 用户{user['name']}, {len(posts)}个帖子, {len(popular)}个热门帖子")
# 4. 嵌套await
print("\n4. 嵌套await示例:")
profile = await processor.get_user_profile(4)
if profile:
print(f"用户档案: {profile['user']['name']} - {profile['post_count']}个帖子")
# 5. 复杂业务逻辑中的await
print("\n5. 复杂业务逻辑中的await:")
report = await processor.generate_user_report(5)
if report:
print(report)
async def test_await_error_handling():
"""测试await的错误处理"""
print("\n=== await错误处理测试 ===")
db = AsyncDatabase()
# 演示await中的异常处理
async def safe_get_user(user_id: int):
try:
user = await db.get_user(user_id)
if user:
return user
else:
raise ValueError(f"用户{user_id}不存在")
except Exception as e:
print(f"获取用户{user_id}时出错: {e}")
return None
# 测试正常情况
user1 = await safe_get_user(1)
print(f"正常获取: {user1['name'] if user1 else 'None'}")
# 测试异常情况
user999 = await safe_get_user(999)
print(f"异常获取: {user999['name'] if user999 else 'None'}")
# 演示asyncio.wait_for超时处理
try:
# 设置很短的超时时间来触发超时
user = await asyncio.wait_for(db.get_user(1), timeout=0.05)
print(f"在超时时间内获取到用户: {user['name']}")
except asyncio.TimeoutError:
print("获取用户超时")
except Exception as e:
print(f"获取用户时发生其他错误: {e}")
async def test_await_performance_comparison():
"""测试await的性能对比"""
print("\n=== await性能对比测试 ===")
db = AsyncDatabase()
user_ids = [1, 2, 3, 4, 5]
# 顺序执行
print("\n顺序执行获取用户数据:")
start_time = time.time()
sequential_users = []
for uid in user_ids:
user = await db.get_user(uid)
if user:
sequential_users.append(user)
sequential_time = time.time() - start_time
print(f"顺序执行耗时: {sequential_time:.3f}秒")
# 并发执行
print("\n并发执行获取用户数据:")
start_time = time.time()
user_tasks = [db.get_user(uid) for uid in user_ids]
concurrent_users = await asyncio.gather(*user_tasks)
concurrent_users = [user for user in concurrent_users if user] # 过滤None值
concurrent_time = time.time() - start_time
print(f"并发执行耗时: {concurrent_time:.3f}秒")
# 性能对比
if concurrent_time > 0:
speedup = sequential_time / concurrent_time
print(f"\n性能提升: {speedup:.2f}倍")
if __name__ == '__main__':
async def main():
await demonstrate_await_patterns()
await test_await_error_handling()
await test_await_performance_comparison()
asyncio.run(main())
---
04.高级异步模式
a.异步上下文管理器
使用async with和__aenter__、__aexit__方法实现异步上下文管理，确保资源的正确获取和释放。
b.代码示例
---
# 高级异步模式示例
import asyncio
import time
import random
from contextlib import asynccontextmanager
from typing import AsyncIterator, Any, Optional
class AsyncResource:
"""模拟异步资源"""
def __init__(self, name: str, setup_time: float = 0.1):
self.name = name
self.setup_time = setup_time
self.is_initialized = False
self.data = None
async def initialize(self):
"""异步初始化资源"""
print(f"正在初始化资源 {self.name}...")
await asyncio.sleep(self.setup_time)
self.data = f"Resource {self.name} data"
self.is_initialized = True
print(f"资源 {self.name} 初始化完成")
async def cleanup(self):
"""异步清理资源"""
print(f"正在清理资源 {self.name}...")
await asyncio.sleep(0.05)
self.data = None
self.is_initialized = False
print(f"资源 {self.name} 清理完成")
async def __aenter__(self):
await self.initialize()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
await self.cleanup()
async def process_data(self, data: Any) -> str:
"""异步处理数据"""
if not self.is_initialized:
raise RuntimeError(f"资源 {self.name} 未初始化")
await asyncio.sleep(0.1) # 模拟处理时间
result = f"Processed by {self.name}: {data} -> {self.data}"
return result
@asynccontextmanager
async def temporary_resource(name: str, setup_time: float = 0.1):
"""临时异步资源上下文管理器"""
resource = AsyncResource(name, setup_time)
try:
await resource.initialize()
yield resource
finally:
await resource.cleanup()
class AsyncDatabaseConnection:
"""异步数据库连接"""
def __init__(self, connection_string: str):
self.connection_string = connection_string
self.is_connected = False
self.connection_id = None
async def connect(self):
"""异步连接数据库"""
print(f"正在连接到数据库: {self.connection_string}")
await asyncio.sleep(0.2) # 模拟连接时间
self.connection_id = random.randint(1000, 9999)
self.is_connected = True
print(f"数据库连接成功，连接ID: {self.connection_id}")
async def disconnect(self):
"""异步断开数据库连接"""
if self.is_connected:
print(f"正在断开数据库连接，连接ID: {self.connection_id}")
await asyncio.sleep(0.1) # 模拟断开时间
self.is_connected = False
self.connection_id = None
print("数据库连接已断开")
async def execute_query(self, query: str) -> list:
"""异步执行查询"""
if not self.is_connected:
raise RuntimeError("数据库未连接")
await asyncio.sleep(0.3) # 模拟查询时间
print(f"执行查询: {query[:50]}...")
# 模拟查询结果
result = [
{"id": i, "name": f"Record {i}", "value": random.randint(1, 100)}
for i in range(1, random.randint(3, 8))
]
return result
async def __aenter__(self):
await self.connect()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
await self.disconnect()
class AsyncFileHandler:
"""异步文件处理器"""
def __init__(self, filename: str, mode: str = 'r'):
self.filename = filename
self.mode = mode
self.file = None
async def __aenter__(self):
"""异步进入上下文"""
print(f"打开文件: {self.filename}")
await asyncio.sleep(0.01) # 模拟文件打开时间
# 这里简化处理，实际应该使用aiofiles
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""异步退出上下文"""
print(f"关闭文件: {self.filename}")
await asyncio.sleep(0.01) # 模拟文件关闭时间
async def read_lines(self) -> list:
"""异步读取文件行"""
await asyncio.sleep(0.1) # 模拟读取时间
# 模拟文件内容
lines = [
f"Line {i}: This is sample data from {self.filename}"
for i in range(1, random.randint(5, 15))
]
return lines
async def write_data(self, data: str):
"""异步写入数据"""
await asyncio.sleep(0.1) # 模拟写入时间
print(f"写入数据到 {self.filename}: {data[:50]}...")
async def demonstrate_async_context_managers():
"""演示异步上下文管理器"""
print("=== 异步上下文管理器演示 ===")
# 1. 使用自定义异步上下文管理器
print("\n1. 自定义异步资源:")
async with AsyncResource("Calculator", 0.1) as resource:
result1 = await resource.process_data("input1")
result2 = await resource.process_data("input2")
print(f"处理结果1: {result1}")
print(f"处理结果2: {result2}")
# 2. 使用函数式异步上下文管理器
print("\n2. 函数式异步资源:")
async with temporary_resource("TempProcessor", 0.05) as temp_resource:
result = await temp_resource.process_data("temporary_data")
print(f"临时资源处理结果: {result}")
# 3. 数据库连接示例
print("\n3. 异步数据库连接:")
async with AsyncDatabaseConnection("postgresql://localhost:5432/mydb") as db:
# 执行多个查询
query1 = "SELECT * FROM users WHERE active = true"
query2 = "SELECT * FROM products WHERE price > 100"
results1 = await db.execute_query(query1)
results2 = await db.execute_query(query2)
print(f"查询1结果: {len(results1)}条记录")
print(f"查询2结果: {len(results2)}条记录")
# 4. 文件处理示例
print("\n4. 异步文件处理:")
async with AsyncFileHandler("data.txt", 'w') as file_handler:
await file_handler.write_data("Sample async file content")
async with AsyncFileHandler("data.txt", 'r') as file_handler:
lines = await file_handler.read_lines()
print(f"读取到 {len(lines)} 行数据")
async def demonstrate_nested_async_contexts():
"""演示嵌套异步上下文"""
print("\n=== 嵌套异步上下文演示 ===")
# 嵌套多个异步上下文管理器
async with AsyncResource("Processor1", 0.1) as resource1:
async with AsyncResource("Processor2", 0.1) as resource2:
async with AsyncDatabaseConnection("db://example") as db:
print("所有资源已准备就绪，开始复合操作...")
# 复合数据处理
data = "complex_data"
result1 = await resource1.process_data(data)
result2 = await resource2.process_data(result1)
query = f"SELECT * FROM analysis WHERE result = '{result2}'"
db_results = await db.execute_query(query)
print(f"复合处理完成: {len(db_results)}条数据库记录")
class AsyncConnectionPool:
"""异步连接池"""
def __init__(self, pool_size: int = 3):
self.pool_size = pool_size
self.connections = []
self.available = asyncio.Queue()
self.semaphore = asyncio.Semaphore(pool_size)
async def __aenter__(self):
"""初始化连接池"""
print(f"初始化连接池，大小: {self.pool_size}")
for i in range(self.pool_size):
conn = AsyncDatabaseConnection(f"connection_{i+1}")
await conn.connect()
self.connections.append(conn)
await self.available.put(conn)
print("连接池初始化完成")
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
"""清理连接池"""
print("清理连接池...")
for conn in self.connections:
await conn.disconnect()
self.connections.clear()
print("连接池清理完成")
async def get_connection(self) -> AsyncDatabaseConnection:
"""获取连接"""
async with self.semaphore:
print("等待获取连接...")
conn = await self.available.get()
print(f"获取到连接: {conn.connection_id}")
return conn
async def release_connection(self, conn: AsyncDatabaseConnection):
"""释放连接"""
await self.available.put(conn)
print(f"释放连接: {conn.connection_id}")
async def demonstrate_connection_pool():
"""演示异步连接池"""
print("\n=== 异步连接池演示 ===")
async with AsyncConnectionPool(pool_size=3) as pool:
# 创建多个并发任务
async def worker_task(worker_id: int, query_count: int):
"""工作任务"""
print(f"工作器{worker_id}开始执行{query_count}个查询")
results = []
for i in range(query_count):
conn = await pool.get_connection()
try:
query = f"Worker{worker_id}_Query{i+1}"
result = await conn.execute_query(query)
results.extend(result)
finally:
await pool.release_connection(conn)
await asyncio.sleep(0.1) # 任务间隔
print(f"工作器{worker_id}完成，共获取{len(results)}条记录")
return results
# 启动多个工作器
tasks = [
worker_task(1, 3),
worker_task(2, 2),
worker_task(3, 4),
worker_task(4, 2)
]
start_time = time.time()
all_results = await asyncio.gather(*tasks)
end_time = time.time()
total_records = sum(len(results) for results in all_results)
print(f"\n所有工作器完成:")
print(f"- 总记录数: {total_records}")
print(f"- 总耗时: {end_time - start_time:.2f}秒")
print(f"- 平均每工作器记录数: {total_records / 4:.1f}")
if __name__ == '__main__':
async def main():
await demonstrate_async_context_managers()
await demonstrate_nested_async_contexts()
await demonstrate_connection_pool()
asyncio.run(main())
---
05.异步迭代器
a.异步迭代器概念
通过实现__aiter__和__anext__方法，可以创建异步迭代器，使用async for进行异步遍历，特别适合流式数据处理。
b.代码示例
---
# 异步迭代器示例
import asyncio
import time
import random
from typing import AsyncIterator, List, Dict, Any
class AsyncDataProducer:
"""异步数据生产者"""
def __init__(self, data_source: str, batch_size: int = 5):
self.data_source = data_source
self.batch_size = batch_size
self.total_items = 20
self.current_index = 0
def __aiter__(self) -> AsyncIterator[List[Dict]]:
"""返回异步迭代器"""
return self
async def __anext__(self) -> List[Dict]:
"""异步获取下一批数据"""
if self.current_index >= self.total_items:
raise StopAsyncIteration
print(f"从{self.data_source}获取第{self.current_index // self.batch_size + 1}批数据...")
# 模拟异步数据获取
await asyncio.sleep(0.5)
# 生成一批数据
batch = []
for i in range(self.batch_size):
if self.current_index < self.total_items:
item = {
"id": self.current_index + 1,
"source": self.data_source,
"value": random.randint(1, 1000),
"timestamp": time.time(),
"category": random.choice(["A", "B", "C", "D"])
}
batch.append(item)
self.current_index += 1
print(f"获取到{len(batch)}个数据项")
return batch
class AsyncStreamProcessor:
"""异步流处理器"""
def __init__(self, data_producer: AsyncDataProducer):
self.data_producer = data_producer
self.processed_count = 0
self.filtered_count = 0
async def process_stream(self) -> AsyncIterator[Dict]:
"""异步处理数据流"""
print("开始异步流处理...")
async for batch in self.data_producer:
print(f"处理批次: {len(batch)}个数据项")
for item in batch:
# 异步处理每个数据项
processed_item = await self.process_item(item)
self.processed_count += 1
# 过滤条件：值大于500且类别为A或B
if processed_item["value"] > 500 and processed_item["category"] in ["A", "B"]:
self.filtered_count += 1
yield processed_item
# 模拟处理延迟
await asyncio.sleep(0.1)
print(f"流处理完成: 处理了{self.processed_count}个项，过滤出{self.filtered_count}个项")
async def process_item(self, item: Dict) -> Dict:
"""异步处理单个数据项"""
# 模拟复杂的异步处理
await asyncio.sleep(0.05)
processed = {
**item,
"processed": True,
"processed_at": time.time(),
"value_squared": item["value"] ** 2
}
return processed
class AsyncFileReader:
"""异步文件读取器"""
def __init__(self, filename: str, chunk_size: int = 1024):
self.filename = filename
self.chunk_size = chunk_size
self.file_content = self._generate_file_content()
self.position = 0
def _generate_file_content(self) -> str:
"""生成模拟文件内容"""
lines = []
for i in range(100): # 100行模拟数据
line = f"Line {i+1}: {random.randint(1, 1000)} {random.choice(['apple', 'banana', 'cherry', 'date'])}"
lines.append(line)
return '\n'.join(lines)
def __aiter__(self) -> AsyncIterator[str]:
"""返回异步迭代器"""
return self
async def __anext__(self) -> str:
"""异步读取下一行"""
if self.position >= len(self.file_content):
raise StopAsyncIteration
# 查找下一行
next_newline = self.file_content.find('\n', self.position)
if next_newline == -1:
line = self.file_content[self.position:]
self.position = len(self.file_content)
else:
line = self.file_content[self.position:next_newline]
self.position = next_newline + 1
# 模拟异步读取
await asyncio.sleep(0.01)
return line.strip()
class AsyncNetworkStream:
"""模拟异步网络数据流"""
def __init__(self, stream_id: str, message_count: int = 15):
self.stream_id = stream_id
self.message_count = message_count
self.current_message = 0
self.message_types = ["info", "warning", "error", "debug"]
def __aiter__(self) -> AsyncIterator[Dict]:
return self
async def __anext__(self) -> Dict:
"""异步获取下一条网络消息"""
if self.current_message >= self.message_count:
raise StopAsyncIteration
# 模拟网络延迟
await asyncio.sleep(random.uniform(0.1, 0.5))
message = {
"id": self.current_message + 1,
"stream_id": self.stream_id,
"type": random.choice(self.message_types),
"content": f"Message {self.current_message + 1} from stream {self.stream_id}",
"timestamp": time.time(),
"priority": random.randint(1, 5)
}
self.current_message += 1
return message
async def demonstrate_async_iterators():
"""演示异步迭代器"""
print("=== 异步迭代器演示 ===")
# 1. 数据生产者示例
print("\n1. 异步数据生产者:")
producer = AsyncDataProducer("Database_A", batch_size=4)
total_items = 0
async for batch in producer:
print(f" 批次包含: {len(batch)}个数据项")
for item in batch:
print(f" ID: {item['id']}, 值: {item['value']}, 类别: {item['category']}")
total_items += 1
print(f"总共获取了{total_items}个数据项")
# 2. 流处理器示例
print("\n2. 异步流处理器:")
producer = AsyncDataProducer("Sensor_Data", batch_size=3)
processor = AsyncStreamProcessor(producer)
filtered_results = []
async for filtered_item in processor.process_stream():
filtered_results.append(filtered_item)
print(f"过滤项: ID={filtered_item['id']}, 值={filtered_item['value']}, 平方值={filtered_item['value_squared']}")
print(f"流处理完成，过滤出{len(filtered_results)}个结果")
# 3. 文件读取器示例
print("\n3. 异步文件读取器:")
reader = AsyncFileReader("sample.log")
line_count = 0
total_numbers = 0
async for line in reader:
# 解析行中的数字
parts = line.split()
for part in parts:
if part.isdigit():
total_numbers += int(part)
line_count += 1
if line_count <= 5: # 只打印前5行
print(f" 第{line_count}行: {line}")
elif line_count == 6:
print(" ...")
print(f"文件读取完成: {line_count}行，数字总和: {total_numbers}")
# 4. 网络流示例
print("\n4. 异步网络流:")
stream = AsyncNetworkStream("Server_A", message_count=10)
message_stats = {}
async for message in stream:
msg_type = message["type"]
if msg_type not in message_stats:
message_stats[msg_type] = 0
message_stats[msg_type] += 1
print(f"消息: {message['type']} - {message['content']} (优先级: {message['priority']})")
print(f"网络流完成，消息统计: {message_stats}")
async def demonstrate_async_comprehensions():
"""演示异步推导式"""
print("\n=== 异步推导式演示 ===")
# 异步列表推导式
print("\n1. 异步列表推导式:")
async def async_number_generator():
"""异步数字生成器"""
for i in range(1, 11):
await asyncio.sleep(0.1)
yield i * i
# 异步列表推导式
squares = [num async for num in async_number_generator()]
print(f"平方数列表: {squares}")
# 带条件的异步推导式
even_squares = [num async for num in async_number_generator() if num % 2 == 0]
print(f"偶数平方数: {even_squares}")
# 异步字典推导式
print("\n2. 异步字典推导式:")
async def async_user_generator():
"""异步用户生成器"""
users = ["Alice", "Bob", "Charlie", "Diana"]
for i, name in enumerate(users):
await asyncio.sleep(0.05)
yield {"id": i + 1, "name": name, "score": random.randint(60, 100)}
user_dict = {user["id"]: user["name"] async for user in async_user_generator()}
print(f"用户字典: {user_dict}")
# 异步集合推导式
print("\n3. 异步集合推导式:")
async def async_word_generator():
"""异步单词生成器"""
words = ["apple", "banana", "apple", "cherry", "banana", "date"]
for word in words:
await asyncio.sleep(0.05)
yield word.upper()
unique_words = {word async for word in async_word_generator()}
print(f"唯一单词集合: {unique_words}")
async def demonstrate_multiple_streams():
"""演示多流并发处理"""
print("\n=== 多流并发处理演示 ===")
# 创建多个数据流
streams = [
AsyncDataProducer(f"Source_{i+1}", batch_size=2)
for i in range(3)
]
# 并发处理多个流
async def process_stream_with_name(stream_name: str, stream: AsyncDataProducer):
"""处理命名流"""
print(f"\n开始处理流: {stream_name}")
items = []
async for batch in stream:
for item in batch:
item["stream_name"] = stream_name
items.append(item)
print(f"流{stream_name}处理完成，共{len(items)}个项")
return items
# 创建并发任务
stream_tasks = [
process_stream_with_name(f"Stream_{i+1}", stream)
for i, stream in enumerate(streams)
]
# 等待所有流处理完成
start_time = time.time()
all_stream_results = await asyncio.gather(*stream_tasks)
end_time = time.time()
# 合并结果
all_items = []
for stream_items in all_stream_results:
all_items.extend(stream_items)
# 分析结果
total_items = len(all_items)
avg_value = sum(item["value"] for item in all_items) / total_items if total_items > 0 else 0
print(f"\n=== 多流处理结果 ===")
print(f"总数据项: {total_items}")
print(f"平均值: {avg_value:.2f}")
print(f"处理耗时: {end_time - start_time:.2f}秒")
print(f"数据源分布: {len(set(item['source'] for item in all_items))}个不同源")
if __name__ == '__main__':
async def main():
await demonstrate_async_iterators()
await demonstrate_async_comprehensions()
await demonstrate_multiple_streams()
asyncio.run(main())
---
7.3 协程与线程对比
01.协程与线程的基本概念
a.定义与特点
协程是轻量级的执行单元，由程序员控制调度，在单线程中实现并发。线程是操作系统调度的执行单元，可以并行运行在多核处理器上。
b.核心区别
协程通过协作式调度，线程通过抢占式调度；协程切换开销小，线程切换开销大；协程无需锁机制，线程需要同步机制。
02.执行模型对比
a.单线程多协程模型
协程在单线程中运行，通过事件循环调度，避免了多线程的上下文切换开销和锁竞争问题。
b.多线程模型
线程由操作系统调度，可以利用多核处理器实现真正的并行，但需要处理线程同步和数据竞争问题。
c.代码示例
---
# 协程与线程执行模型对比示例
import asyncio
import threading
import time
import random
from concurrent.futures import ThreadPoolExecutor
from typing import List, Dict, Any
# 共享数据模拟
shared_counter = 0
counter_lock = threading.Lock()
class TaskResult:
"""任务结果类"""
def __init__(self, worker_id: str, task_id: int, processing_time: float, result: int):
self.worker_id = worker_id
self.task_id = task_id
self.processing_time = processing_time
self.result = result
self.start_time = time.time()
self.end_time = None
def finish(self):
"""标记任务完成"""
self.end_time = time.time()
@property
def duration(self) -> float:
"""获取任务执行时间"""
return (self.end_time or time.time()) - self.start_time
# ==================== 协程版本 ====================
async def coroutine_worker(worker_id: str, task_count: int, delay_range: tuple = (0.1, 0.5)):
"""协程工作函数"""
print(f"协程工作者{worker_id}启动，将处理{task_count}个任务")
results = []
for i in range(task_count):
task_id = f"{worker_id}_task_{i+1}"
# 模拟异步I/O操作
delay = random.uniform(*delay_range)
await asyncio.sleep(delay)
# 执行计算
computation_result = sum(random.randint(1, 100) for _ in range(1000))
result = TaskResult(worker_id, task_id, delay, computation_result)
result.finish()
results.append(result)
print(f"协程工作者{worker_id}完成任务{task_id}，结果: {computation_result:,}")
print(f"协程工作者{worker_id}完成所有任务")
return results
async def coroutine_based_processing(total_tasks: int, num_workers: int = 3):
"""基于协程的并发处理"""
print(f"\n=== 协程并发处理 {total_tasks} 个任务，{num_workers} 个工作者 ===")
tasks_per_worker = total_tasks // num_workers
remaining_tasks = total_tasks % num_workers
# 创建协程任务
worker_tasks = []
for i in range(num_workers):
worker_id = f"coroutine_{i+1}"
task_count = tasks_per_worker + (1 if i < remaining_tasks else 0)
worker_tasks.append(coroutine_worker(worker_id, task_count))
start_time = time.time()
# 并发执行所有协程工作者
all_results = await asyncio.gather(*worker_tasks)
end_time = time.time()
# 汇总结果
flat_results = [result for worker_results in all_results for result in worker_results]
print(f"\n协程处理完成:")
print(f"- 总任务数: {len(flat_results)}")
print(f"- 总耗时: {end_time - start_time:.3f}秒")
print(f"- 平均每任务耗时: {sum(r.duration for r in flat_results) / len(flat_results):.3f}秒")
return {
"model": "协程",
"total_tasks": len(flat_results),
"total_time": end_time - start_time,
"results": flat_results
}
# ==================== 线程版本 ====================
def thread_worker(worker_id: str, task_count: int, results: List[TaskResult],
delay_range: tuple = (0.1, 0.5)):
"""线程工作函数"""
print(f"线程工作者{worker_id}启动，将处理{task_count}个任务")
for i in range(task_count):
task_id = f"{worker_id}_task_{i+1}"
# 模拟I/O操作（使用time.sleep，会阻塞整个线程）
delay = random.uniform(*delay_range)
time.sleep(delay)
# 执行计算
computation_result = sum(random.randint(1, 100) for _ in range(1000))
result = TaskResult(worker_id, task_id, delay, computation_result)
result.finish()
# 线程安全地添加结果
with counter_lock:
results.append(result)
print(f"线程工作者{worker_id}完成任务{task_id}，结果: {computation_result:,}")
print(f"线程工作者{worker_id}完成所有任务")
def thread_based_processing(total_tasks: int, num_workers: int = 3):
"""基于线程的并发处理"""
print(f"\n=== 线程并发处理 {total_tasks} 个任务，{num_workers} 个工作者 ===")
tasks_per_worker = total_tasks // num_workers
remaining_tasks = total_tasks % num_workers
# 共享结果列表
results = []
start_time = time.time()
# 创建并启动线程
threads = []
for i in range(num_workers):
worker_id = f"thread_{i+1}"
task_count = tasks_per_worker + (1 if i < remaining_tasks else 0)
thread = threading.Thread(
target=thread_worker,
args=(worker_id, task_count, results)
)
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
end_time = time.time()
print(f"\n线程处理完成:")
print(f"- 总任务数: {len(results)}")
print(f"- 总耗时: {end_time - start_time:.3f}秒")
print(f"- 平均每任务耗时: {sum(r.duration for r in results) / len(results):.3f}秒")
return {
"model": "线程",
"total_tasks": len(results),
"total_time": end_time - start_time,
"results": results
}
# ==================== 性能对比分析 ====================
def analyze_results(coroutine_result: Dict, thread_result: Dict) -> Dict:
"""分析性能对比结果"""
print(f"\n=== 性能对比分析 ===")
coro_data = coroutine_result
thread_data = thread_result
# 基本性能对比
coro_time = coro_data["total_time"]
thread_time = thread_data["total_time"]
if thread_time > 0:
speedup = thread_time / coro_time
efficiency = coro_time / thread_time * 100
else:
speedup = float('inf')
efficiency = 0
print(f"总耗时对比:")
print(f"- 协程模型: {coro_time:.3f}秒")
print(f"- 线程模型: {thread_time:.3f}秒")
print(f"- 性能提升: {speedup:.2f}倍")
print(f"- 相对效率: {efficiency:.1f}%")
# 任务处理效率
coro_tasks = coro_data["total_tasks"]
thread_tasks = thread_data["total_tasks"]
if coro_time > 0 and thread_time > 0:
coro_throughput = coro_tasks / coro_time
thread_throughput = thread_tasks / thread_time
print(f"\n吞吐量对比:")
print(f"- 协程模型: {coro_throughput:.1f} 任务/秒")
print(f"- 线程模型: {thread_throughput:.1f} 任务/秒")
return {
"speedup": speedup,
"efficiency": efficiency,
"coro_time": coro_time,
"thread_time": thread_time
}
async def run_performance_comparison():
"""运行性能对比测试"""
print("=== 协程 vs 线程性能对比测试 ===")
test_scenarios = [
{"tasks": 30, "workers": 3, "name": "小规模"},
{"tasks": 60, "workers": 5, "name": "中等规模"},
{"tasks": 100, "workers": 8, "name": "大规模"}
]
all_results = []
for scenario in test_scenarios:
print(f"\n{'='*50}")
print(f"测试场景: {scenario['name']} - {scenario['tasks']}个任务，{scenario['workers']}个工作者")
print(f"{'='*50}")
# 协程测试
coro_result = await coroutine_based_processing(
scenario["tasks"],
scenario["workers"]
)
# 线程测试
thread_result = thread_based_processing(
scenario["tasks"],
scenario["workers"]
)
# 分析结果
analysis = analyze_results(coro_result, thread_result)
all_results.append({
"scenario": scenario["name"],
"coroutine": coro_result,
"thread": thread_result,
"analysis": analysis
})
# 测试间隔
await asyncio.sleep(1)
# 总结报告
print(f"\n{'='*60}")
print("性能对比总结报告")
print(f"{'='*60}")
for result in all_results:
scenario = result["scenario"]
analysis = result["analysis"]
print(f"\n{scenario}测试:")
print(f" 性能提升: {analysis['speedup']:.2f}倍")
print(f" 协程耗时: {analysis['coro_time']:.3f}秒")
print(f" 线程耗时: {analysis['thread_time']:.3f}秒")
if __name__ == '__main__':
asyncio.run(run_performance_comparison())
---
03.内存使用对比
a.协程内存效率
协程是轻量级执行单元，内存占用极小（通常几KB），单线程可以运行数万个协程，内存使用效率极高。
b.线程内存开销
线程需要独立的栈空间（通常几MB），操作系统对线程数量有限制，内存使用效率相对较低。
c.代码示例
---
# 协程与线程内存使用对比示例
import asyncio
import threading
import time
import psutil
import os
from typing import List, Dict, Any
class MemoryMonitor:
"""内存监控器"""
def __init__(self):
self.process = psutil.Process(os.getpid())
self.baseline_memory = self.get_memory_usage()
self.measurements = []
def get_memory_usage(self) -> float:
"""获取当前内存使用量（MB）"""
return self.process.memory_info().rss / 1024 / 1024
def record_measurement(self, label: str):
"""记录内存测量"""
current_memory = self.get_memory_usage()
memory_increase = current_memory - self.baseline_memory
measurement = {
"label": label,
"timestamp": time.time(),
"memory_mb": current_memory,
"increase_mb": memory_increase
}
self.measurements.append(measurement)
print(f"内存测量 - {label}: {current_memory:.2f}MB (+{memory_increase:.2f}MB)")
def get_summary(self) -> Dict:
"""获取内存使用总结"""
if not self.measurements:
return {}
max_memory = max(m["memory_mb"] for m in self.measurements)
max_increase = max(m["increase_mb"] for m in self.measurements)
return {
"baseline_mb": self.baseline_memory,
"peak_mb": max_memory,
"peak_increase_mb": max_increase,
"measurements": self.measurements
}
# ==================== 协程内存测试 ====================
async def memory_intensive_coroutine(coroutine_id: int, data_size: int = 1000):
"""内存密集型协程"""
# 创建一些数据来增加内存使用
data = list(range(data_size))
# 模拟处理时间
await asyncio.sleep(0.1)
# 处理数据
result = sum(data) * coroutine_id
return result
async def coroutine_memory_test(coroutine_count: int):
"""协程内存使用测试"""
print(f"\n=== 协程内存测试：{coroutine_count}个协程 ===")
monitor = MemoryMonitor()
monitor.record_measurement("协程测试开始")
# 创建大量协程
tasks = []
for i in range(coroutine_count):
task = memory_intensive_coroutine(i, data_size=1000)
tasks.append(task)
# 每100个协程记录一次内存
if (i + 1) % 100 == 0:
monitor.record_measurement(f"创建{i+1}个协程")
monitor.record_measurement(f"所有{coroutine_count}个协程创建完成")
# 执行所有协程
start_time = time.time()
results = await asyncio.gather(*tasks)
end_time = time.time()
monitor.record_measurement("协程执行完成")
# 清理结果
del results
monitor.record_measurement("协程结果清理完成")
summary = monitor.get_summary()
print(f"\n协程内存测试结果:")
print(f"- 协程数量: {coroutine_count}")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
print(f"- 基准内存: {summary['baseline_mb']:.2f}MB")
print(f"- 峰值内存: {summary['peak_mb']:.2f}MB")
print(f"- 内存增长: {summary['peak_increase_mb']:.2f}MB")
print(f"- 平均每协程: {summary['peak_increase_mb'] / coroutine_count * 1024:.2f}KB")
return summary
# ==================== 线程内存测试 ====================
def memory_intensive_thread(thread_id: int, data_size: int = 1000, results: List = None):
"""内存密集型线程"""
if results is None:
results = []
# 创建一些数据来增加内存使用
data = list(range(data_size))
# 模拟处理时间
time.sleep(0.1)
# 处理数据
result = sum(data) * thread_id
results.append(result)
return result
def thread_memory_test(thread_count: int):
"""线程内存使用测试"""
print(f"\n=== 线程内存测试：{thread_count}个线程 ===")
monitor = MemoryMonitor()
monitor.record_measurement("线程测试开始")
results = []
threads = []
# 创建大量线程
for i in range(thread_count):
thread = threading.Thread(
target=memory_intensive_thread,
args=(i, 1000, results)
)
threads.append(thread)
# 每10个线程记录一次内存
if (i + 1) % 10 == 0:
monitor.record_measurement(f"创建{i+1}个线程")
monitor.record_measurement(f"所有{thread_count}个线程创建完成")
# 启动所有线程
start_time = time.time()
for thread in threads:
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
end_time = time.time()
monitor.record_measurement("线程执行完成")
# 清理结果
del results
monitor.record_measurement("线程结果清理完成")
summary = monitor.get_summary()
print(f"\n线程内存测试结果:")
print(f"- 线程数量: {thread_count}")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
print(f"- 基准内存: {summary['baseline_mb']:.2f}MB")
print(f"- 峰值内存: {summary['peak_mb']:.2f}MB")
print(f"- 内存增长: {summary['peak_increase_mb']:.2f}MB")
print(f"- 平均每线程: {summary['peak_increase_mb'] / thread_count:.2f}MB")
return summary
# ==================== 综合内存对比测试 ====================
async def comprehensive_memory_comparison():
"""综合内存使用对比测试"""
print("=== 协程 vs 线程内存使用对比 ===")
test_cases = [
{"coroutines": 1000, "threads": 100, "name": "小规模"},
{"coroutines": 5000, "threads": 200, "name": "中等规模"},
{"coroutines": 10000, "threads": 300, "name": "大规模"}
]
comparison_results = []
for case in test_cases:
print(f"\n{'='*60}")
print(f"测试案例: {case['name']}")
print(f"协程数量: {case['coroutines']}, 线程数量: {case['threads']}")
print(f"{'='*60}")
# 协程测试
coro_summary = await coroutine_memory_test(case["coroutines"])
# 等待内存稳定
await asyncio.sleep(2)
# 线程测试
thread_summary = thread_memory_test(case["threads"])
# 对比分析
coro_memory_per = coro_summary["peak_increase_mb"] / case["coroutines"] * 1024 # KB
thread_memory_per = thread_summary["peak_increase_mb"] / case["threads"] # MB
print(f"\n{case['name']}内存对比:")
print(f"- 协程总增长: {coro_summary['peak_increase_mb']:.2f}MB")
print(f"- 线程总增长: {thread_summary['peak_increase_mb']:.2f}MB")
print(f"- 协程单个: {coro_memory_per:.2f}KB")
print(f"- 线程单个: {thread_memory_per:.2f}MB")
print(f"- 内存效率比: {thread_memory_per * 1024 / coro_memory_per:.1f}倍")
comparison_results.append({
"case": case["name"],
"coroutine": coro_summary,
"thread": thread_summary,
"memory_efficiency_ratio": thread_memory_per * 1024 / coro_memory_per
})
# 测试间隔，让系统内存稳定
await asyncio.sleep(3)
# 总结报告
print(f"\n{'='*60}")
print("内存使用对比总结报告")
print(f"{'='*60}")
for result in comparison_results:
case = result["case"]
efficiency_ratio = result["memory_efficiency_ratio"]
print(f"\n{case}测试:")
print(f" 内存效率优势: {efficiency_ratio:.1f}倍")
if efficiency_ratio > 10:
print(f" ✓ 协程内存效率显著优于线程")
elif efficiency_ratio > 5:
print(f" ✓ 协程内存效率明显优于线程")
else:
print(f" ✓ 协程内存效率优于线程")
if __name__ == '__main__':
try:
import psutil
except ImportError:
print("请先安装psutil库: pip install psutil")
exit(1)
asyncio.run(comprehensive_memory_comparison())
---
04.资源管理与错误处理
a.协程资源管理
协程的资源管理更加简单，因为只有一个线程，不存在资源竞争。使用async with可以优雅地管理资源生命周期。
b.线程同步复杂性
线程需要复杂的同步机制（锁、信号量、条件变量等）来避免数据竞争，增加了编程复杂性和出错风险。
c.代码示例
---
# 协程与线程资源管理与错误处理对比示例
import asyncio
import threading
import time
import random
from contextlib import contextmanager, asynccontextmanager
from typing import List, Dict, Any, Optional
from collections import defaultdict
# ==================== 共享资源模拟 ====================
class SharedResource:
"""共享资源模拟"""
def __init__(self, name: str, initial_value: int = 0):
self.name = name
self.value = initial_value
self.access_count = 0
self.lock = threading.Lock() # 线程版本需要锁
self.async_lock = asyncio.Lock() # 协程版本需要异步锁
def increment_sync(self, amount: int = 1) -> int:
"""同步增加值"""
with self.lock:
old_value = self.value
time.sleep(0.001) # 模拟处理时间
self.value += amount
self.access_count += 1
return self.value - old_value
async def increment_async(self, amount: int = 1) -> int:
"""异步增加值"""
async with self.async_lock:
old_value = self.value
await asyncio.sleep(0.001) # 模拟异步处理
self.value += amount
self.access_count += 1
return self.value - old_value
def get_stats(self) -> Dict:
"""获取资源统计"""
return {
"name": self.name,
"value": self.value,
"access_count": self.access_count
}
# ==================== 协程版本 ====================
@asynccontextmanager
async def async_resource_manager(resource_name: str):
"""异步资源管理器"""
resource = SharedResource(resource_name)
print(f"协程获取资源: {resource_name}")
try:
yield resource
except Exception as e:
print(f"协程资源处理异常: {e}")
raise
finally:
print(f"协程释放资源: {resource_name}")
async def coroutine_worker_with_resource(worker_id: str, resource: SharedResource,
operations: int = 10):
"""使用资源的协程工作者"""
results = []
for i in range(operations):
try:
# 模拟随机延迟
await asyncio.sleep(random.uniform(0.01, 0.05))
# 增加资源值
increment = await resource.increment_async(random.randint(1, 5))
results.append(increment)
print(f"协程{worker_id}操作{i+1}: +{increment}")
# 模拟可能的异常
if random.random() < 0.1: # 10%概率出错
raise RuntimeError(f"协程{worker_id}随机错误")
except Exception as e:
print(f"协程{worker_id}操作{i+1}出错: {e}")
# 协程中错误处理相对简单，不会影响其他协程
continue
return results
async def coroutine_resource_test():
"""协程资源管理测试"""
print("\n=== 协程资源管理测试 ===")
async with async_resource_manager("shared_counter") as resource:
# 创建多个协程工作者
worker_tasks = []
for i in range(5):
worker_id = f"coro_worker_{i+1}"
task = asyncio.create_task(
coroutine_worker_with_resource(worker_id, resource, 8)
)
worker_tasks.append(task)
start_time = time.time()
# 等待所有协程完成
all_results = await asyncio.gather(*worker_tasks, return_exceptions=True)
end_time = time.time()
# 统计结果
successful_results = []
error_count = 0
for result in all_results:
if isinstance(result, Exception):
print(f"协程任务异常: {result}")
error_count += 1
else:
successful_results.extend(result)
stats = resource.get_stats()
print(f"\n协程资源管理结果:")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
print(f"- 成功操作: {len(successful_results)}")
print(f"- 错误次数: {error_count}")
print(f"- 最终值: {stats['value']}")
print(f"- 总访问次数: {stats['access_count']}")
return stats
# ==================== 线程版本 ====================
@contextmanager
def sync_resource_manager(resource_name: str):
"""同步资源管理器"""
resource = SharedResource(resource_name)
print(f"线程获取资源: {resource_name}")
try:
yield resource
except Exception as e:
print(f"线程资源处理异常: {e}")
raise
finally:
print(f"线程释放资源: {resource_name}")
def thread_worker_with_resource(worker_id: str, resource: SharedResource,
operations: int = 10, results_list: List = None):
"""使用资源的线程工作者"""
if results_list is None:
results_list = []
try:
for i in range(operations):
try:
# 模拟随机延迟
time.sleep(random.uniform(0.01, 0.05))
# 增加资源值
increment = resource.increment_sync(random.randint(1, 5))
results_list.append(increment)
print(f"线程{worker_id}操作{i+1}: +{increment}")
# 模拟可能的异常
if random.random() < 0.1: # 10%概率出错
raise RuntimeError(f"线程{worker_id}随机错误")
except Exception as e:
print(f"线程{worker_id}操作{i+1}出错: {e}")
# 线程中的错误处理需要特别小心
continue
except Exception as e:
print(f"线程{worker_id}严重错误: {e}")
def thread_resource_test():
"""线程资源管理测试"""
print("\n=== 线程资源管理测试 ===")
with sync_resource_manager("shared_counter") as resource:
threads = []
results_list = []
# 创建多个线程工作者
for i in range(5):
worker_id = f"thread_worker_{i+1}"
thread = threading.Thread(
target=thread_worker_with_resource,
args=(worker_id, resource, 8, results_list)
)
threads.append(thread)
start_time = time.time()
# 启动所有线程
for thread in threads:
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
end_time = time.time()
stats = resource.get_stats()
print(f"\n线程资源管理结果:")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
print(f"- 成功操作: {len(results_list)}")
print(f"- 最终值: {stats['value']}")
print(f"- 总访问次数: {stats['access_count']}")
return stats
# ==================== 复杂场景对比 ====================
async def complex_resource_scenario_coroutine():
"""复杂资源场景 - 协程版本"""
print("\n=== 复杂资源场景 - 协程版本 ===")
# 创建多个共享资源
resources = {
"counter1": SharedResource("counter1"),
"counter2": SharedResource("counter2"),
"counter3": SharedResource("counter3")
}
async def complex_worker(worker_id: str, resource_names: List[str]):
"""复杂工作者 - 协程版本"""
results = {}
for resource_name in resource_names:
try:
resource = resources[resource_name]
operations = []
# 对每个资源执行多次操作
for i in range(3):
await asyncio.sleep(random.uniform(0.01, 0.03))
increment = await resource.increment_async(random.randint(1, 3))
operations.append(increment)
results[resource_name] = operations
print(f"协程{worker_id}完成{resource_name}操作")
except Exception as e:
print(f"协程{worker_id}处理{resource_name}时出错: {e}")
results[resource_name] = []
return results
# 启动多个复杂工作者
worker_tasks = []
resource_groups = [
["counter1", "counter2"],
["counter2", "counter3"],
["counter1", "counter3"],
["counter1", "counter2", "counter3"]
]
for i, group in enumerate(resource_groups):
worker_id = f"complex_coro_{i+1}"
task = asyncio.create_task(complex_worker(worker_id, group))
worker_tasks.append(task)
start_time = time.time()
all_results = await asyncio.gather(*worker_tasks, return_exceptions=True)
end_time = time.time()
# 统计最终结果
final_stats = {name: resource.get_stats() for name, resource in resources.items()}
print(f"\n复杂协程场景结果:")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
for name, stats in final_stats.items():
print(f"- {name}: 值={stats['value']}, 访问次数={stats['access_count']}")
return final_stats
def complex_resource_scenario_thread():
"""复杂资源场景 - 线程版本"""
print("\n=== 复杂资源场景 - 线程版本 ===")
# 创建多个共享资源
resources = {
"counter1": SharedResource("counter1"),
"counter2": SharedResource("counter2"),
"counter3": SharedResource("counter3")
}
def complex_worker_sync(worker_id: str, resource_names: List[str],
results_dict: Dict):
"""复杂工作者 - 线程版本"""
worker_results = {}
for resource_name in resource_names:
try:
resource = resources[resource_name]
operations = []
# 对每个资源执行多次操作
for i in range(3):
time.sleep(random.uniform(0.01, 0.03))
increment = resource.increment_sync(random.randint(1, 3))
operations.append(increment)
worker_results[resource_name] = operations
print(f"线程{worker_id}完成{resource_name}操作")
except Exception as e:
print(f"线程{worker_id}处理{resource_name}时出错: {e}")
worker_results[resource_name] = []
results_dict[worker_id] = worker_results
# 启动多个复杂工作者
threads = []
results_dict = {}
resource_groups = [
["counter1", "counter2"],
["counter2", "counter3"],
["counter1", "counter3"],
["counter1", "counter2", "counter3"]
]
for i, group in enumerate(resource_groups):
worker_id = f"complex_thread_{i+1}"
thread = threading.Thread(
target=complex_worker_sync,
args=(worker_id, group, results_dict)
)
threads.append(thread)
start_time = time.time()
# 启动所有线程
for thread in threads:
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
end_time = time.time()
# 统计最终结果
final_stats = {name: resource.get_stats() for name, resource in resources.items()}
print(f"\n复杂线程场景结果:")
print(f"- 执行时间: {end_time - start_time:.3f}秒")
for name, stats in final_stats.items():
print(f"- {name}: 值={stats['value']}, 访问次数={stats['access_count']}")
return final_stats
async def comprehensive_comparison():
"""综合对比测试"""
print("=== 资源管理与错误处理综合对比 ===")
# 简单资源测试
coro_stats = await coroutine_resource_test()
thread_stats = thread_resource_test()
# 复杂资源测试
coro_complex = await complex_resource_scenario_coroutine()
thread_complex = complex_resource_scenario_thread()
print(f"\n{'='*60}")
print("资源管理对比总结")
print(f"{'='*60}")
print(f"\n简单场景对比:")
print(f"协程 - 最终值: {coro_stats['value']}, 访问次数: {coro_stats['access_count']}")
print(f"线程 - 最终值: {thread_stats['value']}, 访问次数: {thread_stats['access_count']}")
print(f"\n复杂场景对比:")
total_coro_value = sum(stats['value'] for stats in coro_complex.values())
total_thread_value = sum(stats['value'] for stats in thread_complex.values())
total_coro_access = sum(stats['access_count'] for stats in coro_complex.values())
total_thread_access = sum(stats['access_count'] for stats in thread_complex.values())
print(f"协程 - 总值: {total_coro_value}, 总访问: {total_coro_access}")
print(f"线程 - 总值: {total_thread_value}, 总访问: {total_thread_access}")
if __name__ == '__main__':
asyncio.run(comprehensive_comparison())
---
05.适用场景分析
a.I/O密集型任务
协程特别适合I/O密集型任务（网络请求、文件操作、数据库查询等），因为可以在等待I/O时切换到其他任务，提高CPU利用率。
b.CPU密集型任务
线程在CPU密集型任务中可能更有优势，因为可以利用多核处理器实现真正的并行计算。
c.代码示例
---
# 协程与线程适用场景对比示例
import asyncio
import threading
import time
import random
import math
from concurrent.futures import ThreadPoolExecutor
from typing import List, Dict, Any, Callable
# ==================== I/O密集型任务测试 ====================
async def io_intensive_coroutine(task_id: int, io_operations: int = 5) -> Dict:
"""I/O密集型协程任务"""
start_time = time.time()
total_bytes = 0
for i in range(io_operations):
# 模拟网络请求或文件读取
await asyncio.sleep(random.uniform(0.1, 0.3)) # I/O等待
bytes_read = random.randint(100, 1000)
total_bytes += bytes_read
print(f"协程I/O任务{task_id}: 完成第{i+1}次I/O操作，读取{bytes_read}字节")
end_time = time.time()
return {
"task_id": task_id,
"model": "协程",
"type": "I/O密集",
"total_bytes": total_bytes,
"execution_time": end_time - start_time,
"io_operations": io_operations
}
def io_intensive_thread(task_id: int, io_operations: int = 5, results: List = None):
"""I/O密集型线程任务"""
if results is None:
results = []
start_time = time.time()
total_bytes = 0
for i in range(io_operations):
# 模拟网络请求或文件读取（使用time.sleep阻塞）
time.sleep(random.uniform(0.1, 0.3)) # I/O等待
bytes_read = random.randint(100, 1000)
total_bytes += bytes_read
print(f"线程I/O任务{task_id}: 完成第{i+1}次I/O操作，读取{bytes_read}字节")
end_time = time.time()
result = {
"task_id": task_id,
"model": "线程",
"type": "I/O密集",
"total_bytes": total_bytes,
"execution_time": end_time - start_time,
"io_operations": io_operations
}
results.append(result)
async def test_io_intensive_tasks(task_count: int):
"""测试I/O密集型任务"""
print(f"\n=== I/O密集型任务测试：{task_count}个任务 ===")
# 协程版本
print("\n协程版本执行I/O密集型任务:")
start_time = time.time()
coro_tasks = [io_intensive_coroutine(i+1, io_operations=4) for i in range(task_count)]
coro_results = await asyncio.gather(*coro_tasks)
coro_total_time = time.time() - start_time
coro_total_bytes = sum(result["total_bytes"] for result in coro_results)
print(f"协程版本完成: 总耗时{coro_total_time:.3f}秒，总处理字节{coro_total_bytes:,}")
# 等待系统稳定
await asyncio.sleep(1)
# 线程版本
print("\n线程版本执行I/O密集型任务:")
start_time = time.time()
thread_results = []
threads = []
for i in range(task_count):
thread = threading.Thread(
target=io_intensive_thread,
args=(i+1, 4, thread_results)
)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
thread_total_time = time.time() - start_time
thread_total_bytes = sum(result["total_bytes"] for result in thread_results)
print(f"线程版本完成: 总耗时{thread_total_time:.3f}秒，总处理字节{thread_total_bytes:,}")
# 性能对比
if thread_total_time > 0:
speedup = thread_total_time / coro_total_time
print(f"\nI/O密集型任务性能对比:")
print(f"- 协程优势: {speedup:.2f}倍")
print(f"- 协程更适合I/O密集型任务: {'✓' if speedup > 1.5 else '✗'}")
return {
"coroutine": {"time": coro_total_time, "bytes": coro_total_bytes},
"thread": {"time": thread_total_time, "bytes": thread_total_bytes},
"speedup": speedup if thread_total_time > 0 else 0
}
# ==================== CPU密集型任务测试 ====================
def cpu_intensive_calculation(n: int) -> int:
"""CPU密集型计算"""
# 计算素数的数量
count = 0
for num in range(2, n + 1):
is_prime = True
for i in range(2, int(math.sqrt(num)) + 1):
if num % i == 0:
is_prime = False
break
if is_prime:
count += 1
return count
async def cpu_intensive_coroutine(task_id: int, calculation_size: int = 10000) -> Dict:
"""CPU密集型协程任务"""
start_time = time.time()
print(f"协程CPU任务{task_id}: 开始计算，大小{calculation_size}")
# CPU密集型计算（会阻塞事件循环）
result = cpu_intensive_calculation(calculation_size)
end_time = time.time()
return {
"task_id": task_id,
"model": "协程",
"type": "CPU密集",
"result": result,
"calculation_size": calculation_size,
"execution_time": end_time - start_time
}
def cpu_intensive_thread(task_id: int, calculation_size: int = 10000, results: List = None):
"""CPU密集型线程任务"""
if results is None:
results = []
start_time = time.time()
print(f"线程CPU任务{task_id}: 开始计算，大小{calculation_size}")
# CPU密集型计算
result = cpu_intensive_calculation(calculation_size)
end_time = time.time()
result_dict = {
"task_id": task_id,
"model": "线程",
"type": "CPU密集",
"result": result,
"calculation_size": calculation_size,
"execution_time": end_time - start_time
}
results.append(result_dict)
async def test_cpu_intensive_tasks(task_count: int):
"""测试CPU密集型任务"""
print(f"\n=== CPU密集型任务测试：{task_count}个任务 ===")
# 协程版本
print("\n协程版本执行CPU密集型任务:")
start_time = time.time()
coro_tasks = [cpu_intensive_coroutine(i+1, calculation_size=8000) for i in range(task_count)]
coro_results = await asyncio.gather(*coro_tasks)
coro_total_time = time.time() - start_time
coro_total_result = sum(result["result"] for result in coro_results)
print(f"协程版本完成: 总耗时{coro_total_time:.3f}秒，总计算结果{coro_total_result:,}")
# 等待系统稳定
await asyncio.sleep(1)
# 线程版本
print("\n线程版本执行CPU密集型任务:")
start_time = time.time()
thread_results = []
threads = []
for i in range(task_count):
thread = threading.Thread(
target=cpu_intensive_thread,
args=(i+1, 8000, thread_results)
)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
thread_total_time = time.time() - start_time
thread_total_result = sum(result["result"] for result in thread_results)
print(f"线程版本完成: 总耗时{thread_total_time:.3f}秒，总计算结果{thread_total_result:,}")
# 性能对比
if coro_total_time > 0:
speedup = coro_total_time / thread_total_time
print(f"\nCPU密集型任务性能对比:")
print(f"- 线程优势: {speedup:.2f}倍")
print(f"- 线程更适合CPU密集型任务: {'✓' if speedup > 1.2 else '✗'}")
return {
"coroutine": {"time": coro_total_time, "result": coro_total_result},
"thread": {"time": thread_total_time, "result": thread_total_result},
"speedup": speedup if coro_total_time > 0 else 0
}
# ==================== 混合型任务测试 ====================
async def mixed_workload_coroutine(task_id: int, io_ops: int = 3, cpu_size: int = 5000) -> Dict:
"""混合工作负载协程任务"""
start_time = time.time()
total_bytes = 0
# I/O操作阶段
for i in range(io_ops):
await asyncio.sleep(random.uniform(0.05, 0.15))
bytes_read = random.randint(50, 200)
total_bytes += bytes_read
# CPU计算阶段
prime_count = cpu_intensive_calculation(cpu_size)
# 更多I/O操作
for i in range(io_ops):
await asyncio.sleep(random.uniform(0.05, 0.15))
bytes_read = random.randint(50, 200)
total_bytes += bytes_read
end_time = time.time()
return {
"task_id": task_id,
"model": "协程",
"type": "混合负载",
"total_bytes": total_bytes,
"prime_count": prime_count,
"execution_time": end_time - start_time
}
def mixed_workload_thread(task_id: int, io_ops: int = 3, cpu_size: int = 5000, results: List = None):
"""混合工作负载线程任务"""
if results is None:
results = []
start_time = time.time()
total_bytes = 0
# I/O操作阶段
for i in range(io_ops):
time.sleep(random.uniform(0.05, 0.15))
bytes_read = random.randint(50, 200)
total_bytes += bytes_read
# CPU计算阶段
prime_count = cpu_intensive_calculation(cpu_size)
# 更多I/O操作
for i in range(io_ops):
time.sleep(random.uniform(0.05, 0.15))
bytes_read = random.randint(50, 200)
total_bytes += bytes_read
end_time = time.time()
result_dict = {
"task_id": task_id,
"model": "线程",
"type": "混合负载",
"total_bytes": total_bytes,
"prime_count": prime_count,
"execution_time": end_time - start_time
}
results.append(result_dict)
async def test_mixed_workload_tasks(task_count: int):
"""测试混合工作负载任务"""
print(f"\n=== 混合工作负载测试：{task_count}个任务 ===")
# 协程版本
print("\n协程版本执行混合工作负载:")
start_time = time.time()
coro_tasks = [mixed_workload_coroutine(i+1, io_ops=2, cpu_size=3000) for i in range(task_count)]
coro_results = await asyncio.gather(*coro_tasks)
coro_total_time = time.time() - start_time
coro_total_bytes = sum(result["total_bytes"] for result in coro_results)
coro_total_primes = sum(result["prime_count"] for result in coro_results)
print(f"协程版本完成: 总耗时{coro_total_time:.3f}秒，I/O字节{coro_total_bytes:,}，素数{coro_total_primes:,}")
# 等待系统稳定
await asyncio.sleep(1)
# 线程版本
print("\n线程版本执行混合工作负载:")
start_time = time.time()
thread_results = []
threads = []
for i in range(task_count):
thread = threading.Thread(
target=mixed_workload_thread,
args=(i+1, 2, 3000, thread_results)
)
threads.append(thread)
thread.start()
for thread in threads:
thread.join()
thread_total_time = time.time() - start_time
thread_total_bytes = sum(result["total_bytes"] for result in thread_results)
thread_total_primes = sum(result["prime_count"] for result in thread_results)
print(f"线程版本完成: 总耗时{thread_total_time:.3f}秒，I/O字节{thread_total_bytes:,}，素数{thread_total_primes:,}")
# 性能对比
if thread_total_time > 0:
speedup = thread_total_time / coro_total_time
print(f"\n混合工作负载性能对比:")
print(f"- 协程优势: {speedup:.2f}倍")
print(f"- 协程更适合混合工作负载: {'✓' if speedup > 1.2 else '✗'}")
return {
"coroutine": {"time": coro_total_time, "bytes": coro_total_bytes, "primes": coro_total_primes},
"thread": {"time": thread_total_time, "bytes": thread_total_bytes, "primes": thread_total_primes},
"speedup": speedup if thread_total_time > 0 else 0
}
# ==================== 适用场景总结 ====================
async def scenario_recommendation_system():
"""场景推荐系统"""
print("\n=== 场景推荐系统 ===")
scenarios = [
{
"name": "Web服务器API调用",
"characteristics": ["大量网络请求", "频繁I/O等待", "并发连接多"],
"recommended": "协程",
"reason": "I/O密集型，协程效率高"
},
{
"name": "图像处理",
"characteristics": ["CPU密集计算", "大量数值运算", "可并行化"],
"recommended": "线程",
"reason": "CPU密集型，线程可利用多核"
},
{
"name": "文件批处理",
"characteristics": ["大量文件读写", "混合I/O和CPU", "任务量大"],
"recommended": "协程",
"reason": "主要是I/O操作，协程更适合"
},
{
"name": "实时数据分析",
"characteristics": ["混合工作负载", "需要快速响应", "数据流处理"],
"recommended": "协程",
"reason": "快速响应和高并发需求"
},
{
"name": "科学计算",
"characteristics": ["纯CPU计算", "计算密集", "需要并行"],
"recommended": "线程/进程",
"reason": "CPU密集，需要真正的并行"
}
]
print("\n推荐场景:")
for scenario in scenarios:
print(f"\n场景: {scenario['name']}")
print(f" 特征: {', '.join(scenario['characteristics'])}")
print(f" 推荐: {scenario['recommended']}")
print(f" 原因: {scenario['reason']}")
async def run_comprehensive_scenario_analysis():
"""运行综合场景分析"""
print("=== 协程与线程适用场景综合分析 ===")
# 测试不同场景
io_result = await test_io_intensive_tasks(6)
await asyncio.sleep(1)
cpu_result = await test_cpu_intensive_tasks(4)
await asyncio.sleep(1)
mixed_result = await test_mixed_workload_tasks(5)
await asyncio.sleep(1)
# 场景推荐
await scenario_recommendation_system()
# 总结报告
print(f"\n{'='*60}")
print("适用场景分析总结")
print(f"{'='*60}")
print(f"\n性能测试结果:")
print(f"I/O密集型 - 协程优势: {io_result['speedup']:.2f}倍")
print(f"CPU密集型 - 线程优势: {cpu_result['speedup']:.2f}倍")
print(f"混合负载 - 协程优势: {mixed_result['speedup']:.2f}倍")
print(f"\n选择建议:")
print(f"✓ I/O密集型任务 → 优先选择协程")
print(f"✓ CPU密集型任务 → 优先选择线程/进程")
print(f"✓ 混合工作负载 → 协程通常更优")
print(f"✓ 高并发需求 → 协程更适合")
print(f"✓ 多核利用需求 → 线程/进程更适合")
if __name__ == '__main__':
asyncio.run(run_comprehensive_scenario_analysis())
---
7.4 事件循环
01.事件循环基础
a.事件循环概念
事件循环是asyncio的核心机制，负责调度和执行异步任务。它维护一个任务队列，在一个循环中不断检查并执行就绪的任务，处理I/O事件和定时器。
b.工作原理
事件循环通过单线程实现并发，当遇到I/O操作时不会阻塞，而是注册回调函数并继续执行其他任务，当I/O完成时再回到原任务继续执行。
02.事件循环生命周期
a.创建与运行
asyncio提供了多种创建和运行事件循环的方式，包括asyncio.run()、loop.run_until_complete()等高级接口，以及更底层的loop.create_task()等方法。
b.代码示例
---
# 事件循环基础生命周期示例
import asyncio
import time
from typing import List, Dict, Any
class EventLoopLifecycleManager:
"""事件循环生命周期管理器"""
def __init__(self):
self.loop = None
self.tasks = []
self.start_time = None
self.end_time = None
async def setup_coroutine(self):
"""设置阶段的协程"""
print("事件循环设置阶段")
await asyncio.sleep(0.1)
print("事件循环设置完成")
async def main_coroutine(self):
"""主要业务协程"""
print("事件循环主要执行阶段")
# 模拟业务逻辑
tasks = [
asyncio.create_task(self.business_task(f"Task_{i+1}", i+1))
for i in range(3)
]
results = await asyncio.gather(*tasks)
print(f"所有业务任务完成: {results}")
return results
async def business_task(self, name: str, duration: int) -> str:
"""模拟业务任务"""
print(f"开始执行 {name}")
await asyncio.sleep(duration * 0.1)
result = f"{name} 完成"
print(result)
return result
async def cleanup_coroutine(self):
"""清理阶段的协程"""
print("事件循环清理阶段")
await asyncio.sleep(0.1)
print("事件循环清理完成")
async def basic_lifecycle_demo():
"""基础生命周期演示"""
print("=== 事件循环基础生命周期演示 ===")
manager = EventLoopLifecycleManager()
# asyncio.run()会自动管理整个生命周期
async def complete_lifecycle():
await manager.setup_coroutine()
results = await manager.main_coroutine()
await manager.cleanup_coroutine()
return results
start_time = time.time()
results = await complete_lifecycle()
end_time = time.time()
print(f"\n完整生命周期耗时: {end_time - start_time:.3f}秒")
print(f"业务结果: {results}")
# 手动事件循环管理
def manual_event_loop_management():
"""手动管理事件循环"""
print("\n=== 手动事件循环管理演示 ===")
manager = EventLoopLifecycleManager()
# 获取当前事件循环
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
start_time = time.time()
# 手动运行各个阶段
print("手动启动事件循环")
loop.run_until_complete(manager.setup_coroutine())
# 运行主要业务逻辑
results = loop.run_until_complete(manager.main_coroutine())
# 清理阶段
loop.run_until_complete(manager.cleanup_coroutine())
end_time = time.time()
print(f"\n手动管理耗时: {end_time - start_time:.3f}秒")
print(f"业务结果: {results}")
finally:
# 确保关闭事件循环
loop.close()
print("事件循环已关闭")
# 嵌套事件循环示例
async def nested_event_loop_demo():
"""嵌套事件循环演示"""
print("\n=== 嵌套事件循环演示 ===")
async def outer_task():
print("外部任务开始")
# 创建内部任务
async def inner_task():
print(" 内部任务开始")
await asyncio.sleep(0.5)
print(" 内部任务完成")
return "内部结果"
# 使用asyncio.gather运行内部任务
inner_result = await asyncio.gather(
inner_task(),
asyncio.create_task(inner_task())
)
print(f"外部任务接收到内部结果: {inner_result}")
return "外部结果"
result = await outer_task()
print(f"嵌套任务最终结果: {result}")
if __name__ == '__main__':
asyncio.run(basic_lifecycle_demo())
manual_event_loop_management()
asyncio.run(nested_event_loop_demo())
---
03.事件循环调度机制
a.任务调度原理
事件循环使用任务队列来管理协程，当协程遇到await时会暂停执行并将控制权交还给事件循环，事件循环会选择下一个就绪的任务执行。
b.优先级与调度策略
事件循环根据任务的就绪状态和优先级进行调度，立即就绪的任务会优先执行，延迟任务会被放入定时器队列。
c.代码示例
---
# 事件循环调度机制详解示例
import asyncio
import time
from typing import List, Dict, Any, Callable
from collections import deque
class TaskScheduler:
"""任务调度器模拟"""
def __init__(self):
self.ready_queue = deque()
self.delayed_tasks = []
self.running_tasks = set()
self.completed_tasks = []
self.task_counter = 0
self.current_time = 0
def create_task(self, coro, priority: int = 0):
"""创建任务"""
task_id = self.task_counter
self.task_counter += 1
task = {
'id': task_id,
'coro': coro,
'priority': priority,
'created_at': self.current_time,
'state': 'ready'
}
self.ready_queue.append(task)
print(f"创建任务{task_id}，优先级: {priority}")
return task_id
async def simulated_io_operation(self, duration: float, task_id: int):
"""模拟I/O操作"""
print(f"任务{task_id}开始I/O操作，预计耗时{duration}秒")
# 在真实的事件循环中，这里会注册I/O回调
await asyncio.sleep(duration)
print(f"任务{task_id}I/O操作完成")
return f"任务{task_id}的I/O结果"
async def simulated_cpu_operation(self, duration: float, task_id: int):
"""模拟CPU操作"""
print(f"任务{task_id}开始CPU操作，耗时{duration}秒")
# 模拟CPU计算
start = time.time()
result = 0
while time.time() - start < duration:
result += 1
print(f"任务{task_id}CPU操作完成")
return f"任务{task_id}的CPU计算结果: {result}"
def schedule_by_priority(self):
"""按优先级调度任务"""
if not self.ready_queue:
return None
# 简单的优先级调度：优先级数字越小优先级越高
self.ready_queue = deque(sorted(self.ready_queue, key=lambda x: x['priority']))
return self.ready_queue.popleft()
async def scheduling_principle_demo():
"""调度原理演示"""
print("=== 事件循环调度原理演示 ===")
scheduler = TaskScheduler()
# 创建不同优先级的任务
tasks = []
# 高优先级任务
high_priority_task = asyncio.create_task(
scheduler.simulated_cpu_operation(0.2, 1)
)
scheduler.create_task(high_priority_task, priority=1)
# 中优先级任务
medium_priority_task = asyncio.create_task(
scheduler.simulated_io_operation(0.3, 2)
)
scheduler.create_task(medium_priority_task, priority=5)
# 低优先级任务
low_priority_task = asyncio.create_task(
scheduler.simulated_cpu_operation(0.1, 3)
)
scheduler.create_task(low_priority_task, priority=10)
# 演示事件循环如何调度这些任务
print("\n事件循环开始调度...")
start_time = time.time()
# 并发执行所有任务（事件循环会自动调度）
results = await asyncio.gather(
high_priority_task,
medium_priority_task,
low_priority_task
)
end_time = time.time()
print(f"\n调度完成，总耗时: {end_time - start_time:.3f}秒")
print(f"任务结果: {results}")
async def task_state_transitions():
"""任务状态转换演示"""
print("\n=== 任务状态转换演示 ===")
class TaskStateMachine:
"""任务状态机"""
def __init__(self, task_id: int):
self.task_id = task_id
self.state = "created"
self.state_history = []
def transition_to(self, new_state: str):
"""状态转换"""
old_state = self.state
self.state = new_state
self.state_history.append((time.time(), old_state, new_state))
print(f"任务{self.task_id}: {old_state} -> {new_state}")
async def execute(self):
"""执行任务"""
self.transition_to("ready")
# 阶段1：I/O操作
self.transition_to("running_io")
await asyncio.sleep(0.2)
self.transition_to("waiting")
# 阶段2：CPU操作
self.transition_to("running_cpu")
await asyncio.sleep(0.1)
self.transition_to("completed")
return f"任务{self.task_id}完成"
# 创建多个任务
tasks = [TaskStateMachine(i+1) for i in range(3)]
# 并发执行
start_time = time.time()
task_coroutines = [task.execute() for task in tasks]
results = await asyncio.gather(*task_coroutines)
end_time = time.time()
# 分析状态转换
print(f"\n状态转换分析:")
print(f"总执行时间: {end_time - start_time:.3f}秒")
for task in tasks:
print(f"\n任务{task.task_id}状态历史:")
for timestamp, old_state, new_state in task.state_history:
elapsed = timestamp - start_time
print(f" {elapsed:.3f}s: {old_state} -> {new_state}")
async def event_loop_internals_demo():
"""事件循环内部机制演示"""
print("\n=== 事件循环内部机制演示 ===")
# 获取当前事件循环
loop = asyncio.get_running_loop()
print(f"当前事件循环: {loop}")
print(f"事件循环是否运行: {loop.is_running()}")
print(f"事件循环是否关闭: {loop.is_closed()}")
# 获取事件循环时间
current_time = loop.time()
print(f"事件循环时间: {current_time:.3f}")
# 演示call_soon
def sync_callback():
print("同步回调函数执行")
loop.call_soon(sync_callback)
# 演示call_later
def delayed_callback():
print("延迟回调函数执行")
loop.call_later(0.5, delayed_callback)
# 创建一些任务来观察调度
async def task_with_monitoring(task_id: int):
print(f"任务{task_id}开始")
# 在任务中检查事件循环状态
current_loop = asyncio.get_running_loop()
print(f"任务{task_id}中的事件循环: {current_loop}")
await asyncio.sleep(0.2)
print(f"任务{task_id}完成")
return task_id
# 创建任务
tasks = [
asyncio.create_task(task_with_monitoring(i+1))
for i in range(3)
]
# 等待所有任务完成
results = await asyncio.gather(*tasks)
# 给延迟回调一些时间执行
await asyncio.sleep(0.6)
print(f"所有任务完成: {results}")
async def custom_task_factory_demo():
"""自定义任务工厂演示"""
print("\n=== 自定义任务工厂演示 ===")
class CustomTaskFactory:
"""自定义任务工厂"""
def __init__(self):
self.created_tasks = 0
self.task_priorities = {}
def create_task(self, coro, priority: int = 0):
"""创建带优先级的任务"""
self.created_tasks += 1
task_id = self.created_tasks
self.task_priorities[task_id] = priority
# 包装原始协程
async def wrapped_coro():
print(f"任务{task_id}开始执行，优先级: {priority}")
try:
result = await coro
print(f"任务{task_id}成功完成")
return result
except Exception as e:
print(f"任务{task_id}执行失败: {e}")
raise
return asyncio.create_task(wrapped_coro())
# 使用自定义工厂
factory = CustomTaskFactory()
async def sample_work(task_name: str, duration: float):
"""示例工作任务"""
await asyncio.sleep(duration)
return f"{task_name}结果"
# 创建不同优先级的任务
tasks = [
factory.create_task(sample_work("高优先级", 0.2), priority=1),
factory.create_task(sample_work("中优先级", 0.1), priority=5),
factory.create_task(sample_work("低优先级", 0.3), priority=10)
]
print(f"工厂创建了{factory.created_tasks}个任务")
# 并发执行
results = await asyncio.gather(*tasks)
print(f"任务执行结果: {results}")
if __name__ == '__main__':
asyncio.run(scheduling_principle_demo())
asyncio.run(task_state_transitions())
asyncio.run(event_loop_internals_demo())
asyncio.run(custom_task_factory_demo())
---
04.高级事件循环操作
a.回调与定时器
事件循环支持各种回调和定时器操作，包括call_soon、call_later、call_at等方法，可以精确控制任务的执行时机。
b.子进程集成
事件循环可以与子进程集成，实现异步的进程管理，避免阻塞主线程。
c.代码示例
---
# 高级事件循环操作示例
import asyncio
import time
import subprocess
import signal
from typing import List, Dict, Any, Callable
from datetime import datetime, timedelta
class AdvancedEventLoopManager:
"""高级事件循环管理器"""
def __init__(self):
self.loop = None
self.callbacks = []
self.timers = []
self.subprocesses = []
self.signals = []
async def setup_advanced_loop(self):
"""设置高级事件循环"""
self.loop = asyncio.get_running_loop()
print(f"高级事件循环设置完成: {self.loop}")
# 设置信号处理
self.setup_signal_handlers()
# 设置定时器
self.setup_timers()
# 设置回调
self.setup_callbacks()
def setup_signal_handlers(self):
"""设置信号处理器"""
def signal_handler(signum, frame):
print(f"接收到信号: {signum}")
# 这里可以执行优雅关闭等操作
# 注意：在Windows上某些信号可能不可用
try:
self.loop.add_signal_handler(signal.SIGINT, signal_handler)
self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
print("信号处理器设置完成")
except (AttributeError, NotImplementedError):
print("当前平台不支持信号处理")
def setup_timers(self):
"""设置定时器"""
# 立即执行的回调
self.loop.call_soon(self.immediate_callback, "立即回调")
# 延迟执行的回调
self.loop.call_later(1.0, self.delayed_callback, "1秒延迟回调")
self.loop.call_later(2.0, self.delayed_callback, "2秒延迟回调")
# 在指定时间执行的回调
target_time = self.loop.time() + 3.0
self.loop.call_at(target_time, self.scheduled_callback, "定时回调")
print("定时器设置完成")
def setup_callbacks(self):
"""设置回调函数"""
# 注册到下次循环迭代时执行的回调
self.loop.call_soon_threadsafe(self.thread_safe_callback, "线程安全回调")
print("回调函数设置完成")
def immediate_callback(self, message: str):
"""立即回调"""
print(f"立即回调执行: {message} (时间: {datetime.now().strftime('%H:%M:%S.%f')[:-3]})")
def delayed_callback(self, message: str):
"""延迟回调"""
print(f"延迟回调执行: {message} (时间: {datetime.now().strftime('%H:%M:%S.%f')[:-3]})")
def scheduled_callback(self, message: str):
"""定时回调"""
print(f"定时回调执行: {message} (时间: {datetime.now().strftime('%H:%M:%S.%f')[:-3]})")
def thread_safe_callback(self, message: str):
"""线程安全回调"""
print(f"线程安全回调执行: {message} (时间: {datetime.now().strftime('%H:%M:%S.%f')[:-3]})")
async def callback_and_timer_demo():
"""回调和定时器演示"""
print("=== 回调和定时器演示 ===")
manager = AdvancedEventLoopManager()
await manager.setup_advanced_loop()
# 给回调一些执行时间
await asyncio.sleep(4)
print("回调和定时器演示完成")
# 子进程集成示例
async def subprocess_integration_demo():
"""子进程集成演示"""
print("\n=== 子进程集成演示 ===")
loop = asyncio.get_running_loop()
# 创建一些临时文件用于演示
test_files = ["test1.txt", "test2.txt"]
try:
# 创建测试文件
for filename in test_files:
with open(filename, 'w') as f:
f.write(f"这是测试文件 {filename} 的内容\n")
f.write(f"创建时间: {datetime.now()}\n")
print("测试文件创建完成")
# 异步执行系统命令
commands = [
# 列出当前目录文件
["dir" if os.name == 'nt' else "ls", "-la"],
# 统计文件行数
["wc", "-l"] + test_files if os.name != 'nt' else ["type", test_files[0]],
# 查找包含特定内容的行
["grep", "测试"] + test_files if os.name != 'nt' else ["findstr", "测试", test_files[0]]
]
results = []
for i, cmd in enumerate(commands):
print(f"\n执行命令 {i+1}: {' '.join(cmd)}")
try:
# 创建子进程
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
# 等待进程完成并获取输出
stdout, stderr = await proc.communicate()
result = {
'command': cmd,
'returncode': proc.returncode,
'stdout': stdout.decode('utf-8', errors='ignore'),
'stderr': stderr.decode('utf-8', errors='ignore')
}
results.append(result)
print(f"命令完成，返回码: {proc.returncode}")
if result['stdout']:
print(f"输出: {result['stdout'][:200]}...") # 只显示前200字符
if result['stderr']:
print(f"错误: {result['stderr'][:200]}...")
except Exception as e:
print(f"命令执行失败: {e}")
# 并行执行多个命令
print(f"\n=== 并行执行多个命令 ===")
parallel_commands = [
["echo", "命令1"],
["echo", "命令2"],
["echo", "命令3"]
] if os.name != 'nt' else [
["cmd", "/c", "echo 命令1"],
["cmd", "/c", "echo 命令2"],
["cmd", "/c", "echo 命令3"]
]
async def run_command(cmd):
"""运行单个命令"""
proc = await asyncio.create_subprocess_exec(
*cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await proc.communicate()
return {
'command': cmd,
'stdout': stdout.decode('utf-8', errors='ignore').strip(),
'returncode': proc.returncode
}
# 并发执行
parallel_tasks = [run_command(cmd) for cmd in parallel_commands]
parallel_results = await asyncio.gather(*parallel_tasks)
print("并行执行结果:")
for result in parallel_results:
print(f" {' '.join(result['command'])}: {result['stdout']}")
# 超时控制的子进程
print(f"\n=== 超时控制的子进程 ===")
try:
# 执行一个可能耗时很长的命令，设置超时
long_cmd = ["sleep", "5"] if os.name != 'nt' else ["timeout", "5"]
proc = await asyncio.create_subprocess_exec(
*long_cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
# 设置2秒超时
try:
stdout, stderr = await asyncio.wait_for(
proc.communicate(),
timeout=2.0
)
print("命令在超时前完成")
except asyncio.TimeoutError:
print("命令执行超时，终止进程")
proc.terminate()
await proc.wait()
except Exception as e:
print(f"超时控制示例失败: {e}")
finally:
# 清理测试文件
for filename in test_files:
try:
os.remove(filename)
print(f"删除测试文件: {filename}")
except:
pass
# 自定义事件循环策略
async def custom_event_loop_policy_demo():
"""自定义事件循环策略演示"""
print("\n=== 自定义事件循环策略演示 ===")
# 获取默认策略
default_policy = asyncio.get_event_loop_policy()
print(f"默认事件循环策略: {default_policy}")
# 获取当前事件循环
current_loop = asyncio.get_running_loop()
print(f"当前事件循环: {current_loop}")
print(f"事件循环调试状态: {current_loop.get_debug()}")
# 设置调试模式
current_loop.set_debug(True)
print(f"调试模式已启用")
# 创建自定义的事件循环策略
class CustomEventLoopPolicy(asyncio.DefaultEventLoopPolicy):
"""自定义事件循环策略"""
def new_event_loop(self):
loop = super().new_event_loop()
print(f"创建自定义事件循环: {loop}")
# 设置自定义属性
loop.custom_attribute = "自定义属性值"
# 设置调试模式
loop.set_debug(True)
return loop
# 注意：在实际应用中，更改策略需要谨慎
print("自定义事件循环策略演示完成")
# 事件循环性能监控
class EventLoopMonitor:
"""事件循环性能监控器"""
def __init__(self):
self.task_count = 0
self.completed_tasks = 0
self.errors = 0
self.start_time = None
self.task_durations = []
async def monitored_task(self, task_id: int, duration: float):
"""被监控的任务"""
self.task_count += 1
start_time = time.time()
try:
print(f"监控任务{task_id}开始")
# 模拟工作
await asyncio.sleep(duration)
# 模拟可能的错误
if random.random() < 0.1: # 10%错误率
raise RuntimeError(f"任务{task_id}随机错误")
task_duration = time.time() - start_time
self.task_durations.append(task_duration)
self.completed_tasks += 1
print(f"监控任务{task_id}完成，耗时: {task_duration:.3f}秒")
return f"任务{task_id}结果"
except Exception as e:
self.errors += 1
print(f"监控任务{task_id}失败: {e}")
raise
def get_statistics(self):
"""获取统计信息"""
if self.task_durations:
avg_duration = sum(self.task_durations) / len(self.task_durations)
max_duration = max(self.task_durations)
min_duration = min(self.task_durations)
else:
avg_duration = max_duration = min_duration = 0
total_time = time.time() - self.start_time if self.start_time else 0
return {
'total_tasks': self.task_count,
'completed_tasks': self.completed_tasks,
'errors': self.errors,
'success_rate': self.completed_tasks / self.task_count if self.task_count > 0 else 0,
'avg_duration': avg_duration,
'max_duration': max_duration,
'min_duration': min_duration,
'total_execution_time': total_time,
'throughput': self.completed_tasks / total_time if total_time > 0 else 0
}
async def event_loop_monitoring_demo():
"""事件循环监控演示"""
print("\n=== 事件循环监控演示 ===")
monitor = EventLoopMonitor()
monitor.start_time = time.time()
# 创建大量监控任务
tasks = []
for i in range(20):
duration = random.uniform(0.1, 0.5)
task = asyncio.create_task(monitor.monitored_task(i+1, duration))
tasks.append(task)
print(f"创建了{len(tasks)}个监控任务")
# 执行所有任务
try:
results = await asyncio.gather(*tasks, return_exceptions=True)
# 统计结果
successful_results = [r for r in results if not isinstance(r, Exception)]
failed_results = [r for r in results if isinstance(r, Exception)]
print(f"\n执行结果:")
print(f"成功任务: {len(successful_results)}")
print(f"失败任务: {len(failed_results)}")
# 获取详细统计
stats = monitor.get_statistics()
print(f"\n=== 性能统计 ===")
print(f"总任务数: {stats['total_tasks']}")
print(f"完成任务数: {stats['completed_tasks']}")
print(f"错误任务数: {stats['errors']}")
print(f"成功率: {stats['success_rate']:.2%}")
print(f"平均耗时: {stats['avg_duration']:.3f}秒")
print(f"最长耗时: {stats['max_duration']:.3f}秒")
print(f"最短耗时: {stats['min_duration']:.3f}秒")
print(f"总执行时间: {stats['total_execution_time']:.3f}秒")
print(f"吞吐量: {stats['throughput']:.2f} 任务/秒")
except Exception as e:
print(f"监控演示失败: {e}")
import os
import random
if __name__ == '__main__':
asyncio.run(callback_and_timer_demo())
asyncio.run(subprocess_integration_demo())
asyncio.run(custom_event_loop_policy_demo())
asyncio.run(event_loop_monitoring_demo())
---
05.事件循环最佳实践
a.性能优化技巧
避免阻塞事件循环、合理使用任务调度、控制并发数量、及时清理资源等是事件循环性能优化的关键。
b.错误处理与调试
正确处理异步异常、使用调试模式、合理设置超时、监控事件循环状态等有助于构建稳定的应用。
c.代码示例
---
# 事件循环最佳实践示例
import asyncio
import time
import logging
import signal
import sys
import random
from typing import List, Dict, Any, Optional
from contextlib import asynccontextmanager
# 配置日志
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class EventLoopBestPractices:
"""事件循环最佳实践示例"""
def __init__(self):
self.loop = None
self.shutdown_event = asyncio.Event()
self.active_tasks = set()
self.performance_monitor = None
async def setup_application(self):
"""应用程序设置"""
logger.info("设置应用程序")
# 获取事件循环
self.loop = asyncio.get_running_loop()
# 设置优雅关闭
self.setup_graceful_shutdown()
# 设置性能监控
self.setup_performance_monitoring()
# 设置错误处理
self.setup_error_handling()
def setup_graceful_shutdown(self):
"""设置优雅关闭"""
def signal_handler(signum, frame):
logger.info(f"接收到信号 {signum}，开始优雅关闭")
self.shutdown_event.set()
try:
for sig in (signal.SIGINT, signal.SIGTERM):
self.loop.add_signal_handler(sig, signal_handler)
logger.info("信号处理器设置完成")
except (AttributeError, NotImplementedError):
logger.warning("当前平台不支持信号处理")
def setup_performance_monitoring(self):
"""设置性能监控"""
self.performance_monitor = PerformanceMonitor()
self.performance_monitor.start()
def setup_error_handling(self):
"""设置错误处理"""
self.loop.set_exception_handler(self.handle_exception)
def handle_exception(self, loop, context):
"""全局异常处理器"""
msg = context.get("message", "未处理的异常")
exception = context.get("exception")
if exception:
logger.error(f"未处理的异常: {exception}")
else:
logger.error(f"循环错误: {msg}")
# 性能监控器
class PerformanceMonitor:
"""性能监控器"""
def __init__(self, sample_interval: float = 1.0):
self.sample_interval = sample_interval
self.monitoring = False
self.samples = []
self.start_time = None
def start(self):
"""开始监控"""
self.monitoring = True
self.start_time = time.time()
logger.info("性能监控器启动")
def stop(self):
"""停止监控"""
self.monitoring = False
logger.info("性能监控器停止")
async def collect_sample(self):
"""收集性能样本"""
while self.monitoring:
sample = {
'timestamp': time.time(),
'loop_time': asyncio.get_running_loop().time(),
}
self.samples.append(sample)
await asyncio.sleep(self.sample_interval)
def get_statistics(self) -> Dict:
"""获取性能统计"""
if not self.samples:
return {}
duration = time.time() - self.start_time
sample_count = len(self.samples)
return {
'monitoring_duration': duration,
'sample_count': sample_count,
'samples_per_second': sample_count / duration,
'last_sample': self.samples[-1] if self.samples else None
}
# 任务包装器 - 提供错误处理和监控
class MonitoredTask:
"""被监控的任务"""
def __init__(self, task_id: str, coro, timeout: Optional[float] = None):
self.task_id = task_id
self.coro = coro
self.timeout = timeout
self.start_time = None
self.end_time = None
self.exception = None
async def execute(self):
"""执行任务"""
self.start_time = time.time()
try:
logger.info(f"任务 {self.task_id} 开始执行")
# 应用超时（如果设置了）
if self.timeout:
result = await asyncio.wait_for(self.coro, timeout=self.timeout)
else:
result = await self.coro
self.end_time = time.time()
duration = self.end_time - self.start_time
logger.info(f"任务 {self.task_id} 完成，耗时: {duration:.3f}秒")
return result
except asyncio.TimeoutError:
self.exception = f"任务 {self.task_id} 执行超时 ({self.timeout}秒)"
logger.error(self.exception)
raise
except Exception as e:
self.exception = f"任务 {self.task_id} 执行失败: {e}"
logger.error(self.exception)
raise
# 最佳实践1：避免阻塞事件循环
async def non_blocking_operations_demo():
"""非阻塞操作演示"""
logger.info("=== 非阻塞操作演示 ===")
# 错误做法：阻塞操作
async def blocking_operation_wrong():
"""错误的做法：CPU密集型操作会阻塞事件循环"""
logger.warning("执行阻塞操作（错误做法）")
# 这样会阻塞整个事件循环
result = sum(i * i for i in range(1000000))
logger.info(f"阻塞操作完成: {result}")
return result
# 正确做法：使用线程池执行阻塞操作
async def blocking_operation_correct():
"""正确的做法：在线程池中执行阻塞操作"""
logger.info("执行非阻塞操作（正确做法）")
# 在线程池中执行CPU密集型操作
loop = asyncio.get_running_loop()
result = await loop.run_in_executor(None, lambda: sum(i * i for i in range(1000000)))
logger.info(f"非阻塞操作完成: {result}")
return result
# 演示对比
start_time = time.time()
# 并发执行多个任务
tasks = [
blocking_operation_correct() for _ in range(3)
]
# 错误做法演示（取消注释以看到阻塞效果）
# tasks.append(blocking_operation_wrong())
results = await asyncio.gather(*tasks)
end_time = time.time()
logger.info(f"非阻塞操作演示完成，总耗时: {end_time - start_time:.3f}秒")
# 最佳实践2：合理的任务调度
async def optimal_task_scheduling_demo():
"""最优任务调度演示"""
logger.info("=== 最优任务调度演示 ===")
# 创建不同类型的任务
async def io_task(task_id: int):
"""I/O密集型任务"""
await asyncio.sleep(random.uniform(0.1, 0.3))
logger.info(f"I/O任务 {task_id} 完成")
return f"io_result_{task_id}"
async def cpu_task(task_id: int):
"""CPU密集型任务（在线程池中执行）"""
loop = asyncio.get_running_loop()
def cpu_intensive():
# 模拟CPU计算
result = 0
for i in range(100000):
result += i * i
return result
result = await loop.run_in_executor(None, cpu_intensive)
logger.info(f"CPU任务 {task_id} 完成")
return f"cpu_result_{task_id}_{result}"
# 批量处理任务
async def batch_processing():
"""批量处理示例"""
batch_size = 5
total_tasks = 15
for batch_start in range(0, total_tasks, batch_size):
batch_end = min(batch_start + batch_size, total_tasks)
logger.info(f"处理批次 {batch_start//batch_size + 1}: 任务 {batch_start+1}-{batch_end}")
# 创建批次任务
batch_tasks = []
for i in range(batch_start, batch_end):
if i % 2 == 0:
batch_tasks.append(io_task(i))
else:
batch_tasks.append(cpu_task(i))
# 执行当前批次
batch_results = await asyncio.gather(*batch_tasks)
logger.info(f"批次完成，结果数: {len(batch_results)}")
# 批次间短暂休息
await asyncio.sleep(0.1)
# 优先级任务调度
async def priority_task_scheduling():
"""优先级任务调度"""
high_priority_tasks = [io_task(i) for i in range(1, 4)]
low_priority_tasks = [cpu_task(i) for i in range(4, 7)]
# 先执行高优先级任务
logger.info("执行高优先级任务")
high_results = await asyncio.gather(*high_priority_tasks)
# 再执行低优先级任务
logger.info("执行低优先级任务")
low_results = await asyncio.gather(*low_priority_tasks)
return high_results + low_results
# 并发控制
async def controlled_concurrency():
"""并发控制示例"""
semaphore = asyncio.Semaphore(3) # 最多同时3个任务
async def limited_task(task_id: int):
async with semaphore:
logger.info(f"受限任务 {task_id} 开始")
await asyncio.sleep(random.uniform(0.2, 0.5))
logger.info(f"受限任务 {task_id} 完成")
return f"limited_result_{task_id}"
# 创建更多任务但限制并发
tasks = [limited_task(i) for i in range(10)]
results = await asyncio.gather(*tasks)
return results
# 执行各种调度策略
await batch_processing()
await priority_task_scheduling()
await controlled_concurrency()
# 最佳实践3：资源管理和清理
@asynccontextmanager
async def managed_resource(name: str):
"""可管理的资源上下文"""
logger.info(f"获取资源: {name}")
try:
# 模拟资源初始化
await asyncio.sleep(0.1)
resource = {"name": name, "data": f"resource_data_{name}"}
yield resource
except Exception as e:
logger.error(f"资源 {name} 使用时出错: {e}")
raise
finally:
# 资源清理
logger.info(f"释放资源: {name}")
await asyncio.sleep(0.05)
async def resource_management_demo():
"""资源管理演示"""
logger.info("=== 资源管理演示 ===")
# 使用上下文管理器管理资源
async with managed_resource("database") as db_resource:
logger.info(f"使用数据库资源: {db_resource}")
async with managed_resource("cache") as cache_resource:
logger.info(f"使用缓存资源: {cache_resource}")
# 执行业务逻辑
await asyncio.sleep(0.2)
logger.info("业务逻辑执行完成")
# 最佳实践4：超时和重试机制
async def timeout_and_retry_demo():
"""超时和重试演示"""
logger.info("=== 超时和重试演示 ===")
async def unreliable_operation(success_rate: float = 0.5):
"""不可靠的操作"""
await asyncio.sleep(random.uniform(0.1, 0.3))
if random.random() < success_rate:
return "操作成功"
else:
raise RuntimeError("操作失败")
async def operation_with_retry(max_retries: int = 3, delay: float = 0.1):
"""带重试的操作"""
for attempt in range(max_retries + 1):
try:
logger.info(f"尝试第 {attempt + 1} 次")
# 设置超时
result = await asyncio.wait_for(
unreliable_operation(0.3), # 30%成功率
timeout=0.5
)
logger.info(f"操作成功: {result}")
return result
except asyncio.TimeoutError:
logger.warning(f"第 {attempt + 1} 次尝试超时")
except Exception as e:
logger.warning(f"第 {attempt + 1} 次尝试失败: {e}")
if attempt < max_retries:
await asyncio.sleep(delay * (2 ** attempt)) # 指数退避
raise RuntimeError(f"操作在 {max_retries + 1} 次尝试后仍然失败")
try:
result = await operation_with_retry()
logger.info(f"最终结果: {result}")
except Exception as e:
logger.error(f"操作最终失败: {e}")
# 完整的最佳实践应用示例
async def best_practices_application():
"""最佳实践应用示例"""
logger.info("=== 最佳实践应用示例 ===")
app = EventLoopBestPractices()
await app.setup_application()
# 启动性能监控
monitor_task = asyncio.create_task(
app.performance_monitor.collect_sample()
)
try:
# 创建被监控的任务
tasks = [
MonitoredTask("task_1", non_blocking_operations_demo()).execute(),
MonitoredTask("task_2", optimal_task_scheduling_demo()).execute(),
MonitoredTask("task_3", resource_management_demo()).execute(),
MonitoredTask("task_4", timeout_and_retry_demo()).execute(),
]
# 监控活动任务
for i, task in enumerate(tasks):
monitored_task = asyncio.create_task(task)
app.active_tasks.add(monitored_task)
monitored_task.add_done_callback(
lambda t, s=app.active_tasks: s.discard(t)
)
logger.info(f"启动了 {len(app.active_tasks)} 个任务")
# 等待任务完成或关闭信号
done, pending = await asyncio.wait(
app.active_tasks,
timeout=30.0,
return_when=asyncio.FIRST_COMPLETED
)
if app.shutdown_event.is_set():
logger.info("收到关闭信号，取消剩余任务")
for task in pending:
task.cancel()
# 等待任务取消完成
await asyncio.gather(*pending, return_exceptions=True)
except Exception as e:
logger.error(f"应用程序执行出错: {e}")
finally:
# 停止性能监控
app.performance_monitor.stop()
monitor_task.cancel()
# 获取性能统计
stats = app.performance_monitor.get_statistics()
logger.info(f"性能统计: {stats}")
if __name__ == '__main__':
# 启用调试模式
asyncio.run(best_practices_application(), debug=True)
---
8 并发模型对比
8.1 多线程模型
01.基本概念
a.线程定义
线程是操作系统能够进行运算调度的最小单位，被包含在进程之中，是进程中的实际运作单位。同一个进程中的多个线程共享进程的内存空间，包括代码段、数据段、堆等，但每个线程拥有独立的栈空间和程序计数器。
b.多线程特点
多线程是指在单个程序中同时运行多个线程的编程模型，通过线程间的并发执行提高程序的性能和响应能力。线程间切换成本较低，共享内存通信效率高，但需要注意线程安全和资源竞争问题。
c.应用场景
多线程模型适用于CPU密集型任务、需要同时处理多个请求的服务器应用、图形界面程序的事件处理等场景，能够充分利用多核CPU资源。
02.线程创建与管理
a.线程创建方式
a.功能说明
Python中使用threading.Thread类创建线程，需要指定目标函数和参数。线程创建后处于就绪状态，等待CPU调度执行。
b.代码示例
---
# Python多线程基础示例
import threading
import time
import logging
# 配置日志
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class WorkerThread(threading.Thread):
"""自定义工作线程类"""
def __init__(self, thread_id, name, counter):
super(WorkerThread, self).__init__()
self.thread_id = thread_id
self.name = name
self.counter = counter
self._stop_event = threading.Event()
def run(self):
"""线程执行的核心方法"""
logging.info(f"线程 {self.name} 开始执行，ID: {self.thread_id}")
while not self._stop_event.is_set() and self.counter < 5:
time.sleep(1)
logging.info(f"线程 {self.name} 执行中... 计数器: {self.counter}")
self.counter += 1
logging.info(f"线程 {self.name} 执行完成")
def stop(self):
"""安全停止线程"""
self._stop_event.set()
def create_worker_threads():
"""创建并管理工作线程"""
threads = []
# 创建3个工作线程
for i in range(3):
thread = WorkerThread(thread_id=i, name=f"Worker-{i}", counter=0)
threads.append(thread)
thread.start() # 启动线程
# 等待线程执行完成
for thread in threads:
thread.join()
logging.info("所有线程执行完成")
if __name__ == "__main__":
create_worker_threads()
---
b.线程池管理
a.功能说明
线程池是一种预先创建并维护一定数量线程的技术，避免频繁创建和销毁线程的开销。Python中的concurrent.futures.ThreadPoolExecutor提供了高效的线程池实现。
b.代码示例
---
# 线程池使用示例
import concurrent.futures
import time
import random
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class TaskManager:
"""任务管理器，使用线程池执行任务"""
def __init__(self, max_workers=4):
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=max_workers)
self.futures = []
def process_data(self, data):
"""模拟数据处理任务"""
task_id = data['id']
processing_time = random.uniform(1, 3)
logging.info(f"任务 {task_id} 开始处理，预计耗时 {processing_time:.2f} 秒")
# 模拟耗时操作
time.sleep(processing_time)
# 模拟可能出现的错误
if random.random() < 0.1: # 10%概率失败
raise Exception(f"任务 {task_id} 处理失败")
result = data['value'] * 2
logging.info(f"任务 {task_id} 处理完成，结果: {result}")
return {'id': task_id, 'result': result, 'processing_time': processing_time}
def submit_tasks(self, tasks):
"""批量提交任务到线程池"""
for task in tasks:
future = self.executor.submit(self.process_data, task)
future.task_id = task['id'] # 为future添加任务ID属性
self.futures.append(future)
def wait_completion(self):
"""等待所有任务完成并收集结果"""
completed_tasks = []
failed_tasks = []
for future in concurrent.futures.as_completed(self.futures):
task_id = future.task_id
try:
result = future.result(timeout=5) # 设置超时时间
completed_tasks.append(result)
except Exception as e:
logging.error(f"任务 {task_id} 执行失败: {str(e)}")
failed_tasks.append({'id': task_id, 'error': str(e)})
return completed_tasks, failed_tasks
def shutdown(self):
"""关闭线程池"""
self.executor.shutdown(wait=True)
logging.info("线程池已关闭")
def main():
"""主函数：演示线程池的使用"""
# 创建任务管理器
manager = TaskManager(max_workers=4)
# 准备测试任务
test_tasks = [
{'id': f'task_{i}', 'value': i * 10}
for i in range(1, 11)
]
logging.info(f"准备处理 {len(test_tasks)} 个任务")
# 提交任务到线程池
start_time = time.time()
manager.submit_tasks(test_tasks)
# 等待所有任务完成
completed, failed = manager.wait_completion()
end_time = time.time()
# 输出统计信息
logging.info(f"任务执行完成 - 耗时: {end_time - start_time:.2f} 秒")
logging.info(f"成功任务: {len(completed)} 个，失败任务: {len(failed)} 个")
# 显示成功任务的结果
for task in completed:
logging.info(f"任务 {task['id']}: 值={task['result']}, 耗时={task['processing_time']:.2f}s")
# 显示失败任务
for task in failed:
logging.error(f"任务 {task['id']} 失败原因: {task['error']}")
# 关闭线程池
manager.shutdown()
if __name__ == "__main__":
main()
---
03.线程同步与通信
a.锁机制
a.功能说明
锁是多线程编程中用于保护共享资源的基本同步机制，确保同一时间只有一个线程可以访问临界资源。Python提供了threading.Lock和threading.RLock等锁类型。
b.代码示例
---
# 线程锁使用示例
import threading
import time
import logging
from collections import defaultdict
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class BankAccount:
"""银行账户类，演示线程安全的重要性"""
def __init__(self, account_number, initial_balance=0):
self.account_number = account_number
self.balance = initial_balance
self.lock = threading.Lock() # 账户锁
self.transaction_history = []
def deposit(self, amount, user):
"""存款操作（线程安全）"""
with self.lock: # 使用上下文管理器自动管理锁
if amount <= 0:
raise ValueError("存款金额必须大于0")
old_balance = self.balance
time.sleep(0.01) # 模拟处理时间
self.balance += amount
# 记录交易历史
transaction = {
'type': 'deposit',
'amount': amount,
'user': user,
'old_balance': old_balance,
'new_balance': self.balance,
'timestamp': time.time()
}
self.transaction_history.append(transaction)
logging.info(f"账户 {self.account_number}: {user} 存入 {amount}, 余额: {self.balance}")
def withdraw(self, amount, user):
"""取款操作（线程安全）"""
with self.lock:
if amount <= 0:
raise ValueError("取款金额必须大于0")
if self.balance < amount:
raise ValueError(f"余额不足，当前余额: {self.balance}")
old_balance = self.balance
time.sleep(0.01) # 模拟处理时间
self.balance -= amount
# 记录交易历史
transaction = {
'type': 'withdraw',
'amount': amount,
'user': user,
'old_balance': old_balance,
'new_balance': self.balance,
'timestamp': time.time()
}
self.transaction_history.append(transaction)
logging.info(f"账户 {self.account_number}: {user} 取出 {amount}, 余额: {self.balance}")
def get_balance(self):
"""获取当前余额（线程安全）"""
with self.lock:
return self.balance
def transfer(self, target_account, amount, user):
"""转账操作（涉及两个账户的锁）"""
# 避免死锁：始终按照账户编号的顺序获取锁
first_account = min(self.account_number, target_account.account_number)
second_account = max(self.account_number, target_account.account_number)
# 确定锁的获取顺序
if self.account_number == first_account:
lock1, lock2 = self.lock, target_account.lock
else:
lock1, lock2 = target_account.lock, self.lock
with lock1:
with lock2:
# 执行转账
self.withdraw(amount, user)
target_account.deposit(amount, user)
logging.info(f"账户 {self.account_number} 向账户 {target_account.account_number} 转账 {amount}")
class TransactionThread(threading.Thread):
"""交易线程类"""
def __init__(self, accounts, user, transaction_type, amount, target_account=None):
super().__init__()
self.accounts = accounts
self.user = user
self.transaction_type = transaction_type
self.amount = amount
self.target_account = target_account
def run(self):
"""执行交易操作"""
try:
if self.transaction_type == 'deposit':
self.accounts[self.user].deposit(self.amount, self.user)
elif self.transaction_type == 'withdraw':
self.accounts[self.user].withdraw(self.amount, self.user)
elif self.transaction_type == 'transfer':
source_account = self.accounts[self.user]
target_account = self.accounts[self.target_account]
source_account.transfer(target_account, self.amount, self.user)
except Exception as e:
logging.error(f"交易失败 - 用户: {self.user}, 操作: {self.transaction_type}, 金额: {self.amount}, 错误: {str(e)}")
def simulate_concurrent_transactions():
"""模拟并发交易"""
# 创建多个银行账户
accounts = {
'Alice': BankAccount('ACC001', 1000),
'Bob': BankAccount('ACC002', 1500),
'Charlie': BankAccount('ACC003', 2000),
'David': BankAccount('ACC004', 800)
}
# 创建交易线程
transactions = [
TransactionThread(accounts, 'Alice', 'deposit', 500),
TransactionThread(accounts, 'Bob', 'withdraw', 200),
TransactionThread(accounts, 'Charlie', 'withdraw', 300),
TransactionThread(accounts, 'David', 'deposit', 1000),
TransactionThread(accounts, 'Alice', 'transfer', 200, 'Bob'),
TransactionThread(accounts, 'Bob', 'transfer', 300, 'Charlie'),
TransactionThread(accounts, 'Charlie', 'withdraw', 500),
TransactionThread(accounts, 'David', 'transfer', 150, 'Alice'),
TransactionThread(accounts, 'Alice', 'withdraw', 100),
TransactionThread(accounts, 'Bob', 'deposit', 800)
]
# 启动所有交易线程
for thread in transactions:
thread.start()
# 等待所有交易完成
for thread in transactions:
thread.join()
# 打印最终余额
logging.info("=== 交易完成后的账户余额 ===")
total_balance = 0
for user, account in accounts.items():
balance = account.get_balance()
total_balance += balance
logging.info(f"用户 {user} (账户 {account.account_number}): 余额 = {balance}")
logging.info(f"总余额: {total_balance}")
return accounts
if __name__ == "__main__":
simulate_concurrent_transactions()
---
b.条件变量
a.功能说明
条件变量允许线程等待某个特定条件成立，并在条件满足时被通知继续执行。它与锁配合使用，可以实现复杂的线程同步模式，如生产者-消费者模式。
b.代码示例
---
# 条件变量使用示例：生产者-消费者模式
import threading
import time
import random
import logging
from collections import deque
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class BoundedBuffer:
"""有界缓冲区，使用条件变量实现生产者-消费者同步"""
def __init__(self, capacity):
self.capacity = capacity
self.buffer = deque()
self.lock = threading.Lock()
self.not_empty = threading.Condition(self.lock) # 缓冲区非空条件
self.not_full = threading.Condition(self.lock) # 缓冲区非满条件
self.produced_count = 0
self.consumed_count = 0
def produce(self, item, producer_id):
"""生产者向缓冲区放入物品"""
with self.not_full:
# 等待缓冲区非满
while len(self.buffer) >= self.capacity:
logging.info(f"生产者 {producer_id} 等待缓冲区有空间...")
self.not_full.wait()
# 生产物品
self.buffer.append(item)
self.produced_count += 1
logging.info(f"生产者 {producer_id} 生产了物品 {item}，缓冲区大小: {len(self.buffer)}")
# 通知消费者缓冲区非空
self.not_empty.notify()
def consume(self, consumer_id):
"""消费者从缓冲区取出物品"""
with self.not_empty:
# 等待缓冲区非空
while len(self.buffer) == 0:
logging.info(f"消费者 {consumer_id} 等待缓冲区有物品...")
self.not_empty.wait()
# 消费物品
item = self.buffer.popleft()
self.consumed_count += 1
logging.info(f"消费者 {consumer_id} 消费了物品 {item}，缓冲区大小: {len(self.buffer)}")
# 通知生产者缓冲区非满
self.not_full.notify()
return item
def get_stats(self):
"""获取缓冲区统计信息"""
with self.lock:
return {
'buffer_size': len(self.buffer),
'capacity': self.capacity,
'produced_count': self.produced_count,
'consumed_count': self.consumed_count
}
class ProducerThread(threading.Thread):
"""生产者线程"""
def __init__(self, buffer, producer_id, items_to_produce):
super().__init__()
self.buffer = buffer
self.producer_id = producer_id
self.items_to_produce = items_to_produce
self.stop_event = threading.Event()
def run(self):
"""执行生产任务"""
items_produced = 0
while items_produced < self.items_to_produce and not self.stop_event.is_set():
# 模拟生产耗时
production_time = random.uniform(0.5, 2.0)
time.sleep(production_time)
# 生产物品
item = f"Item_{self.producer_id}_{items_produced + 1}"
self.buffer.produce(item, self.producer_id)
items_produced += 1
logging.info(f"生产者 {self.producer_id} 完成生产，共生产 {items_produced} 个物品")
def stop(self):
"""停止生产者线程"""
self.stop_event.set()
class ConsumerThread(threading.Thread):
"""消费者线程"""
def __init__(self, buffer, consumer_id, max_items_to_consume):
super().__init__()
self.buffer = buffer
self.consumer_id = consumer_id
self.max_items_to_consume = max_items_to_consume
self.items_consumed = 0
self.stop_event = threading.Event()
def run(self):
"""执行消费任务"""
while self.items_consumed < self.max_items_to_consume and not self.stop_event.is_set():
try:
# 模拟消费耗时
consumption_time = random.uniform(0.3, 1.5)
time.sleep(consumption_time)
# 消费物品
item = self.buffer.consume(self.consumer_id)
self.items_consumed += 1
# 模拟处理物品
processing_time = random.uniform(0.2, 0.8)
time.sleep(processing_time)
except Exception as e:
logging.error(f"消费者 {self.consumer_id} 消费时出错: {str(e)}")
logging.info(f"消费者 {self.consumer_id} 完成消费，共消费 {self.items_consumed} 个物品")
def stop(self):
"""停止消费者线程"""
self.stop_event.set()
def monitor_buffer(buffer, duration=10):
"""监控缓冲区状态的线程"""
start_time = time.time()
while time.time() - start_time < duration:
stats = buffer.get_stats()
logging.info(f"缓冲区监控 - 大小: {stats['buffer_size']}/{stats['capacity']}, "
f"已生产: {stats['produced_count']}, 已消费: {stats['consumed_count']}")
time.sleep(2)
def main():
"""主函数：演示生产者-消费者模式"""
# 创建有界缓冲区，容量为5
buffer = BoundedBuffer(capacity=5)
# 创建生产者线程
producers = []
for i in range(2): # 2个生产者
producer = ProducerThread(buffer, producer_id=f"P{i+1}", items_to_produce=8)
producers.append(producer)
producer.start()
# 创建消费者线程
consumers = []
for i in range(3): # 3个消费者
consumer = ConsumerThread(buffer, consumer_id=f"C{i+1}", max_items_to_consume=6)
consumers.append(consumer)
consumer.start()
# 启动监控线程
import threading
monitor_thread = threading.Thread(
target=monitor_buffer,
args=(buffer, 15),
daemon=True
)
monitor_thread.start()
# 等待生产者完成
for producer in producers:
producer.join()
# 等待消费者完成
for consumer in consumers:
consumer.join()
# 最终统计
final_stats = buffer.get_stats()
logging.info("=== 生产者-消费者模拟完成 ===")
logging.info(f"最终统计 - 已生产: {final_stats['produced_count']}, "
f"已消费: {final_stats['consumed_count']}, "
f"剩余: {final_stats['buffer_size']}")
if __name__ == "__main__":
main()
---
04.多线程性能优化
a.GIL锁问题
a.功能说明
Python的全局解释器锁（GIL）限制了同一时间只有一个线程执行Python字节码，这影响了多线程在CPU密集型任务中的性能表现。但在I/O密集型任务中，多线程仍然能显著提升性能。
b.代码示例
---
# GIL影响测试：CPU密集型 vs I/O密集型
import threading
import multiprocessing
import time
import logging
import requests
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
def cpu_intensive_task(n):
"""CPU密集型任务：计算大数的平方和"""
total = 0
for i in range(n):
total += i * i
return total
def io_intensive_task(url):
"""I/O密集型任务：发送HTTP请求"""
try:
response = requests.get(url, timeout=5)
return {
'status_code': response.status_code,
'content_length': len(response.content),
'url': url
}
except Exception as e:
return {'error': str(e), 'url': url}
def benchmark_cpu_tasks():
"""测试CPU密集型任务的性能"""
n = 1000000 # 大数计算
num_tasks = 4
logging.info("=== CPU密集型任务性能测试 ===")
# 单线程基准测试
start_time = time.time()
for i in range(num_tasks):
result = cpu_intensive_task(n)
logging.info(f"单线程任务 {i+1} 完成，结果: {result}")
single_thread_time = time.time() - start_time
logging.info(f"单线程耗时: {single_thread_time:.2f} 秒")
# 多线程测试
start_time = time.time()
with ThreadPoolExecutor(max_workers=num_tasks) as executor:
futures = [executor.submit(cpu_intensive_task, n) for _ in range(num_tasks)]
for i, future in enumerate(futures):
result = future.result()
logging.info(f"多线程任务 {i+1} 完成，结果: {result}")
multi_thread_time = time.time() - start_time
logging.info(f"多线程耗时: {multi_thread_time:.2f} 秒")
# 多进程测试（绕过GIL）
start_time = time.time()
with ProcessPoolExecutor(max_workers=num_tasks) as executor:
futures = [executor.submit(cpu_intensive_task, n) for _ in range(num_tasks)]
for i, future in enumerate(futures):
result = future.result()
logging.info(f"多进程任务 {i+1} 完成，结果: {result}")
multi_process_time = time.time() - start_time
logging.info(f"多进程耗时: {multi_process_time:.2f} 秒")
# 性能对比
logging.info("=== CPU密集型任务性能对比 ===")
logging.info(f"单线程: {single_thread_time:.2f}s (基准)")
logging.info(f"多线程: {multi_thread_time:.2f}s ({single_thread_time/multi_thread_time:.2f}x)")
logging.info(f"多进程: {multi_process_time:.2f}s ({single_thread_time/multi_process_time:.2f}x)")
return {
'single_thread': single_thread_time,
'multi_thread': multi_thread_time,
'multi_process': multi_process_time
}
def benchmark_io_tasks():
"""测试I/O密集型任务的性能"""
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/status/200'
]
logging.info("=== I/O密集型任务性能测试 ===")
# 串行执行基准测试
start_time = time.time()
for i, url in enumerate(urls):
result = io_intensive_task(url)
logging.info(f"串行任务 {i+1} 完成: {result}")
serial_time = time.time() - start_time
logging.info(f"串行耗时: {serial_time:.2f} 秒")
# 多线程执行
start_time = time.time()
with ThreadPoolExecutor(max_workers=len(urls)) as executor:
futures = [executor.submit(io_intensive_task, url) for url in urls]
for i, future in enumerate(futures):
result = future.result()
logging.info(f"多线程任务 {i+1} 完成: {result}")
multi_thread_time = time.time() - start_time
logging.info(f"多线程耗时: {multi_thread_time:.2f} 秒")
# 性能对比
logging.info("=== I/O密集型任务性能对比 ===")
logging.info(f"串行执行: {serial_time:.2f}s (基准)")
logging.info(f"多线程: {multi_thread_time:.2f}s ({serial_time/multi_thread_time:.2f}x)")
return {
'serial': serial_time,
'multi_thread': multi_thread_time
}
def analyze_gil_impact():
"""分析GIL对多线程性能的影响"""
logging.info("=== GIL影响分析 ===")
# 测试GIL释放
def gil_releasing_task():
"""执行包含大量I/O操作的任务，会释放GIL"""
import socket
import time
# 网络I/O会释放GIL
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(2)
try:
sock.connect(('httpbin.org', 80))
sock.send(b'GET / HTTP/1.1\r\nHost: httpbin.org\r\n\r\n')
data = sock.recv(1024)
except:
pass
finally:
sock.close()
# 模拟I/O等待
time.sleep(0.1)
return "GIL released task completed"
def gil_holding_task():
"""执行纯CPU计算，会持有GIL"""
total = 0
for i in range(100000):
total += i ** 2
return f"GIL holding task completed: {total}"
# 并发执行GIL释放任务
start_time = time.time()
with ThreadPoolExecutor(max_workers=4) as executor:
futures = [executor.submit(gil_releasing_task) for _ in range(10)]
results = [future.result() for future in futures]
gil_release_time = time.time() - start_time
logging.info(f"GIL释放任务耗时: {gil_release_time:.2f}秒")
# 并发执行GIL持有任务
start_time = time.time()
with ThreadPoolExecutor(max_workers=4) as executor:
futures = [executor.submit(gil_holding_task) for _ in range(10)]
results = [future.result() for future in futures]
gil_hold_time = time.time() - start_time
logging.info(f"GIL持有任务耗时: {gil_hold_time:.2f}秒")
logging.info(f"GIL释放任务 vs GIL持有任务性能比: {gil_hold_time/gil_release_time:.2f}x")
def main():
"""主函数：执行所有性能测试"""
logging.info("开始多线程性能基准测试...")
# CPU密集型任务测试
cpu_results = benchmark_cpu_tasks()
print() # 空行分隔
# I/O密集型任务测试
io_results = benchmark_io_tasks()
print() # 空行分隔
# GIL影响分析
analyze_gil_impact()
# 总结建议
logging.info("=== 性能优化建议 ===")
logging.info("1. CPU密集型任务：建议使用多进程(multiprocessing)")
logging.info("2. I/O密集型任务：多线程表现良好，可以充分利用并发优势")
logging.info("3. 混合型任务：考虑使用多进程+多线程的混合模式")
logging.info("4. 避免在Python多线程中进行大量CPU计算")
if __name__ == "__main__":
main()
---
b.线程数优化
a.功能说明
合理设置线程数是提升多线程性能的关键。线程数过少无法充分利用CPU资源，过多则会导致频繁的线程切换和资源竞争。通常需要根据任务类型和硬件配置来确定最佳线程数。
b.代码示例
---
# 线程数优化测试
import threading
import time
import logging
import multiprocessing
import psutil
from concurrent.futures import ThreadPoolExecutor, as_completed
import random
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class ThreadPoolOptimizer:
"""线程池优化器"""
def __init__(self):
self.cpu_count = multiprocessing.cpu_count()
self.logical_cores = psutil.cpu_count(logical=True)
self.physical_cores = psutil.cpu_count(logical=False)
def get_system_info(self):
"""获取系统硬件信息"""
return {
'cpu_count': self.cpu_count,
'logical_cores': self.logical_cores,
'physical_cores': self.physical_cores,
'memory_gb': psutil.virtual_memory().total / (1024**3)
}
def calculate_optimal_thread_count(self, task_type='mixed'):
"""计算最优线程数"""
system_info = self.get_system_info()
if task_type == 'cpu_intensive':
# CPU密集型任务：通常等于物理核心数
return self.physical_cores
elif task_type == 'io_intensive':
# I/O密集型任务：可以是逻辑核心数的2-3倍
return min(self.logical_cores * 3, 32)
elif task_type == 'mixed':
# 混合型任务：逻辑核心数的1.5-2倍
return min(int(self.logical_cores * 1.5), 16)
else:
return self.cpu_count
def simulate_cpu_intensive_task(duration=0.5):
"""模拟CPU密集型任务"""
start_time = time.time()
count = 0
while time.time() - start_time < duration:
count += 1
# 进行一些计算
_ = sum(range(1000))
return count
def simulate_io_intensive_task(duration=0.3):
"""模拟I/O密集型任务"""
start_time = time.time()
count = 0
while time.time() - start_time < duration:
count += 1
# 模拟I/O等待
time.sleep(0.1)
return count
def simulate_mixed_task(duration=0.4):
"""模拟混合型任务"""
start_time = time.time()
count = 0
while time.time() - start_time < duration:
count += 1
# CPU计算
_ = sum(range(500))
# I/O等待
time.sleep(0.05)
return count
def benchmark_thread_pool(task_func, num_tasks, max_workers_list, task_name):
"""测试不同线程数下的性能"""
results = []
for max_workers in max_workers_list:
logging.info(f"测试 {task_name} - 线程数: {max_workers}")
start_time = time.time()
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(task_func) for _ in range(num_tasks)]
completed_tasks = 0
total_results = 0
for future in as_completed(futures):
try:
result = future.result()
completed_tasks += 1
total_results += result
except Exception as e:
logging.error(f"任务执行失败: {e}")
execution_time = time.time() - start_time
throughput = num_tasks / execution_time
result = {
'max_workers': max_workers,
'execution_time': execution_time,
'throughput': throughput,
'completed_tasks': completed_tasks,
'total_results': total_results
}
results.append(result)
logging.info(f"线程数 {max_workers}: 耗时 {execution_time:.2f}s, "
f"吞吐量 {throughput:.2f} tasks/s")
return results
def find_optimal_thread_count(results):
"""找出最优线程数"""
if not results:
return None
# 以吞吐量为指标找出最优线程数
optimal = max(results, key=lambda x: x['throughput'])
return optimal
def analyze_scaling_efficiency(results, base_result):
"""分析扩展效率"""
efficiency_results = []
for result in results:
if result['max_workers'] == base_result['max_workers']:
efficiency = 1.0
else:
ideal_speedup = result['max_workers'] / base_result['max_workers']
actual_speedup = result['throughput'] / base_result['throughput']
efficiency = actual_speedup / ideal_speedup
result['efficiency'] = efficiency
result['ideal_speedup'] = result['max_workers'] / base_result['max_workers']
result['actual_speedup'] = result['throughput'] / base_result['throughput']
efficiency_results.append(result)
return efficiency_results
def main():
"""主函数：执行线程数优化测试"""
logging.info("=== 线程数优化测试开始 ===")
optimizer = ThreadPoolOptimizer()
system_info = optimizer.get_system_info()
logging.info("系统信息:")
for key, value in system_info.items():
logging.info(f" {key}: {value}")
# 测试不同的线程数范围
max_workers_list = [1, 2, 4, 8, 12, 16, 20, 24, 32]
num_tasks = 50
# 测试不同类型的任务
task_configs = [
(simulate_cpu_intensive_task, 'CPU密集型任务'),
(simulate_io_intensive_task, 'I/O密集型任务'),
(simulate_mixed_task, '混合型任务')
]
all_results = {}
for task_func, task_name in task_configs:
logging.info(f"\n=== 测试 {task_name} ===")
# 执行性能测试
results = benchmark_thread_pool(task_func, num_tasks, max_workers_list, task_name)
all_results[task_name] = results
# 找出最优线程数
optimal = find_optimal_thread_count(results)
if optimal:
logging.info(f"{task_name} 最优线程数: {optimal['max_workers']} "
f"(吞吐量: {optimal['throughput']:.2f} tasks/s)")
# 分析扩展效率
if results:
base_result = results[0] # 单线程作为基准
efficiency_results = analyze_scaling_efficiency(results, base_result)
logging.info(f"\n{task_name} 扩展效率分析:")
for result in efficiency_results[::3]: # 每3个显示一个，避免输出过多
logging.info(f" 线程数 {result['max_workers']}: "
f"理论加速比 {result['ideal_speedup']:.2f}x, "
f"实际加速比 {result['actual_speedup']:.2f}x, "
f"效率 {result['efficiency']:.2%}")
# 生成优化建议
logging.info("\n=== 线程数优化建议 ===")
for task_name, results in all_results.items():
if results:
optimal = find_optimal_thread_count(results)
if optimal:
optimal_threads = optimal['max_workers']
if 'CPU' in task_name:
recommended = optimizer.calculate_optimal_thread_count('cpu_intensive')
elif 'I/O' in task_name:
recommended = optimizer.calculate_optimal_thread_count('io_intensive')
else:
recommended = optimizer.calculate_optimal_thread_count('mixed')
logging.info(f"{task_name}:")
logging.info(f" 实测最优线程数: {optimal_threads}")
logging.info(f" 理论推荐线程数: {recommended}")
logging.info(f" 建议使用线程数: {min(optimal_threads, recommended)}")
if __name__ == "__main__":
main()
---
8.2 多进程模型
01.基本概念
a.进程定义
进程是操作系统进行资源分配和调度的基本单位，每个进程拥有独立的内存空间、文件句柄和系统资源。进程间通过进程间通信(IPC)机制进行数据交换，提供了良好的隔离性和安全性。
b.多进程特点
多进程编程通过创建多个进程实现真正的并行执行，每个进程有独立的GIL解释器，能够充分利用多核CPU资源。进程间通信成本较高，但避免了GIL限制，适合CPU密集型任务。
c.应用场景
多进程模型适用于CPU密集型计算、科学计算、图像处理、数据分析等需要充分利用多核资源的场景，以及对安全性和隔离性要求较高的应用程序。
02.进程创建与管理
a.基础进程创建
a.功能说明
Python的multiprocessing模块提供了强大的多进程支持，Process类用于创建子进程。每个进程都有独立的内存空间和Python解释器，实现了真正的并行执行。
b.代码示例
---
# 多进程基础示例
import multiprocessing
import time
import logging
import os
from datetime import datetime
# 配置日志，显示进程ID
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class WorkerProcess(multiprocessing.Process):
"""自定义工作进程类"""
def __init__(self, process_id, name, task_count=5):
super(WorkerProcess, self).__init__()
self.process_id = process_id
self.name = name
self.task_count = task_count
self.stop_event = multiprocessing.Event()
def run(self):
"""进程执行的核心方法"""
pid = os.getpid()
ppid = os.getppid()
logging.info(f"进程 {self.name} 启动，PID: {pid}, 父进程PID: {ppid}")
for i in range(self.task_count):
if self.stop_event.is_set():
logging.info(f"进程 {self.name} 收到停止信号，准备退出")
break
# 模拟耗时计算任务
start_time = time.time()
result = self.calculate_fibonacci(30)
execution_time = time.time() - start_time
logging.info(f"进程 {self.name} 完成任务 {i+1}/{self.task_count}, "
f"Fibonacci(30)={result}, 耗时: {execution_time:.4f}s")
# 进程间短暂休息
time.sleep(0.1)
logging.info(f"进程 {self.name} 执行完成，PID: {pid}")
def calculate_fibonacci(self, n):
"""计算斐波那契数列（CPU密集型任务）"""
if n <= 1:
return n
a, b = 0, 1
for _ in range(2, n + 1):
a, b = b, a + b
return b
def stop(self):
"""安全停止进程"""
self.stop_event.set()
class ProcessManager:
"""进程管理器"""
def __init__(self, max_processes=4):
self.max_processes = max_processes
self.processes = []
self.start_time = None
def create_worker_processes(self):
"""创建工作进程"""
for i in range(self.max_processes):
process = WorkerProcess(
process_id=i,
name=f"Worker-{i+1}",
task_count=5
)
self.processes.append(process)
def start_all_processes(self):
"""启动所有进程"""
logging.info(f"开始启动 {len(self.processes)} 个工作进程...")
self.start_time = time.time()
for process in self.processes:
process.start()
logging.info(f"已启动进程 {process.name}, PID: {process.pid}")
def wait_completion(self):
"""等待所有进程完成"""
logging.info("等待所有进程完成...")
for process in self.processes:
process.join()
total_time = time.time() - self.start_time
logging.info(f"所有进程执行完成，总耗时: {total_time:.2f}秒")
def stop_all_processes(self):
"""停止所有进程"""
logging.info("正在停止所有进程...")
for process in self.processes:
if process.is_alive():
process.stop()
process.join(timeout=5)
if process.is_alive():
process.terminate()
logging.warning(f"强制终止进程 {process.name}")
else:
logging.info(f"进程 {process.name} 正常停止")
def get_process_statistics(self):
"""获取进程统计信息"""
stats = {
'total_processes': len(self.processes),
'completed_processes': sum(1 for p in self.processes if not p.is_alive()),
'running_processes': sum(1 for p in self.processes if p.is_alive())
}
# 获取系统进程信息
current_process = multiprocessing.current_process()
stats['current_pid'] = os.getpid()
stats['current_ppid'] = os.getppid()
return stats
def demonstrate_process_creation():
"""演示进程创建和管理"""
logging.info("=== 多进程创建演示开始 ===")
# 获取进程信息
main_pid = os.getpid()
logging.info(f"主进程PID: {main_pid}")
logging.info(f"系统CPU核心数: {multiprocessing.cpu_count()}")
# 创建进程管理器
manager = ProcessManager(max_processes=3)
try:
# 创建工作进程
manager.create_worker_processes()
# 启动所有进程
manager.start_all_processes()
# 定期显示进程状态
for i in range(10):
time.sleep(1)
stats = manager.get_process_statistics()
logging.info(f"进程状态监控 - 运行中: {stats['running_processes']}, "
f"已完成: {stats['completed_processes']}")
if stats['running_processes'] == 0:
break
# 等待所有进程完成
manager.wait_completion()
except KeyboardInterrupt:
logging.info("收到中断信号，正在停止所有进程...")
manager.stop_all_processes()
# 最终统计
final_stats = manager.get_process_statistics()
logging.info(f"最终统计 - 总进程数: {final_stats['total_processes']}, "
f"已完成: {final_stats['completed_processes']}")
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
demonstrate_process_creation()
---
b.进程池管理
a.功能说明
进程池是一种预先创建并管理多个工作进程的技术，避免了频繁创建和销毁进程的开销。Python的multiprocessing.Pool提供了高效的进程池实现，支持多种任务分配和结果收集方式。
b.代码示例
---
# 进程池使用示例
import multiprocessing
import time
import logging
import os
import random
from multiprocessing import Pool, Manager
from functools import partial
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class ProcessPoolManager:
"""进程池管理器"""
def __init__(self, pool_size=None):
self.pool_size = pool_size or multiprocessing.cpu_count()
self.pool = None
self.task_counter = multiprocessing.Value('i', 0)
def cpu_intensive_task(self, data):
"""CPU密集型任务：计算数组统计信息"""
task_id = self.task_counter.value
with self.task_counter.get_lock():
self.task_counter.value += 1
array = data['array']
process_id = os.getpid()
logging.info(f"任务 {task_id} 开始处理，数组长度: {len(array)}, 进程PID: {process_id}")
start_time = time.time()
# 执行复杂计算
total = sum(array)
mean = total / len(array) if array else 0
variance = sum((x - mean) ** 2 for x in array) / len(array) if array else 0
# 模拟更多计算负载
result = 1
for i in range(1000000):
result += i % len(array) if array else 1
execution_time = time.time() - start_time
result_data = {
'task_id': task_id,
'process_id': process_id,
'array_length': len(array),
'total': total,
'mean': mean,
'variance': variance,
'calculation_result': result,
'execution_time': execution_time
}
logging.info(f"任务 {task_id} 完成，耗时: {execution_time:.4f}s")
return result_data
def parallel_file_processing(self, file_path):
"""并行文件处理任务"""
task_id = self.task_counter.value
with self.task_counter.get_lock():
self.task_counter.value += 1
process_id = os.getpid()
logging.info(f"文件处理任务 {task_id} 开始: {file_path}, 进程PID: {process_id}")
try:
# 模拟文件读取和处理
time.sleep(random.uniform(0.1, 0.5))
# 模拟文件内容分析
file_size = random.randint(1000, 10000)
word_count = random.randint(100, 1000)
line_count = random.randint(10, 100)
processing_time = random.uniform(0.05, 0.2)
time.sleep(processing_time)
result = {
'task_id': task_id,
'process_id': process_id,
'file_path': file_path,
'file_size': file_size,
'word_count': word_count,
'line_count': line_count,
'processing_time': processing_time
}
logging.info(f"文件处理任务 {task_id} 完成: {file_path}")
return result
except Exception as e:
error_result = {
'task_id': task_id,
'process_id': process_id,
'file_path': file_path,
'error': str(e)
}
logging.error(f"文件处理任务 {task_id} 失败: {str(e)}")
return error_result
def batch_process_with_pool(self, tasks, task_func):
"""使用进程池批量处理任务"""
logging.info(f"开始批量处理 {len(tasks)} 个任务，进程池大小: {self.pool_size}")
results = []
failed_tasks = []
with Pool(processes=self.pool_size) as pool:
try:
# 使用imap_unordered获取实时结果
start_time = time.time()
for i, result in enumerate(pool.imap_unordered(task_func, tasks)):
if 'error' in result:
failed_tasks.append(result)
else:
results.append(result)
# 显示进度
if (i + 1) % 5 == 0:
logging.info(f"已完成 {i + 1}/{len(tasks)} 个任务")
total_time = time.time() - start_time
except Exception as e:
logging.error(f"进程池执行出错: {str(e)}")
return [], []
return results, failed_tasks, total_time
def analyze_results(self, results):
"""分析执行结果"""
if not results:
return {}
# 计算统计信息
total_execution_time = sum(r['execution_time'] for r in results)
avg_execution_time = total_execution_time / len(results)
process_ids = list(set(r['process_id'] for r in results))
process_distribution = {pid: sum(1 for r in results if r['process_id'] == pid)
for pid in process_ids}
analysis = {
'total_tasks': len(results),
'total_execution_time': total_execution_time,
'avg_execution_time': avg_execution_time,
'min_execution_time': min(r['execution_time'] for r in results),
'max_execution_time': max(r['execution_time'] for r in results),
'process_count': len(process_ids),
'process_distribution': process_distribution,
'load_balance_ratio': min(process_distribution.values()) / max(process_distribution.values())
}
return analysis
def generate_test_data(num_arrays=20):
"""生成测试数据"""
data = []
for i in range(num_arrays):
array_size = random.randint(1000, 5000)
array = [random.randint(1, 1000) for _ in range(array_size)]
data.append({'array': array, 'id': i})
return data
def generate_file_list(num_files=15):
"""生成文件列表"""
return [f'/tmp/test_file_{i}.txt' for i in range(num_files)]
def demonstrate_pool_usage():
"""演示进程池的使用"""
logging.info("=== 进程池使用演示开始 ===")
# 显示系统信息
logging.info(f"CPU核心数: {multiprocessing.cpu_count()}")
logging.info(f"当前进程PID: {os.getpid()}")
# 创建进程池管理器
pool_manager = ProcessPoolManager(pool_size=4)
# 测试1: CPU密集型任务
logging.info("\n--- 测试1: CPU密集型任务 ---")
test_arrays = generate_test_data(12)
results1, failed1, time1 = pool_manager.batch_process_with_pool(
test_arrays, pool_manager.cpu_intensive_task
)
if results1:
analysis1 = pool_manager.analyze_results(results1)
logging.info(f"CPU密集型任务分析:")
logging.info(f" 总任务数: {analysis1['total_tasks']}")
logging.info(f" 平均执行时间: {analysis1['avg_execution_time']:.4f}s")
logging.info(f" 使用进程数: {analysis1['process_count']}")
logging.info(f" 负载均衡比: {analysis1['load_balance_ratio']:.2f}")
logging.info(f" 总执行时间: {time1:.2f}s")
# 测试2: 文件处理任务
logging.info("\n--- 测试2: 并行文件处理 ---")
test_files = generate_file_list(15)
results2, failed2, time2 = pool_manager.batch_process_with_pool(
test_files, pool_manager.parallel_file_processing
)
logging.info(f"文件处理完成:")
logging.info(f" 成功处理: {len(results2)} 个文件")
logging.info(f" 处理失败: {len(failed2)} 个文件")
logging.info(f" 总处理时间: {time2:.2f}s")
# 显示一些成功结果
for result in results2[:3]: # 显示前3个结果
logging.info(f" 文件 {result['file_path']}: "
f"大小={result['file_size']}, "
f"词数={result['word_count']}")
# 显示失败的任务
for failed in failed2:
logging.error(f" 失败文件 {failed['file_path']}: {failed['error']}")
def demonstrate_different_pool_methods():
"""演示不同的进程池使用方法"""
logging.info("\n=== 不同进程池使用方法演示 ===")
with Pool(processes=3) as pool:
# 方法1: map - 同步执行，保持顺序
logging.info("--- 方法1: map ---")
data = [i for i in range(1, 6)]
start_time = time.time()
results_map = pool.map(lambda x: x * x, data)
map_time = time.time() - start_time
logging.info(f"map结果: {results_map}, 耗时: {map_time:.4f}s")
# 方法2: map_async - 异步执行
logging.info("--- 方法2: map_async ---")
start_time = time.time()
result_async = pool.map_async(lambda x: x * x, data)
async_results = result_async.get()
async_time = time.time() - start_time
logging.info(f"map_async结果: {async_results}, 耗时: {async_time:.4f}s")
# 方法3: apply - 单个任务
logging.info("--- 方法3: apply ---")
start_time = time.time()
result_apply = pool.apply(lambda x: x * x, (10,))
apply_time = time.time() - start_time
logging.info(f"apply结果: {result_apply}, 耗时: {apply_time:.4f}s")
# 方法4: apply_async - 异步单个任务
logging.info("--- 方法4: apply_async ---")
start_time = time.time()
result_async_single = pool.apply_async(lambda x: x * x, (10,))
async_single_result = result_async_single.get()
apply_async_time = time.time() - start_time
logging.info(f"apply_async结果: {async_single_result}, 耗时: {apply_async_time:.4f}s")
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
try:
demonstrate_pool_usage()
demonstrate_different_pool_methods()
except KeyboardInterrupt:
logging.info("程序被用户中断")
except Exception as e:
logging.error(f"程序执行出错: {str(e)}")
---
03.进程间通信(IPC)
a.队列通信
a.功能说明
multiprocessing.Queue提供了进程安全的队列通信机制，支持多进程间的数据交换。队列使用管道和锁机制实现，确保了线程和进程安全，适合生产者-消费者模式。
b.代码示例
---
# 进程间队列通信示例
import multiprocessing
import time
import logging
import os
import random
from multiprocessing import Queue, Process, Manager
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class TaskData:
"""任务数据类"""
def __init__(self, task_id, data, priority=0):
self.task_id = task_id
self.data = data
self.priority = priority
self.created_time = datetime.now()
self.processing_start_time = None
self.processing_end_time = None
def start_processing(self):
"""开始处理"""
self.processing_start_time = datetime.now()
def finish_processing(self):
"""完成处理"""
self.processing_end_time = datetime.now()
def get_processing_duration(self):
"""获取处理时长"""
if self.processing_start_time and self.processing_end_time:
return (self.processing_end_time - self.processing_start_time).total_seconds()
return 0
def get_total_duration(self):
"""获取总时长"""
end_time = self.processing_end_time or datetime.now()
return (end_time - self.created_time).total_seconds()
def __lt__(self, other):
"""用于优先级队列排序"""
return self.priority < other.priority
class ProducerProcess(Process):
"""生产者进程"""
def __init__(self, producer_id, task_queue, num_tasks=10):
super(ProducerProcess, self).__init__()
self.producer_id = producer_id
self.task_queue = task_queue
self.num_tasks = num_tasks
self.stop_event = multiprocessing.Event()
def run(self):
"""生产者执行逻辑"""
logging.info(f"生产者 {self.producer_id} 开始运行，PID: {os.getpid()}")
for i in range(self.num_tasks):
if self.stop_event.is_set():
logging.info(f"生产者 {self.producer_id} 收到停止信号")
break
# 创建任务
task_data = self.generate_task(i)
# 模拟生产间隔
production_time = random.uniform(0.1, 0.5)
time.sleep(production_time)
try:
# 将任务放入队列
self.task_queue.put(task_data, timeout=5)
logging.info(f"生产者 {self.producer_id} 生产了任务 {task_data.task_id}, "
f"数据大小: {len(task_data.data)}")
except Exception as e:
logging.error(f"生产者 {self.producer_id} 放入任务失败: {str(e)}")
logging.info(f"生产者 {self.producer_id} 生产完成，共生产 {self.num_tasks} 个任务")
def generate_task(self, task_index):
"""生成任务数据"""
task_id = f"Task_{self.producer_id}_{task_index}"
data_size = random.randint(100, 1000)
data = [random.randint(1, 1000) for _ in range(data_size)]
priority = random.randint(1, 5)
return TaskData(task_id, data, priority)
def stop(self):
"""停止生产者"""
self.stop_event.set()
class ConsumerProcess(Process):
"""消费者进程"""
def __init__(self, consumer_id, task_queue, result_queue):
super(ConsumerProcess, self).__init__()
self.consumer_id = consumer_id
self.task_queue = task_queue
self.result_queue = result_queue
self.processed_count = 0
self.stop_event = multiprocessing.Event()
def run(self):
"""消费者执行逻辑"""
logging.info(f"消费者 {self.consumer_id} 开始运行，PID: {os.getpid()}")
while not self.stop_event.is_set():
try:
# 从队列获取任务，设置超时避免无限等待
task_data = self.task_queue.get(timeout=2)
# 开始处理任务
task_data.start_processing()
logging.info(f"消费者 {self.consumer_id} 开始处理任务 {task_data.task_id}")
# 模拟处理任务
processing_result = self.process_task(task_data)
# 完成处理
task_data.finish_processing()
self.processed_count += 1
# 将结果放入结果队列
result = {
'consumer_id': self.consumer_id,
'task_id': task_data.task_id,
'result': processing_result,
'processing_duration': task_data.get_processing_duration(),
'total_duration': task_data.get_total_duration()
}
self.result_queue.put(result)
logging.info(f"消费者 {self.consumer_id} 完成任务 {task_data.task_id}, "
f"处理时长: {result['processing_duration']:.3f}s")
except Exception as e:
if isinstance(e, Exception) and 'Empty' in str(e):
# 队列为空，继续等待
continue
else:
logging.error(f"消费者 {self.consumer_id} 处理任务时出错: {str(e)}")
break
logging.info(f"消费者 {self.consumer_id} 处理完成，共处理 {self.processed_count} 个任务")
def process_task(self, task_data):
"""处理具体任务"""
# 模拟复杂计算
data = task_data.data
processing_time = random.uniform(0.2, 1.0)
time.sleep(processing_time)
# 执行一些计算
total = sum(data)
mean = total / len(data) if data else 0
max_val = max(data) if data else 0
min_val = min(data) if data else 0
return {
'total': total,
'mean': mean,
'max': max_val,
'min': min_val,
'processing_time': processing_time
}
def stop(self):
"""停止消费者"""
self.stop_event.set()
class QueueCommunicationDemo:
"""队列通信演示类"""
def __init__(self, num_producers=2, num_consumers=3):
self.num_producers = num_producers
self.num_consumers = num_consumers
self.task_queue = Queue()
self.result_queue = Queue()
self.producers = []
self.consumers = []
def start_demo(self):
"""开始演示"""
logging.info(f"=== 队列通信演示开始 ===")
logging.info(f"生产者数量: {self.num_producers}, 消费者数量: {self.num_consumers}")
# 创建生产者
for i in range(self.num_producers):
producer = ProducerProcess(
producer_id=f"P{i+1}",
task_queue=self.task_queue,
num_tasks=8
)
self.producers.append(producer)
# 创建消费者
for i in range(self.num_consumers):
consumer = ConsumerProcess(
consumer_id=f"C{i+1}",
task_queue=self.task_queue,
result_queue=self.result_queue
)
self.consumers.append(consumer)
# 启动所有进程
logging.info("启动所有生产者和消费者进程...")
for producer in self.producers:
producer.start()
for consumer in self.consumers:
consumer.start()
# 等待生产者完成
for producer in self.producers:
producer.join()
# 等待一段时间让消费者处理完剩余任务
logging.info("等待消费者处理剩余任务...")
time.sleep(3)
# 停止所有消费者
for consumer in self.consumers:
consumer.stop()
consumer.join(timeout=5)
# 收集结果
self.collect_results()
def collect_results(self):
"""收集和分析结果"""
logging.info("收集处理结果...")
results = []
while not self.result_queue.empty():
try:
result = self.result_queue.get(timeout=1)
results.append(result)
except:
break
if results:
# 分析结果
total_processed = len(results)
consumer_stats = {}
total_processing_time = sum(r['processing_duration'] for r in results)
for result in results:
consumer_id = result['consumer_id']
if consumer_id not in consumer_stats:
consumer_stats[consumer_id] = {'count': 0, 'total_time': 0}
consumer_stats[consumer_id]['count'] += 1
consumer_stats[consumer_id]['total_time'] += result['processing_duration']
# 输出统计信息
logging.info("=== 处理结果统计 ===")
logging.info(f"总处理任务数: {total_processed}")
logging.info(f"平均处理时间: {total_processing_time/total_processed:.3f}s")
logging.info("各消费者处理统计:")
for consumer_id, stats in consumer_stats.items():
avg_time = stats['total_time'] / stats['count']
logging.info(f" {consumer_id}: 处理 {stats['count']} 个任务, "
f"平均耗时 {avg_time:.3f}s")
def demonstrate_priority_queue():
"""演示优先级队列"""
logging.info("\n=== 优先级队列演示 ===")
# Manager可以创建特殊的队列类型
manager = Manager()
priority_queue = manager.Queue()
def priority_producer(queue, items):
"""优先级生产者"""
for item in items:
queue.put(item)
logging.info(f"放入优先级 {item[0]} 的任务: {item[1]}")
def priority_consumer(queue):
"""优先级消费者"""
while True:
try:
task = queue.get(timeout=2)
logging.info(f"获取到优先级 {task[0]} 的任务: {task[1]}")
time.sleep(0.5)
except:
break
# 创建测试数据（优先级，任务名）
tasks = [
(3, "低优先级任务1"),
(1, "高优先级任务1"),
(2, "中优先级任务1"),
(1, "高优先级任务2"),
(3, "低优先级任务2"),
(2, "中优先级任务2")
]
# 创建生产者和消费者
producer = Process(target=priority_producer, args=(priority_queue, tasks))
consumer = Process(target=priority_consumer, args=(priority_queue,))
producer.start()
time.sleep(1) # 让生产者先放入一些任务
consumer.start()
producer.join()
time.sleep(1)
consumer.terminate() # 强制终止消费者
consumer.join()
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
try:
# 演示基本队列通信
demo = QueueCommunicationDemo(num_producers=2, num_consumers=3)
demo.start_demo()
# 演示优先级队列
demonstrate_priority_queue()
except KeyboardInterrupt:
logging.info("程序被用户中断")
except Exception as e:
logging.error(f"程序执行出错: {str(e)}")
---
b.共享内存通信
a.功能说明
multiprocessing.Value和Array提供了进程间共享内存的机制，允许多个进程直接访问同一块内存区域。这种方式性能较高，适合大量数据的共享，但需要注意同步和并发控制。
b.代码示例
---
# 共享内存通信示例
import multiprocessing
import time
import logging
import os
import random
from multiprocessing import Value, Array, Process, Lock
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class SharedMemoryCounter:
"""基于共享内存的计数器"""
def __init__(self):
# 创建共享整数，初始值为0
self.counter = Value('i', 0)
self.lock = Lock()
def increment(self, amount=1):
"""线程安全的递增"""
with self.lock:
self.counter.value += amount
return self.counter.value
def get_value(self):
"""获取当前值"""
with self.lock:
return self.counter.value
def reset(self):
"""重置计数器"""
with self.lock:
self.counter.value = 0
class SharedMemoryArray:
"""基于共享内存的数组操作"""
def __init__(self, size):
# 创建共享数组，类型为'd'（双精度浮点数）
self.data = Array('d', size)
self.size = size
self.lock = Lock()
def set_data(self, index, value):
"""设置数组元素"""
if 0 <= index < self.size:
with self.lock:
self.data[index] = value
else:
raise IndexError("Index out of range")
def get_data(self, index):
"""获取数组元素"""
if 0 <= index < self.size:
with self.lock:
return self.data[index]
else:
raise IndexError("Index out of range")
def get_all_data(self):
"""获取所有数据"""
with self.lock:
return list(self.data)
def calculate_statistics(self):
"""计算统计信息"""
with self.lock:
data_list = list(self.data)
return {
'sum': sum(data_list),
'mean': sum(data_list) / len(data_list) if data_list else 0,
'max': max(data_list) if data_list else 0,
'min': min(data_list) if data_list else 0,
'length': len(data_list)
}
class DataProcessor(Process):
"""数据处理进程"""
def __init__(self, processor_id, shared_array, shared_counter, operation_count=100):
super(DataProcessor, self).__init__()
self.processor_id = processor_id
self.shared_array = shared_array
self.shared_counter = shared_counter
self.operation_count = operation_count
def run(self):
"""执行数据处理"""
logging.info(f"处理器 {self.processor_id} 开始运行，PID: {os.getpid()}")
for i in range(self.operation_count):
# 随机选择操作类型
operation = random.choice(['write', 'read', 'update'])
try:
if operation == 'write':
self.perform_write_operation()
elif operation == 'read':
self.perform_read_operation()
else:
self.perform_update_operation()
# 更新操作计数器
self.shared_counter.increment()
# 模拟处理间隔
time.sleep(random.uniform(0.01, 0.1))
except Exception as e:
logging.error(f"处理器 {self.processor_id} 操作失败: {str(e)}")
logging.info(f"处理器 {self.processor_id} 完成所有操作")
def perform_write_operation(self):
"""执行写操作"""
# 随机选择一个位置写入数据
index = random.randint(0, self.shared_array.size - 1)
value = random.uniform(1.0, 1000.0)
self.shared_array.set_data(index, value)
logging.debug(f"处理器 {self.processor_id} 在位置 {index} 写入值 {value:.2f}")
def perform_read_operation(self):
"""执行读操作"""
index = random.randint(0, self.shared_array.size - 1)
value = self.shared_array.get_data(index)
logging.debug(f"处理器 {self.processor_id} 从位置 {index} 读取值 {value:.2f}")
def perform_update_operation(self):
"""执行更新操作"""
index = random.randint(0, self.shared_array.size - 1)
old_value = self.shared_array.get_data(index)
new_value = old_value * random.uniform(0.5, 2.0)
self.shared_array.set_data(index, new_value)
logging.debug(f"处理器 {self.processor_id} 在位置 {index} 更新值 {old_value:.2f} -> {new_value:.2f}")
class SharedMemoryManager:
"""共享内存管理器"""
def __init__(self, array_size=100, num_processors=4):
self.array_size = array_size
self.num_processors = num_processors
self.shared_array = SharedMemoryArray(array_size)
self.shared_counter = SharedMemoryCounter()
self.processors = []
def initialize_data(self):
"""初始化共享数据"""
logging.info("初始化共享数组数据...")
for i in range(self.array_size):
initial_value = random.uniform(1.0, 100.0)
self.shared_array.set_data(i, initial_value)
initial_stats = self.shared_array.calculate_statistics()
logging.info(f"初始数据统计: sum={initial_stats['sum']:.2f}, "
f"mean={initial_stats['mean']:.2f}")
def create_processors(self):
"""创建处理器进程"""
for i in range(self.num_processors):
processor = DataProcessor(
processor_id=f"P{i+1}",
shared_array=self.shared_array,
shared_counter=self.shared_counter,
operation_count=200
)
self.processors.append(processor)
def start_processing(self, duration=10):
"""开始数据处理"""
logging.info(f"开始并发数据处理，持续时间: {duration}秒")
# 记录开始时间
start_time = time.time()
# 启动所有处理器
for processor in self.processors:
processor.start()
# 监控处理进度
while time.time() - start_time < duration:
time.sleep(2)
current_ops = self.shared_counter.get_value()
current_stats = self.shared_array.calculate_statistics()
logging.info(f"监控报告 - 已执行操作: {current_ops}, "
f"数组统计: sum={current_stats['sum']:.2f}, "
f"mean={current_stats['mean']:.2f}")
# 停止所有处理器
logging.info("停止所有处理器...")
for processor in self.processors:
processor.join(timeout=5)
if processor.is_alive():
processor.terminate()
# 最终统计
final_ops = self.shared_counter.get_value()
final_stats = self.shared_array.calculate_statistics()
logging.info("=== 最终处理结果 ===")
logging.info(f"总执行操作数: {final_ops}")
logging.info(f"最终数组统计:")
logging.info(f" 总和: {final_stats['sum']:.2f}")
logging.info(f" 平均值: {final_stats['mean']:.2f}")
logging.info(f" 最大值: {final_stats['max']:.2f}")
logging.info(f" 最小值: {final_stats['min']:.2f}")
def demonstrate_complex_shared_structure():
"""演示复杂共享内存结构"""
logging.info("\n=== 复杂共享内存结构演示 ===")
# 使用Manager创建复杂共享对象
manager = multiprocessing.Manager()
# 共享字典
shared_dict = manager.dict()
shared_dict['count'] = 0
shared_dict['values'] = manager.list()
shared_dict['metadata'] = manager.dict()
# 共享列表
shared_list = manager.list()
# 共享命名空间
shared_namespace = manager.Namespace()
shared_namespace.total_operations = 0
shared_namespace.active_processes = 0
def worker_process(shared_dict, shared_list, namespace, worker_id):
"""工作进程"""
namespace.active_processes += 1
for i in range(10):
# 更新共享字典
shared_dict['count'] += 1
shared_dict['values'].append(worker_id * 100 + i)
# 更新共享列表
shared_list.append(f"Worker-{worker_id}-Task-{i}")
# 更新命名空间
namespace.total_operations += 1
time.sleep(0.1)
namespace.active_processes -= 1
logging.info(f"Worker-{worker_id} 完成")
# 启动多个工作进程
processes = []
for i in range(3):
p = Process(target=worker_process,
args=(shared_dict, shared_list, shared_namespace, i+1))
processes.append(p)
p.start()
# 等待所有进程完成
for p in processes:
p.join()
# 显示共享数据
logging.info(f"共享字典计数: {shared_dict['count']}")
logging.info(f"共享字典值数量: {len(shared_dict['values'])}")
logging.info(f"共享列表长度: {len(shared_list)}")
logging.info(f"命名空间总操作数: {shared_namespace.total_operations}")
def demonstrate_memory_efficiency():
"""演示共享内存的效率优势"""
logging.info("\n=== 共享内存效率测试 ===")
# 使用Manager创建共享数据
manager = multiprocessing.Manager()
shared_data = manager.list(range(10000))
def process_shared_data(data_slice, result_queue):
"""处理共享数据"""
process_id = os.getpid()
start_time = time.time()
# 计算数据切片的总和
total = sum(data_slice)
processing_time = time.time() - start_time
result = {
'process_id': process_id,
'total': total,
'processing_time': processing_time,
'data_size': len(data_slice)
}
result_queue.put(result)
# 创建结果队列
result_queue = manager.Queue()
# 分割数据并创建进程
data_size = len(shared_data)
chunk_size = data_size // 4
processes = []
start_time = time.time()
for i in range(4):
start_idx = i * chunk_size
end_idx = start_idx + chunk_size if i < 3 else data_size
data_slice = shared_data[start_idx:end_idx]
p = Process(target=process_shared_data,
args=(data_slice, result_queue))
processes.append(p)
p.start()
# 收集结果
total_sum = 0
for p in processes:
p.join()
while not result_queue.empty():
result = result_queue.get()
total_sum += result['total']
logging.info(f"进程 {result['process_id']}: "
f"处理 {result['data_size']} 个数据, "
f"部分和 {result['total']}, "
f"耗时 {result['processing_time']:.4f}s")
total_time = time.time() - start_time
expected_sum = sum(range(10000))
logging.info(f"共享内存处理结果:")
logging.info(f" 计算总和: {total_sum}")
logging.info(f" 期望总和: {expected_sum}")
logging.info(f" 结果正确: {total_sum == expected_sum}")
logging.info(f" 总处理时间: {total_time:.4f}s")
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
try:
# 基本共享内存演示
manager = SharedMemoryManager(array_size=1000, num_processors=4)
manager.initialize_data()
manager.create_processors()
manager.start_processing(duration=8)
# 复杂共享结构演示
demonstrate_complex_shared_structure()
# 内存效率演示
demonstrate_memory_efficiency()
except KeyboardInterrupt:
logging.info("程序被用户中断")
except Exception as e:
logging.error(f"程序执行出错: {str(e)}")
---
04.多进程性能分析
a.进程 vs 线程性能对比
a.功能说明
通过对比测试分析多进程和多线程在不同类型任务中的性能表现，帮助理解它们的适用场景。CPU密集型任务中多进程通常表现更好，而I/O密集型任务中多线程可能更优。
b.代码示例
---
# 进程与线程性能对比示例
import multiprocessing
import threading
import time
import logging
import os
import random
import math
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from multiprocessing import cpu_count
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class PerformanceBenchmark:
"""性能基准测试类"""
def __init__(self):
self.cpu_count = cpu_count()
self.results = {}
def cpu_intensive_task(self, n):
"""CPU密集型任务：计算素数"""
def is_prime(num):
if num < 2:
return False
for i in range(2, int(math.sqrt(num)) + 1):
if num % i == 0:
return False
return True
primes = []
for i in range(2, n):
if is_prime(i):
primes.append(i)
return len(primes)
def io_intensive_task(self, delay_range=(0.1, 0.5)):
"""I/O密集型任务：模拟文件读取和网络请求"""
delay = random.uniform(*delay_range)
time.sleep(delay)
return delay
def mixed_task(self, complexity=1000):
"""混合型任务：包含CPU计算和I/O等待"""
# CPU计算部分
result = 0
for i in range(complexity):
result += math.sqrt(i)
# I/O等待部分
time.sleep(0.1)
return result
def benchmark_single_thread(self, task_func, num_tasks, task_args=None):
"""单线程基准测试"""
logging.info(f"单线程执行 {num_tasks} 个任务...")
start_time = time.time()
results = []
for i in range(num_tasks):
args = task_args[i] if task_args else []
result = task_func(*args)
results.append(result)
execution_time = time.time() - start_time
throughput = num_tasks / execution_time
return {
'execution_time': execution_time,
'throughput': throughput,
'results': results
}
def benchmark_multi_thread(self, task_func, num_tasks, num_workers, task_args=None):
"""多线程基准测试"""
logging.info(f"多线程执行 {num_tasks} 个任务，工作线程数: {num_workers}")
start_time = time.time()
results = []
with ThreadPoolExecutor(max_workers=num_workers) as executor:
futures = []
for i in range(num_tasks):
args = task_args[i] if task_args else []
future = executor.submit(task_func, *args)
futures.append(future)
for future in futures:
result = future.result()
results.append(result)
execution_time = time.time() - start_time
throughput = num_tasks / execution_time
return {
'execution_time': execution_time,
'throughput': throughput,
'results': results
}
def benchmark_multi_process(self, task_func, num_tasks, num_workers, task_args=None):
"""多进程基准测试"""
logging.info(f"多进程执行 {num_tasks} 个任务，工作进程数: {num_workers}")
start_time = time.time()
results = []
with ProcessPoolExecutor(max_workers=num_workers) as executor:
futures = []
for i in range(num_tasks):
args = task_args[i] if task_args else []
future = executor.submit(task_func, *args)
futures.append(future)
for future in futures:
result = future.result()
results.append(result)
execution_time = time.time() - start_time
throughput = num_tasks / execution_time
return {
'execution_time': execution_time,
'throughput': throughput,
'results': results
}
def run_comprehensive_benchmark(self):
"""运行综合性能基准测试"""
logging.info("=== 开始综合性能基准测试 ===")
logging.info(f"系统CPU核心数: {self.cpu_count}")
logging.info(f"当前进程PID: {os.getpid()}")
# 测试配置
num_tasks = 20
worker_counts = [2, 4, 8]
# 任务1: CPU密集型任务
logging.info("\n--- 任务1: CPU密集型任务（素数计算） ---")
cpu_args = [(1000,) for _ in range(num_tasks)]
# 单线程基准
single_thread_cpu = self.benchmark_single_thread(
self.cpu_intensive_task, num_tasks, cpu_args
)
self.results['cpu_single'] = single_thread_cpu
logging.info(f"单线程: 耗时 {single_thread_cpu['execution_time']:.3f}s, "
f"吞吐量 {single_thread_cpu['throughput']:.2f} tasks/s")
# 多线程测试
for workers in worker_counts:
multi_thread_cpu = self.benchmark_multi_thread(
self.cpu_intensive_task, num_tasks, workers, cpu_args
)
self.results[f'cpu_thread_{workers}'] = multi_thread_cpu
speedup = single_thread_cpu['execution_time'] / multi_thread_cpu['execution_time']
efficiency = speedup / workers
logging.info(f"多线程({workers}线程): 耗时 {multi_thread_cpu['execution_time']:.3f}s, "
f"吞吐量 {multi_thread_cpu['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
# 多进程测试
for workers in worker_counts:
multi_process_cpu = self.benchmark_multi_process(
self.cpu_intensive_task, num_tasks, workers, cpu_args
)
self.results[f'cpu_process_{workers}'] = multi_process_cpu
speedup = single_thread_cpu['execution_time'] / multi_process_cpu['execution_time']
efficiency = speedup / workers
logging.info(f"多进程({workers}进程): 耗时 {multi_process_cpu['execution_time']:.3f}s, "
f"吞吐量 {multi_process_cpu['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
# 任务2: I/O密集型任务
logging.info("\n--- 任务2: I/O密集型任务（模拟延迟） ---")
# 单线程基准
single_thread_io = self.benchmark_single_thread(
self.io_intensive_task, num_tasks
)
self.results['io_single'] = single_thread_io
logging.info(f"单线程: 耗时 {single_thread_io['execution_time']:.3f}s, "
f"吞吐量 {single_thread_io['throughput']:.2f} tasks/s")
# 多线程测试
for workers in worker_counts:
multi_thread_io = self.benchmark_multi_thread(
self.io_intensive_task, num_tasks, workers
)
self.results[f'io_thread_{workers}'] = multi_thread_io
speedup = single_thread_io['execution_time'] / multi_thread_io['execution_time']
efficiency = speedup / workers
logging.info(f"多线程({workers}线程): 耗时 {multi_thread_io['execution_time']:.3f}s, "
f"吞吐量 {multi_thread_io['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
# 多进程测试
for workers in worker_counts:
multi_process_io = self.benchmark_multi_process(
self.io_intensive_task, num_tasks, workers
)
self.results[f'io_process_{workers}'] = multi_process_io
speedup = single_thread_io['execution_time'] / multi_process_io['execution_time']
efficiency = speedup / workers
logging.info(f"多进程({workers}进程): 耗时 {multi_process_io['execution_time']:.3f}s, "
f"吞吐量 {multi_process_io['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
# 任务3: 混合型任务
logging.info("\n--- 任务3: 混合型任务 ---")
mixed_args = [(500,) for _ in range(num_tasks)]
# 单线程基准
single_thread_mixed = self.benchmark_single_thread(
self.mixed_task, num_tasks, mixed_args
)
self.results['mixed_single'] = single_thread_mixed
logging.info(f"单线程: 耗时 {single_thread_mixed['execution_time']:.3f}s, "
f"吞吐量 {single_thread_mixed['throughput']:.2f} tasks/s")
# 多线程测试
for workers in worker_counts:
multi_thread_mixed = self.benchmark_multi_thread(
self.mixed_task, num_tasks, workers, mixed_args
)
self.results[f'mixed_thread_{workers}'] = multi_thread_mixed
speedup = single_thread_mixed['execution_time'] / multi_thread_mixed['execution_time']
efficiency = speedup / workers
logging.info(f"多线程({workers}线程): 耗时 {multi_thread_mixed['execution_time']:.3f}s, "
f"吞吐量 {multi_thread_mixed['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
# 多进程测试
for workers in worker_counts:
multi_process_mixed = self.benchmark_multi_process(
self.mixed_task, num_tasks, workers, mixed_args
)
self.results[f'mixed_process_{workers}'] = multi_process_mixed
speedup = single_thread_mixed['execution_time'] / multi_process_mixed['execution_time']
efficiency = speedup / workers
logging.info(f"多进程({workers}进程): 耗时 {multi_process_mixed['execution_time']:.3f}s, "
f"吞吐量 {multi_process_mixed['throughput']:.2f} tasks/s, "
f"加速比 {speedup:.2f}x, 效率 {efficiency:.2%}")
def analyze_results(self):
"""分析和总结测试结果"""
logging.info("\n=== 性能分析结果 ===")
# CPU密集型任务分析
logging.info("\n--- CPU密集型任务分析 ---")
cpu_single = self.results['cpu_single']
best_thread_key = max([k for k in self.results.keys() if 'cpu_thread' in k],
key=lambda k: self.results[k]['throughput'])
best_process_key = max([k for k in self.results.keys() if 'cpu_process' in k],
key=lambda k: self.results[k]['throughput'])
best_thread = self.results[best_thread_key]
best_process = self.results[best_process_key]
thread_speedup = cpu_single['throughput'] / best_thread['throughput']
process_speedup = cpu_single['throughput'] / best_process['throughput']
logging.info(f"CPU密集型任务:")
logging.info(f" 单线程吞吐量: {cpu_single['throughput']:.2f} tasks/s")
logging.info(f" 最佳多线程({best_thread_key}): {best_thread['throughput']:.2f} tasks/s, "
f"加速比 {thread_speedup:.2f}x")
logging.info(f" 最佳多进程({best_process_key}): {best_process['throughput']:.2f} tasks/s, "
f"加速比 {process_speedup:.2f}x")
if process_speedup > thread_speedup:
logging.info(" 结论: 多进程在CPU密集型任务中表现更优")
else:
logging.info(" 结论: 多线程在CPU密集型任务中表现更优")
# I/O密集型任务分析
logging.info("\n--- I/O密集型任务分析 ---")
io_single = self.results['io_single']
best_thread_key = max([k for k in self.results.keys() if 'io_thread' in k],
key=lambda k: self.results[k]['throughput'])
best_process_key = max([k for k in self.results.keys() if 'io_process' in k],
key=lambda k: self.results[k]['throughput'])
best_thread = self.results[best_thread_key]
best_process = self.results[best_process_key]
thread_speedup = io_single['throughput'] / best_thread['throughput']
process_speedup = io_single['throughput'] / best_process['throughput']
logging.info(f"I/O密集型任务:")
logging.info(f" 单线程吞吐量: {io_single['throughput']:.2f} tasks/s")
logging.info(f" 最佳多线程({best_thread_key}): {best_thread['throughput']:.2f} tasks/s, "
f"加速比 {thread_speedup:.2f}x")
logging.info(f" 最佳多进程({best_process_key}): {best_process['throughput']:.2f} tasks/s, "
f"加速比 {process_speedup:.2f}x")
if thread_speedup > process_speedup:
logging.info(" 结论: 多线程在I/O密集型任务中表现更优")
else:
logging.info(" 结论: 多进程在I/O密集型任务中表现更优")
# 混合型任务分析
logging.info("\n--- 混合型任务分析 ---")
mixed_single = self.results['mixed_single']
best_thread_key = max([k for k in self.results.keys() if 'mixed_thread' in k],
key=lambda k: self.results[k]['throughput'])
best_process_key = max([k for k in self.results.keys() if 'mixed_process' in k],
key=lambda k: self.results[k]['throughput'])
best_thread = self.results[best_thread_key]
best_process = self.results[best_process_key]
thread_speedup = mixed_single['throughput'] / best_thread['throughput']
process_speedup = mixed_single['throughput'] / best_process['throughput']
logging.info(f"混合型任务:")
logging.info(f" 单线程吞吐量: {mixed_single['throughput']:.2f} tasks/s")
logging.info(f" 最佳多线程({best_thread_key}): {best_thread['throughput']:.2f} tasks/s, "
f"加速比 {thread_speedup:.2f}x")
logging.info(f" 最佳多进程({best_process_key}): {best_process['throughput']:.2f} tasks/s, "
f"加速比 {process_speedup:.2f}x")
if thread_speedup > process_speedup:
logging.info(" 结论: 多线程在混合型任务中表现更优")
else:
logging.info(" 结论: 多进程在混合型任务中表现更优")
# 总体建议
logging.info("\n=== 总体建议 ===")
logging.info("1. CPU密集型任务：推荐使用多进程，可充分利用多核CPU")
logging.info("2. I/O密集型任务：推荐使用多线程，创建开销小，上下文切换快")
logging.info("3. 混合型任务：根据具体任务特征选择，可能需要混合模式")
logging.info("4. 内存使用：多进程内存消耗大，多线程共享内存")
logging.info("5. 开发复杂度：多线程开发相对简单，但需要注意同步问题")
def main():
"""主函数"""
benchmark = PerformanceBenchmark()
benchmark.run_comprehensive_benchmark()
benchmark.analyze_results()
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
main()
---
b.进程数量优化
a.功能说明
根据系统资源和任务特性优化进程数量是提升多进程性能的关键。过多的进程会导致上下文切换开销和内存压力，过少则无法充分利用硬件资源。需要通过实际测试找到最佳进程数。
b.代码示例
---
# 进程数量优化示例
import multiprocessing
import time
import logging
import os
import psutil
import random
import math
from concurrent.futures import ProcessPoolExecutor, as_completed
from multiprocessing import cpu_count
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - P%(process)d - %(levelname)s - %(message)s'
)
class ProcessCountOptimizer:
"""进程数量优化器"""
def __init__(self):
self.cpu_count = cpu_count()
self.logical_cores = psutil.cpu_count(logical=True)
self.physical_cores = psutil.cpu_count(logical=False)
self.memory_gb = psutil.virtual_memory().total / (1024**3)
def get_system_info(self):
"""获取系统信息"""
return {
'cpu_count': self.cpu_count,
'logical_cores': self.logical_cores,
'physical_cores': self.physical_cores,
'memory_gb': self.memory_gb,
'cpu_usage': psutil.cpu_percent(interval=1)
}
def cpu_intensive_task(self, complexity=50000):
"""CPU密集型任务"""
result = 0
for i in range(complexity):
result += math.sqrt(i) + math.sin(i)
return result
def memory_intensive_task(self, data_size=100000):
"""内存密集型任务"""
# 创建大量数据
data = [random.random() for _ in range(data_size)]
# 进行一些计算
return sum(data) / len(data)
def mixed_task(self, cpu_complexity=25000, data_size=50000):
"""混合型任务"""
# CPU部分
result = 0
for i in range(cpu_complexity):
result += math.sqrt(i)
# 内存部分
data = [random.random() for _ in range(data_size)]
memory_result = sum(data)
return result + memory_result
def benchmark_process_count(self, task_func, task_args_list, max_workers_list):
"""测试不同进程数的性能"""
results = []
num_tasks = len(task_args_list)
for max_workers in max_workers_list:
logging.info(f"测试进程数: {max_workers}")
# 记录开始时间和系统资源
start_time = time.time()
start_memory = psutil.virtual_memory().used / (1024**3) # GB
# 执行任务
completed_tasks = 0
total_results = []
try:
with ProcessPoolExecutor(max_workers=max_workers) as executor:
# 提交所有任务
future_to_args = {
executor.submit(task_func, *args): args
for args in task_args_list
}
# 收集结果
for future in as_completed(future_to_args):
try:
result = future.result(timeout=30)
total_results.append(result)
completed_tasks += 1
except Exception as e:
logging.error(f"任务执行失败: {str(e)}")
except Exception as e:
logging.error(f"进程池执行出错: {str(e)}")
continue
# 记录结束时间和系统资源
execution_time = time.time() - start_time
end_memory = psutil.virtual_memory().used / (1024**3) # GB
memory_usage = end_memory - start_memory
# 计算性能指标
throughput = completed_tasks / execution_time
avg_task_time = execution_time / completed_tasks if completed_tasks > 0 else 0
result = {
'max_workers': max_workers,
'execution_time': execution_time,
'throughput': throughput,
'completed_tasks': completed_tasks,
'total_tasks': num_tasks,
'success_rate': completed_tasks / num_tasks,
'avg_task_time': avg_task_time,
'memory_usage_gb': memory_usage,
'cpu_utilization': psutil.cpu_percent(interval=1)
}
results.append(result)
logging.info(f"进程数 {max_workers}: 耗时 {execution_time:.3f}s, "
f"吞吐量 {throughput:.2f} tasks/s, "
f"成功率 {result['success_rate']:.2%}, "
f"内存使用 {memory_usage:.2f}GB")
return results
def analyze_optimal_process_count(self, results):
"""分析最优进程数"""
if not results:
return None
# 找出吞吐量最高的配置
best_throughput = max(results, key=lambda x: x['throughput'])
# 找出执行时间最短的配置
best_time = min(results, key=lambda x: x['execution_time'])
# 找出内存效率最高的配置
best_memory = min(results, key=lambda x: x['memory_usage_gb'])
analysis = {
'best_throughput': best_throughput,
'best_time': best_time,
'best_memory': best_memory,
'throughput_score': best_throughput['throughput'],
'time_score': best_time['execution_time'],
'memory_score': best_memory['memory_usage_gb']
}
return analysis
def calculate_scaling_efficiency(self, results):
"""计算扩展效率"""
if not results:
return {}
# 单进程作为基准
baseline = results[0]
scaling_results = []
for result in results:
if result['max_workers'] == 1:
result['speedup'] = 1.0
result['efficiency'] = 1.0
else:
ideal_speedup = result['max_workers']
actual_speedup = baseline['execution_time'] / result['execution_time']
result['speedup'] = actual_speedup
result['efficiency'] = actual_speedup / ideal_speedup
scaling_results.append(result)
return scaling_results
def optimize_for_task_type(self, task_type='cpu_intensive'):
"""为特定任务类型优化进程数"""
logging.info(f"\n=== 优化 {task_type} 任务 ===")
# 准备测试数据
num_tasks = 40
task_args_list = []
if task_type == 'cpu_intensive':
task_func = self.cpu_intensive_task
task_args_list = [(30000,) for _ in range(num_tasks)]
max_workers_list = [1, 2, 4, 6, 8, 12, 16]
elif task_type == 'memory_intensive':
task_func = self.memory_intensive_task
task_args_list = [(80000,) for _ in range(num_tasks)]
max_workers_list = [1, 2, 4, 6, 8, 10, 12]
else: # mixed
task_func = self.mixed_task
task_args_list = [(15000, 40000) for _ in range(num_tasks)]
max_workers_list = [1, 2, 4, 6, 8, 10, 12, 14]
# 运行基准测试
results = self.benchmark_process_count(task_func, task_args_list, max_workers_list)
if not results:
logging.error("测试失败，没有获得结果")
return
# 分析结果
analysis = self.analyze_optimal_process_count(results)
scaling_results = self.calculate_scaling_efficiency(results)
# 输出分析结果
logging.info(f"\n{task_type} 任务优化结果:")
if analysis:
logging.info(f"最佳吞吐量配置: 进程数 {analysis['best_throughput']['max_workers']}, "
f"吞吐量 {analysis['best_throughput']['throughput']:.2f} tasks/s")
logging.info(f"最短时间配置: 进程数 {analysis['best_time']['max_workers']}, "
f"执行时间 {analysis['best_time']['execution_time']:.3f}s")
logging.info(f"最低内存配置: 进程数 {analysis['best_memory']['max_workers']}, "
f"内存使用 {analysis['best_memory']['memory_usage_gb']:.2f}GB")
# 输出扩展效率
logging.info(f"\n扩展效率分析:")
for result in scaling_results:
if result['max_workers'] <= 8: # 只显示前几个
logging.info(f" {result['max_workers']}进程: "
f"加速比 {result['speedup']:.2f}x, "
f"效率 {result['efficiency']:.2%}")
# 推荐配置
best_overall = analysis['best_throughput'] if analysis else results[0]
recommended_processes = best_overall['max_workers']
logging.info(f"\n推荐进程数: {recommended_processes}")
# 根据任务类型给出具体建议
if task_type == 'cpu_intensive':
recommended = self.physical_cores
logging.info(f"理论建议: {recommended} (等于物理核心数)")
elif task_type == 'memory_intensive':
recommended = min(self.physical_cores,
int(self.memory_gb / 2)) # 每个进程大约2GB内存
logging.info(f"理论建议: {recommended} (考虑内存限制)")
else:
recommended = int(self.physical_cores * 1.5)
logging.info(f"理论建议: {recommended} (物理核心数的1.5倍)")
return recommended_processes
def run_comprehensive_optimization(self):
"""运行综合优化测试"""
logging.info("=== 开始进程数量综合优化测试 ===")
# 显示系统信息
system_info = self.get_system_info()
logging.info("系统信息:")
for key, value in system_info.items():
logging.info(f" {key}: {value}")
# 优化不同类型的任务
cpu_recommended = self.optimize_for_task_type('cpu_intensive')
memory_recommended = self.optimize_for_task_type('memory_intensive')
mixed_recommended = self.optimize_for_task_type('mixed')
# 综合建议
logging.info("\n=== 综合优化建议 ===")
logging.info(f"CPU密集型任务: 推荐进程数 {cpu_recommended}")
logging.info(f"内存密集型任务: 推荐进程数 {memory_recommended}")
logging.info(f"混合型任务: 推荐进程数 {mixed_recommended}")
# 通用建议
logging.info("\n通用优化原则:")
logging.info("1. CPU密集型任务: 进程数 = 物理核心数")
logging.info("2. I/O密集型任务: 进程数 = 物理核心数 × 2~4")
logging.info("3. 内存密集型任务: 考虑内存限制，避免过度分配")
logging.info("4. 混合型任务: 通过实际测试确定最佳值")
logging.info("5. 监控系统资源使用情况，动态调整")
return {
'cpu_intensive': cpu_recommended,
'memory_intensive': memory_recommended,
'mixed': mixed_recommended
}
def demonstrate_adaptive_process_count():
"""演示自适应进程数量调整"""
logging.info("\n=== 自适应进程数量演示 ===")
class AdaptiveProcessPool:
"""自适应进程池"""
def __init__(self, initial_workers=2, max_workers=8):
self.current_workers = initial_workers
self.max_workers = max_workers
self.min_workers = 1
self.executor = None
self.performance_history = []
def adjust_workers(self, throughput, cpu_usage, memory_usage):
"""根据性能指标调整进程数"""
# 简单的自适应逻辑
if throughput > 10 and cpu_usage < 70 and memory_usage < 80:
# 性能良好，资源充足，可以增加进程数
if self.current_workers < self.max_workers:
self.current_workers += 1
logging.info(f"性能良好，增加进程数至 {self.current_workers}")
elif (throughput < 5 or cpu_usage > 85 or memory_usage > 90):
# 性能不佳或资源紧张，减少进程数
if self.current_workers > self.min_workers:
self.current_workers -= 1
logging.info(f"资源紧张，减少进程数至 {self.current_workers}")
return self.current_workers
def get_current_workers(self):
"""获取当前进程数"""
return self.current_workers
# 模拟自适应调整过程
adaptive_pool = AdaptiveProcessPool()
# 模拟不同负载情况
scenarios = [
{'throughput': 8, 'cpu': 60, 'memory': 50}, # 正常负载
{'throughput': 15, 'cpu': 50, 'memory': 60}, # 轻负载，资源充足
{'throughput': 3, 'cpu': 90, 'memory': 85}, # 高负载，资源紧张
{'throughput': 12, 'cpu': 40, 'memory': 45}, # 轻负载
{'throughput': 4, 'cpu': 88, 'memory': 92}, # 极高负载
]
logging.info("模拟自适应进程数量调整:")
for i, scenario in enumerate(scenarios, 1):
old_workers = adaptive_pool.get_current_workers()
new_workers = adaptive_pool.adjust_workers(
scenario['throughput'],
scenario['cpu'],
scenario['memory']
)
logging.info(f"场景{i}: 吞吐量 {scenario['throughput']}, "
f"CPU {scenario['cpu']}%, 内存 {scenario['memory']}% -> "
f"进程数 {old_workers} -> {new_workers}")
def main():
"""主函数"""
optimizer = ProcessCountOptimizer()
optimizer.run_comprehensive_optimization()
demonstrate_adaptive_process_count()
if __name__ == "__main__":
# 设置多进程启动方法
multiprocessing.set_start_method('spawn', force=True)
main()
---
8.3 协程模型
01.基本概念
a.协程定义
协程是一种特殊的函数，可以在执行过程中暂停并在适当时机恢复执行。协程通过yield关键字实现状态保存和恢复，支持协作式多任务调度，由程序自身控制调度时机而非操作系统强制调度。
b.协程特点
协程具有轻量级、高并发、无阻塞等待的特性，通过事件循环机制管理多个协程的执行。协程切换开销极小，适合处理大量I/O密集型任务，能够在单线程中实现高效并发。
c.应用场景
协程模型特别适合网络编程、Web服务器、实时通信、数据库访问等I/O密集型场景，能够在保持代码简洁性的同时实现高并发性能，是现代异步编程的核心技术。
02.Python协程基础
a.生成器协程
a.功能说明
Python早期通过生成器(yield)实现协程功能，利用yield表达式在函数间传递数据和控制权。这种方式虽然功能有限，但为理解协程原理提供了基础。
b.代码示例
---
# 生成器协程基础示例
import time
import logging
from collections import deque
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class Task:
"""基础任务类，使用生成器实现协程"""
def __init__(self, name, target_coroutine):
self.name = name
self.target = target_coroutine
self.send_value = None
self.is_finished = False
def run(self):
"""执行任务一步"""
try:
if self.send_value is not None:
returned = self.target.send(self.send_value)
self.send_value = None
else:
returned = next(self.target)
return returned
except StopIteration:
self.is_finished = True
return None
def send(self, value):
"""向协程发送数据"""
self.send_value = value
return self.run()
class SimpleScheduler:
"""简单协程调度器"""
def __init__(self):
self.tasks = deque()
self.current_task = None
def new_task(self, name, target_coroutine):
"""创建新任务"""
task = Task(name, target_coroutine)
self.tasks.append(task)
logging.info(f"创建任务: {name}")
return task
def run(self):
"""运行调度器"""
logging.info("协程调度器开始运行...")
step_count = 0
while self.tasks:
step_count += 1
# 获取下一个任务
current_task = self.tasks.popleft()
logging.debug(f"执行步骤 {step_count}: 任务 {current_task.name}")
try:
# 执行任务一步
result = current_task.run()
if result is not None:
# 如果返回值，说明是yield操作
if isinstance(result, SleepOperation):
# 处理睡眠操作
logging.info(f"任务 {current_task.name} 睡眠 {result.duration} 秒")
time.sleep(result.duration)
elif isinstance(result, GetOperation):
# 处理获取数据操作
data = self.get_data(result.key)
current_task.send(data)
elif isinstance(result, PutOperation):
# 处理存储数据操作
self.put_data(result.key, result.value)
current_task.send(None)
# 如果任务未完成，重新加入队列
if not current_task.is_finished:
self.tasks.append(current_task)
else:
logging.info(f"任务 {current_task.name} 执行完成")
except Exception as e:
logging.error(f"任务 {current_task.name} 执行出错: {str(e)}")
logging.info(f"协程调度器运行完成，总共执行 {step_count} 步")
def get_data(self, key):
"""获取数据操作"""
# 模拟数据获取
time.sleep(0.1)
return f"Data for {key}"
def put_data(self, key, value):
"""存储数据操作"""
# 模拟数据存储
time.sleep(0.1)
logging.info(f"存储数据: {key} = {value}")
class SleepOperation:
"""睡眠操作"""
def __init__(self, duration):
self.duration = duration
class GetOperation:
"""数据获取操作"""
def __init__(self, key):
self.key = key
class PutOperation:
"""数据存储操作"""
def __init__(self, key, value):
self.key = key
self.value = value
def producer_coroutine(name, items_to_produce):
"""生产者协程"""
logging.info(f"生产者 {name} 开始运行")
for i in range(items_to_produce):
# 生产数据
item = f"{name}_item_{i}"
logging.info(f"生产者 {name} 生产: {item}")
# 存储数据
result = yield PutOperation(f"key_{i}", item)
logging.info(f"生产者 {name} 存储结果: {result}")
# 模拟生产间隔
yield SleepOperation(0.2)
logging.info(f"生产者 {name} 生产完成")
def consumer_coroutine(name, items_to_consume):
"""消费者协程"""
logging.info(f"消费者 {name} 开始运行")
for i in range(items_to_consume):
# 获取数据
data = yield GetOperation(f"key_{i}")
logging.info(f"消费者 {name} 消费: {data}")
# 模拟处理时间
yield SleepOperation(0.3)
logging.info(f"消费者 {name} 消费完成")
def coroutine_demonstration():
"""协程演示主函数"""
logging.info("=== 生成器协程演示开始 ===")
# 创建调度器
scheduler = SimpleScheduler()
# 创建生产者和消费者协程
producer = producer_coroutine("Producer1", 5)
consumer = consumer_coroutine("Consumer1", 5)
# 添加任务到调度器
scheduler.new_task("ProducerTask", producer)
scheduler.new_task("ConsumerTask", consumer)
# 运行调度器
scheduler.run()
def bidirectional_communication():
"""双向通信协程演示"""
logging.info("\n=== 双向通信协程演示 ===")
def echo_coroutine():
"""回声协程"""
while True:
received = yield # 等待接收数据
if received is None: # 结束信号
break
logging.info(f"回声协程收到: {received}")
yield f"Echo: {received}"
def sender_coroutine(echo):
"""发送者协程"""
messages = ["Hello", "World", "Coroutine", "Python"]
for msg in messages:
# 发送消息并等待回声
response = echo.send(msg)
logging.info(f"发送者收到回声: {response}")
# 发送结束信号
echo.send(None)
# 创建协程对象
echo_gen = echo_coroutine()
sender_gen = sender_coroutine(echo_gen)
# 启动回声协程
next(echo_gen)
# 运行发送者协程
try:
for step in range(10): # 限制执行步骤避免无限循环
try:
next(sender_gen)
except StopIteration:
break
except Exception as e:
logging.error(f"双向通信出错: {str(e)}")
if __name__ == "__main__":
coroutine_demonstration()
bidirectional_communication()
---
b.async/await协程
a.功能说明
Python 3.5引入了async/await语法，使协程编程更加直观和易用。async定义协程函数，await暂停协程执行等待异步操作完成，配合asyncio库实现高效的异步编程。
b.代码示例
---
# async/await协程基础示例
import asyncio
import time
import logging
import random
from datetime import datetime
from typing import List, Dict, Any
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class AsyncTask:
"""异步任务基类"""
def __init__(self, name: str):
self.name = name
self.created_at = datetime.now()
self.started_at = None
self.completed_at = None
self.result = None
self.error = None
async def execute(self) -> Any:
"""异步执行方法，子类需要重写"""
raise NotImplementedError("子类必须实现execute方法")
async def run(self) -> Any:
"""运行任务并记录状态"""
self.started_at = datetime.now()
logging.info(f"任务 {self.name} 开始执行")
try:
self.result = await self.execute()
self.completed_at = datetime.now()
logging.info(f"任务 {self.name} 执行完成，结果: {self.result}")
return self.result
except Exception as e:
self.error = e
self.completed_at = datetime.now()
logging.error(f"任务 {self.name} 执行失败: {str(e)}")
raise
def get_execution_time(self) -> float:
"""获取执行时间"""
if self.started_at and self.completed_at:
return (self.completed_at - self.started_at).total_seconds()
return 0.0
class CalculationTask(AsyncTask):
"""计算任务"""
def __init__(self, name: str, numbers: List[int], operation: str = 'sum'):
super().__init__(name)
self.numbers = numbers
self.operation = operation
async def execute(self) -> float:
"""执行计算任务"""
logging.info(f"计算任务 {self.name}: 开始计算 {len(self.numbers)} 个数字")
# 模拟复杂计算，使用异步睡眠
for i, num in enumerate(self.numbers):
# 模拟每个数字的计算时间
await asyncio.sleep(0.1)
logging.debug(f"计算任务 {self.name}: 处理第 {i+1}/{len(self.numities)} 个数字")
# 执行实际计算
if self.operation == 'sum':
result = sum(self.numbers)
elif self.operation == 'average':
result = sum(self.numbers) / len(self.numbers) if self.numbers else 0
elif self.operation == 'max':
result = max(self.numbers) if self.numbers else 0
elif self.operation == 'min':
result = min(self.numbers) if self.numbers else 0
else:
raise ValueError(f"不支持的操作: {self.operation}")
return result
class NetworkTask(AsyncTask):
"""网络请求任务"""
def __init__(self, name: str, url: str, timeout: float = 5.0):
super().__init__(name)
self.url = url
self.timeout = timeout
async def execute(self) -> Dict[str, Any]:
"""模拟网络请求"""
logging.info(f"网络任务 {self.name}: 请求 {self.url}")
# 模拟网络延迟
await asyncio.sleep(random.uniform(0.5, 2.0))
# 模拟网络响应
response = {
'url': self.url,
'status_code': 200,
'content_length': random.randint(1000, 10000),
'response_time': random.uniform(0.1, 1.0),
'timestamp': datetime.now().isoformat()
}
# 模拟可能的网络错误
if random.random() < 0.1: # 10%概率失败
raise ConnectionError(f"网络请求失败: {self.url}")
return response
class DatabaseTask(AsyncTask):
"""数据库操作任务"""
def __init__(self, name: str, query: str, params: Dict[str, Any] = None):
super().__init__(name)
self.query = query
self.params = params or {}
async def execute(self) -> List[Dict[str, Any]]:
"""模拟数据库查询"""
logging.info(f"数据库任务 {self.name}: 执行查询 {self.query}")
# 模拟数据库连接和查询时间
await asyncio.sleep(random.uniform(0.2, 1.0))
# 模拟查询结果
if self.query == "SELECT * FROM users":
results = [
{'id': 1, 'name': 'Alice', 'age': 25},
{'id': 2, 'name': 'Bob', 'age': 30},
{'id': 3, 'name': 'Charlie', 'age': 35}
]
elif self.query == "SELECT COUNT(*) FROM orders":
results = [{'count': random.randint(100, 1000)}]
else:
results = [{'result': f"Query result for {self.query}"}]
return results
class AsyncTaskManager:
"""异步任务管理器"""
def __init__(self):
self.tasks = []
self.results = {}
self.start_time = None
def add_task(self, task: AsyncTask):
"""添加任务"""
self.tasks.append(task)
logging.info(f"添加任务: {task.name}")
async def run_all_sequential(self) -> Dict[str, Any]:
"""顺序执行所有任务"""
logging.info("开始顺序执行所有任务...")
self.start_time = datetime.now()
for task in self.tasks:
try:
result = await task.run()
self.results[task.name] = result
except Exception as e:
self.results[task.name] = {'error': str(e)}
total_time = (datetime.now() - self.start_time).total_seconds()
logging.info(f"顺序执行完成，总耗时: {total_time:.2f}秒")
return self.results
async def run_all_concurrent(self) -> Dict[str, Any]:
"""并发执行所有任务"""
logging.info("开始并发执行所有任务...")
self.start_time = datetime.now()
# 创建任务协程列表
task_coroutines = [task.run() for task in self.tasks]
# 使用asyncio.gather并发执行
try:
results = await asyncio.gather(*task_coroutines, return_exceptions=True)
for i, task in enumerate(self.tasks):
if isinstance(results[i], Exception):
self.results[task.name] = {'error': str(results[i])}
else:
self.results[task.name] = results[i]
except Exception as e:
logging.error(f"并发执行出错: {str(e)}")
total_time = (datetime.now() - self.start_time).total_seconds()
logging.info(f"并发执行完成，总耗时: {total_time:.2f}秒")
return self.results
async def run_with_semaphore(self, max_concurrent: int = 3) -> Dict[str, Any]:
"""使用信号量限制并发数"""
logging.info(f"开始执行任务，最大并发数: {max_concurrent}")
self.start_time = datetime.now()
semaphore = asyncio.Semaphore(max_concurrent)
async def limited_run(task):
"""限制并发的任务执行"""
async with semaphore:
return await task.run()
# 创建限制并发的任务协程列表
task_coroutines = [limited_run(task) for task in self.tasks]
try:
results = await asyncio.gather(*task_coroutines, return_exceptions=True)
for i, task in enumerate(self.tasks):
if isinstance(results[i], Exception):
self.results[task.name] = {'error': str(results[i])}
else:
self.results[task.name] = results[i]
except Exception as e:
logging.error(f"限制并发执行出错: {str(e)}")
total_time = (datetime.now() - self.start_time).total_seconds()
logging.info(f"限制并发执行完成，总耗时: {total_time:.2f}秒")
return self.results
def get_statistics(self) -> Dict[str, Any]:
"""获取执行统计信息"""
if not self.start_time:
return {}
total_time = (datetime.now() - self.start_time).total_seconds()
successful_tasks = sum(1 for r in self.results.values() if 'error' not in r)
failed_tasks = len(self.results) - successful_tasks
task_times = [task.get_execution_time() for task in self.tasks if task.get_execution_time() > 0]
avg_task_time = sum(task_times) / len(task_times) if task_times else 0
return {
'total_tasks': len(self.tasks),
'successful_tasks': successful_tasks,
'failed_tasks': failed_tasks,
'total_execution_time': total_time,
'average_task_time': avg_task_time,
'success_rate': successful_tasks / len(self.tasks) if self.tasks else 0
}
async def data_pipeline_example():
"""数据处理管道示例"""
logging.info("\n=== 数据处理管道示例 ===")
async def fetch_data(source_name: str) -> List[Dict[str, Any]]:
"""获取数据"""
logging.info(f"从 {source_name} 获取数据...")
await asyncio.sleep(1.0) # 模拟网络请求
# 模拟获取的数据
data = [
{'id': i, 'value': random.randint(1, 100), 'source': source_name}
for i in range(10)
]
logging.info(f"从 {source_name} 获取了 {len(data)} 条数据")
return data
async def process_data(data: List[Dict[str, Any]], processor_name: str) -> List[Dict[str, Any]]:
"""处理数据"""
logging.info(f"处理器 {processor_name} 开始处理 {len(data)} 条数据...")
processed_data = []
for item in data:
# 模拟数据处理
await asyncio.sleep(0.1)
processed_item = {
**item,
'processed_by': processor_name,
'processed_at': datetime.now().isoformat(),
'processed_value': item['value'] * 2
}
processed_data.append(processed_item)
logging.info(f"处理器 {processor_name} 处理完成")
return processed_data
async def save_data(data: List[Dict[str, Any]], destination: str) -> bool:
"""保存数据"""
logging.info(f"保存 {len(data)} 条数据到 {destination}...")
await asyncio.sleep(0.5) # 模拟保存操作
# 模拟可能的保存错误
if random.random() < 0.1:
raise Exception(f"保存到 {destination} 失败")
logging.info(f"数据保存成功到 {destination}")
return True
# 创建数据处理管道
start_time = time.time()
# 获取数据
sources = ["database", "api", "file"]
fetch_tasks = [fetch_data(source) for source in sources]
data_lists = await asyncio.gather(*fetch_tasks)
# 合并数据
all_data = []
for data_list in data_lists:
all_data.extend(data_list)
logging.info(f"总共获取了 {len(all_data)} 条数据")
# 并发处理数据
processors = ["filter", "transform", "validate"]
process_tasks = [process_data(all_data, processor) for processor in processors]
processed_results = await asyncio.gather(*process_tasks)
# 保存处理后的数据
destinations = ["output1.json", "output2.json", "output3.json"]
save_tasks = [save_data(processed_results[i], dest)
for i, dest in enumerate(destinations)]
save_results = await asyncio.gather(*save_tasks, return_exceptions=True)
end_time = time.time()
successful_saves = sum(1 for r in save_results if r is True)
logging.info(f"数据处理管道完成:")
logging.info(f" 总耗时: {end_time - start_time:.2f}秒")
logging.info(f" 处理数据: {len(all_data)} 条")
logging.info(f" 成功保存: {successful_saves}/{len(destinations)} 个文件")
async def demonstrate_async_patterns():
"""演示异步编程模式"""
logging.info("\n=== 异步编程模式演示 ===")
# 创建各种类型的异步任务
tasks = [
CalculationTask("sum_task", list(range(1, 101)), 'sum'),
CalculationTask("avg_task", list(range(1, 51)), 'average'),
NetworkTask("httpbin_task", "https://httpbin.org/get"),
NetworkTask("api_task", "https://jsonplaceholder.typicode.com/posts/1"),
DatabaseTask("users_task", "SELECT * FROM users"),
DatabaseTask("orders_task", "SELECT COUNT(*) FROM orders")
]
# 创建任务管理器
manager = AsyncTaskManager()
for task in tasks:
manager.add_task(task)
# 顺序执行
logging.info("--- 顺序执行 ---")
sequential_results = await manager.run_all_sequential()
sequential_stats = manager.get_statistics()
logging.info(f"顺序执行统计: {sequential_stats}")
# 重置管理器
manager = AsyncTaskManager()
for task in tasks:
manager.add_task(task)
# 并发执行
logging.info("\n--- 并发执行 ---")
concurrent_results = await manager.run_all_concurrent()
concurrent_stats = manager.get_statistics()
logging.info(f"并发执行统计: {concurrent_stats}")
# 限制并发执行
logging.info("\n--- 限制并发执行 (max_concurrent=3) ---")
manager = AsyncTaskManager()
for task in tasks:
manager.add_task(task)
limited_results = await manager.run_with_semaphore(max_concurrent=3)
limited_stats = manager.get_statistics()
logging.info(f"限制并发统计: {limited_stats}")
async def main():
"""主函数"""
# 演示异步编程模式
await demonstrate_async_patterns()
# 演示数据处理管道
await data_pipeline_example()
if __name__ == "__main__":
# 运行异步主函数
asyncio.run(main())
---
03.异步编程框架
a.asyncio框架详解
a.功能说明
asyncio是Python的官方异步编程框架，提供事件循环、协程调度、同步原语等功能。它支持高并发网络编程，通过事件循环机制实现非阻塞I/O操作，是现代Python异步编程的基础。
b.代码示例
---
# asyncio框架详解示例
import asyncio
import time
import logging
import random
import socket
import ssl
from datetime import datetime
from typing import Dict, List, Any, Optional, Callable
asyncio.create_task()
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class AsyncEventLoopManager:
"""异步事件循环管理器"""
def __init__(self):
self.loop = None
self.running_tasks = set()
self.background_tasks = set()
self.is_running = False
async def setup_loop(self):
"""设置事件循环"""
self.loop = asyncio.get_running_loop()
logging.info(f"设置事件循环: {self.loop}")
# 设置事件循环的调试模式
if hasattr(self.loop, 'set_debug'):
self.loop.set_debug(True)
# 添加信号处理
if hasattr(asyncio, 'unix_add_signal_handler'):
try:
import signal
for sig in (signal.SIGINT, signal.SIGTERM):
self.loop.add_signal_handler(sig, self._signal_handler)
except (ImportError, NotImplementedError):
pass
def _signal_handler(self):
"""信号处理器"""
logging.info("收到终止信号，正在关闭...")
self.is_running = False
async def create_server(self, host: str, port: int, handler: Callable):
"""创建异步服务器"""
server = await asyncio.start_server(
self._handle_client, host, port,
reuse_address=True, reuse_port=True
)
logging.info(f"服务器启动在 {host}:{port}")
return server
async def _handle_client(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter):
"""处理客户端连接"""
client_addr = writer.get_extra_info('peername')
logging.info(f"客户端连接: {client_addr}")
try:
while True:
# 读取客户端数据
data = await reader.read(1024)
if not data:
break
message = data.decode('utf-8')
logging.info(f"收到 {client_addr} 消息: {message}")
# 处理消息
response = await self._process_message(message, client_addr)
# 发送响应
writer.write(response.encode('utf-8'))
await writer.drain()
except Exception as e:
logging.error(f"处理客户端 {client_addr} 时出错: {str(e)}")
finally:
writer.close()
await writer.wait_closed()
logging.info(f"客户端 {client_addr} 断开连接")
async def _process_message(self, message: str, client_addr: tuple) -> str:
"""处理客户端消息"""
# 模拟处理时间
await asyncio.sleep(0.1)
if message.lower() == 'ping':
return 'pong'
elif message.lower() == 'time':
return f"Current time: {datetime.now().isoformat()}"
elif message.lower() == 'quit':
return 'Goodbye!'
else:
return f"Echo: {message}"
async def create_periodic_task(self, coro: Callable, interval: float):
"""创建周期性任务"""
while self.is_running:
try:
await coro()
await asyncio.sleep(interval)
except asyncio.CancelledError:
break
except Exception as e:
logging.error(f"周期性任务出错: {str(e)}")
await asyncio.sleep(interval)
async def run_with_timeout(self, coro, timeout: float):
"""带超时运行协程"""
try:
return await asyncio.wait_for(coro, timeout=timeout)
except asyncio.TimeoutError:
logging.warning(f"协程执行超时 ({timeout}秒)")
raise
async def gather_with_semaphore(self, coros, semaphore):
"""使用信号量限制并发数"""
async def limited_coro(coro):
async with semaphore:
return await coro
return await asyncio.gather(*(limited_coro(coro) for coro in coros))
class AsyncHTTPClient:
"""异步HTTP客户端"""
def __init__(self):
self.session = None
async def __aenter__(self):
self.session = aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout(total=10),
connector=aiohttp.TCPConnector(limit=100, limit_per_host=10)
)
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
if self.session:
await self.session.close()
async def get(self, url: str, **kwargs) -> Dict[str, Any]:
"""发送GET请求"""
if not self.session:
raise RuntimeError("客户端未初始化")
start_time = time.time()
try:
async with self.session.get(url, **kwargs) as response:
content = await response.text()
response_time = time.time() - start_time
return {
'url': url,
'status_code': response.status,
'content_length': len(content),
'response_time': response_time,
'headers': dict(response.headers),
'content': content[:1000] # 只返回前1000字符
}
except Exception as e:
response_time = time.time() - start_time
return {
'url': url,
'error': str(e),
'response_time': response_time
}
async def post(self, url: str, data: Dict[str, Any], **kwargs) -> Dict[str, Any]:
"""发送POST请求"""
if not self.session:
raise RuntimeError("客户端未初始化")
start_time = time.time()
try:
async with self.session.post(url, json=data, **kwargs) as response:
content = await response.text()
response_time = time.time() - start_time
return {
'url': url,
'method': 'POST',
'status_code': response.status,
'content_length': len(content),
'response_time': response_time,
'data': data,
'content': content[:1000]
}
except Exception as e:
response_time = time.time() - start_time
return {
'url': url,
'method': 'POST',
'error': str(e),
'response_time': response_time
}
class AsyncWorkerPool:
"""异步工作池"""
def __init__(self, max_workers: int = 10):
self.max_workers = max_workers
self.queue = asyncio.Queue()
self.workers = []
self.results = {}
self.is_running = False
async def start(self):
"""启动工作池"""
self.is_running = True
self.workers = [
asyncio.create_task(self._worker(f"worker-{i}"))
for i in range(self.max_workers)
]
logging.info(f"异步工作池启动，工作线程数: {self.max_workers}")
async def stop(self):
"""停止工作池"""
self.is_running = False
# 发送停止信号
for _ in range(self.max_workers):
await self.queue.put(None)
# 等待所有工作线程完成
await asyncio.gather(*self.workers, return_exceptions=True)
logging.info("异步工作池已停止")
async def _worker(self, name: str):
"""工作线程"""
logging.info(f"工作线程 {name} 启动")
while self.is_running:
try:
# 获取任务，设置超时避免无限等待
task = await asyncio.wait_for(self.queue.get(), timeout=1.0)
if task is None: # 停止信号
break
# 执行任务
await self._process_task(task, name)
except asyncio.TimeoutError:
continue
except Exception as e:
logging.error(f"工作线程 {name} 处理任务时出错: {str(e)}")
logging.info(f"工作线程 {name} 停止")
async def _process_task(self, task: Dict[str, Any], worker_name: str):
"""处理单个任务"""
task_id = task['id']
task_type = task['type']
task_data = task['data']
try:
start_time = time.time()
logging.info(f"工作线程 {worker_name} 开始处理任务 {task_id} ({task_type})")
if task_type == 'http_request':
result = await self._handle_http_request(task_data)
elif task_type == 'calculation':
result = await self._handle_calculation(task_data)
elif task_type == 'file_processing':
result = await self._handle_file_processing(task_data)
else:
result = {'error': f'未知任务类型: {task_type}'}
execution_time = time.time() - start_time
self.results[task_id] = {
'task_id': task_id,
'task_type': task_type,
'worker': worker_name,
'result': result,
'execution_time': execution_time,
'completed_at': datetime.now().isoformat()
}
logging.info(f"任务 {task_id} 完成，耗时: {execution_time:.3f}秒")
except Exception as e:
self.results[task_id] = {
'task_id': task_id,
'task_type': task_type,
'worker': worker_name,
'error': str(e),
'completed_at': datetime.now().isoformat()
}
async def _handle_http_request(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理HTTP请求任务"""
# 这里应该实现实际的HTTP请求逻辑
# 为了演示，我们模拟一个HTTP请求
await asyncio.sleep(random.uniform(0.5, 2.0))
return {
'url': data.get('url', 'http://example.com'),
'status_code': 200,
'response_size': random.randint(1000, 10000)
}
async def _handle_calculation(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理计算任务"""
# 模拟CPU密集型计算
numbers = data.get('numbers', [])
operation = data.get('operation', 'sum')
await asyncio.sleep(0.1) # 模拟计算时间
if operation == 'sum':
result = sum(numbers)
elif operation == 'average':
result = sum(numbers) / len(numbers) if numbers else 0
elif operation == 'max':
result = max(numbers) if numbers else 0
else:
result = 0
return {
'operation': operation,
'input_count': len(numbers),
'result': result
}
async def _handle_file_processing(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理文件处理任务"""
# 模拟文件处理
file_path = data.get('file_path', '/tmp/test.txt')
operation = data.get('operation', 'read')
await asyncio.sleep(random.uniform(0.2, 1.0))
return {
'file_path': file_path,
'operation': operation,
'file_size': random.randint(1024, 10240),
'processed': True
}
async def submit_task(self, task_type: str, task_data: Dict[str, Any]) -> str:
"""提交任务"""
task_id = f"task_{int(time.time() * 1000)}_{random.randint(1000, 9999)}"
task = {
'id': task_id,
'type': task_type,
'data': task_data,
'submitted_at': datetime.now().isoformat()
}
await self.queue.put(task)
logging.info(f"提交任务: {task_id} ({task_type})")
return task_id
def get_result(self, task_id: str) -> Optional[Dict[str, Any]]:
"""获取任务结果"""
return self.results.get(task_id)
def get_all_results(self) -> Dict[str, Any]:
"""获取所有结果"""
return self.results
async def demonstrate_asyncio_features():
"""演示asyncio框架特性"""
logging.info("=== asyncio框架特性演示 ===")
# 创建事件循环管理器
loop_manager = AsyncEventLoopManager()
await loop_manager.setup_loop()
# 1. 演示并发协程执行
logging.info("\n--- 1. 并发协程执行 ---")
async def quick_task(name: str, duration: float):
"""快速任务"""
logging.info(f"任务 {name} 开始")
await asyncio.sleep(duration)
logging.info(f"任务 {name} 完成")
return f"{name}_result"
# 并发执行多个任务
tasks = [
quick_task("task1", 0.5),
quick_task("task2", 0.3),
quick_task("task3", 0.7),
quick_task("task4", 0.2)
]
start_time = time.time()
results = await asyncio.gather(*tasks)
total_time = time.time() - start_time
logging.info(f"并发执行完成，总耗时: {total_time:.2f}秒，结果: {results}")
# 2. 演示超时控制
logging.info("\n--- 2. 超时控制 ---")
async def slow_task():
"""慢任务"""
await asyncio.sleep(2.0)
return "slow_task_completed"
try:
result = await asyncio.wait_for(slow_task(), timeout=1.0)
logging.info(f"慢任务结果: {result}")
except asyncio.TimeoutError:
logging.warning("慢任务超时")
# 3. 演示事件循环调度
logging.info("\n--- 3. 事件循环调度 ---")
current_loop = asyncio.get_running_loop()
logging.info(f"当前事件循环: {current_loop}")
logging.info(f"是否运行: {current_loop.is_running()}")
# 4. 演示队列和生产者-消费者模式
logging.info("\n--- 4. 生产者-消费者模式 ---")
queue = asyncio.Queue(maxsize=5)
async def producer(name: str, items: int):
"""生产者"""
for i in range(items):
item = f"{name}_item_{i}"
await queue.put(item)
logging.info(f"生产者 {name} 生产: {item}")
await asyncio.sleep(0.1)
await queue.put(None) # 结束信号
logging.info(f"生产者 {name} 完成生产")
async def consumer(name: str):
"""消费者"""
while True:
item = await queue.get()
if item is None: # 结束信号
queue.task_done()
break
logging.info(f"消费者 {name} 消费: {item}")
await asyncio.sleep(0.2)
queue.task_done()
# 创建生产者和消费者
producers = [producer(f"prod_{i}", 3) for i in range(2)]
consumers = [consumer(f"cons_{i}") for i in range(2)]
# 并发运行
await asyncio.gather(*producers, *consumers)
async def demonstrate_async_worker_pool():
"""演示异步工作池"""
logging.info("\n=== 异步工作池演示 ===")
# 创建并启动工作池
worker_pool = AsyncWorkerPool(max_workers=3)
await worker_pool.start()
try:
# 提交各种类型的任务
task_ids = []
# HTTP请求任务
for i in range(3):
task_id = await worker_pool.submit_task('http_request', {
'url': f'https://jsonplaceholder.typicode.com/posts/{i+1}',
'method': 'GET'
})
task_ids.append(task_id)
# 计算任务
for i in range(3):
task_id = await worker_pool.submit_task('calculation', {
'numbers': list(range(1, 100)),
'operation': random.choice(['sum', 'average', 'max'])
})
task_ids.append(task_id)
# 文件处理任务
for i in range(2):
task_id = await worker_pool.submit_task('file_processing', {
'file_path': f'/tmp/test_file_{i}.txt',
'operation': 'process'
})
task_ids.append(task_id)
# 等待所有任务完成
logging.info("等待所有任务完成...")
await asyncio.sleep(5.0) # 给任务足够的时间完成
# 显示结果
all_results = worker_pool.get_all_results()
completed_tasks = sum(1 for r in all_results.values() if 'error' not in r)
failed_tasks = len(all_results) - completed_tasks
logging.info(f"任务完成情况: 成功 {completed_tasks}, 失败 {failed_tasks}")
# 显示一些示例结果
for i, (task_id, result) in enumerate(all_results.items()):
if i < 3: # 只显示前3个结果
logging.info(f"任务 {task_id}: {result}")
finally:
await worker_pool.stop()
async def main():
"""主函数"""
# 演示asyncio特性
await demonstrate_asyncio_features()
# 演示异步工作池
await demonstrate_async_worker_pool()
# 注意：需要安装aiohttp库来运行HTTP客户端示例
try:
import aiohttp
except ImportError:
logging.warning("需要安装aiohttp库: pip install aiohttp")
# 创建模拟的aiohttp模块以避免导入错误
class MockAiohttp:
class ClientSession:
def __init__(self, **kwargs):
pass
async def __aenter__(self):
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
pass
async def close(self):
pass
async def get(self, url, **kwargs):
class MockResponse:
status = 200
text = self.async_text
async def async_text(self):
return "Mock response"
return MockResponse()
async def post(self, url, **kwargs):
return await self.get(url)
class ClientTimeout:
def __init__(self, **kwargs):
pass
class TCPConnector:
def __init__(self, **kwargs):
pass
import sys
sys.modules['aiohttp'] = MockAiohttp()
if __name__ == "__main__":
# 运行异步主函数
asyncio.run(main())
---
b.第三方异步框架
a.功能说明
除了asyncio，Python生态中还有许多优秀的异步框架，如Tornado、aiohttp、FastAPI等。这些框架提供了更高级的异步编程功能，简化了Web开发和网络编程的复杂性。
b.代码示例
---
# 第三方异步框架示例
import asyncio
import time
import logging
import random
import json
from datetime import datetime
from typing import Dict, List, Any, Optional
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# 模拟Tornado异步Web框架特性
class TornadoAsyncHandler:
"""模拟Tornado异步处理器"""
def __init__(self, name: str):
self.name = name
self.request_count = 0
async def handle_request(self, method: str, path: str, data: Dict[str, Any] = None) -> Dict[str, Any]:
"""处理HTTP请求"""
self.request_count += 1
request_id = f"{self.name}_{self.request_count}_{int(time.time())}"
logging.info(f"处理器 {self.name} 处理请求 {request_id}: {method} {path}")
try:
# 模拟请求处理时间
await asyncio.sleep(random.uniform(0.1, 0.5))
# 根据路径选择处理逻辑
if path == '/api/users':
result = await self._handle_users(method, data)
elif path == '/api/orders':
result = await self._handle_orders(method, data)
elif path == '/api/data':
result = await self._handle_data(method, data)
else:
result = {'error': 'Not Found', 'status': 404}
return {
'request_id': request_id,
'method': method,
'path': path,
'status': result.get('status', 200),
'data': result,
'timestamp': datetime.now().isoformat()
}
except Exception as e:
logging.error(f"处理请求 {request_id} 时出错: {str(e)}")
return {
'request_id': request_id,
'method': method,
'path': path,
'status': 500,
'error': str(e),
'timestamp': datetime.now().isoformat()
}
async def _handle_users(self, method: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理用户相关请求"""
if method == 'GET':
# 模拟获取用户列表
await asyncio.sleep(0.2)
return {
'status': 200,
'users': [
{'id': 1, 'name': 'Alice', 'email': '[email protected]'},
{'id': 2, 'name': 'Bob', 'email': '[email protected]'},
{'id': 3, 'name': 'Charlie', 'email': '[email protected]'}
]
}
elif method == 'POST':
# 模拟创建用户
await asyncio.sleep(0.3)
user_id = random.randint(1000, 9999)
return {
'status': 201,
'user': {
'id': user_id,
'name': data.get('name'),
'email': data.get('email'),
'created_at': datetime.now().isoformat()
}
}
else:
return {'status': 405, 'error': 'Method Not Allowed'}
async def _handle_orders(self, method: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理订单相关请求"""
if method == 'GET':
# 模拟获取订单列表
await asyncio.sleep(0.15)
return {
'status': 200,
'orders': [
{'id': 1, 'user_id': 1, 'total': 99.99, 'status': 'completed'},
{'id': 2, 'user_id': 2, 'total': 149.99, 'status': 'pending'},
{'id': 3, 'user_id': 3, 'total': 79.99, 'status': 'shipped'}
]
}
elif method == 'POST':
# 模拟创建订单
await asyncio.sleep(0.4)
order_id = random.randint(1000, 9999)
return {
'status': 201,
'order': {
'id': order_id,
'user_id': data.get('user_id'),
'items': data.get('items', []),
'total': sum(item.get('price', 0) for item in data.get('items', [])),
'created_at': datetime.now().isoformat()
}
}
else:
return {'status': 405, 'error': 'Method Not Allowed'}
async def _handle_data(self, method: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""处理数据相关请求"""
if method == 'GET':
# 模拟数据查询
query_params = data or {}
limit = query_params.get('limit', 10)
offset = query_params.get('offset', 0)
await asyncio.sleep(0.1)
return {
'status': 200,
'data': [f"data_item_{i}" for i in range(offset, offset + limit)],
'total': 1000,
'limit': limit,
'offset': offset
}
else:
return {'status': 405, 'error': 'Method Not Allowed'}
class FastAPIAsyncHandler:
"""模拟FastAPI异步处理器"""
def __init__(self):
self.endpoints = {}
def route(self, path: str, methods: List[str] = None):
"""路由装饰器"""
def decorator(func):
if path not in self.endpoints:
self.endpoints[path] = {}
for method in methods or ['GET']:
self.endpoints[path][method] = func
return func
return decorator
async def handle_request(self, method: str, path: str, data: Dict[str, Any] = None) -> Dict[str, Any]:
"""处理请求"""
if path in self.endpoints and method in self.endpoints[path]:
handler = self.endpoints[path][method]
try:
result = await handler(data or {})
return {
'status': 200,
'data': result,
'path': path,
'method': method
}
except Exception as e:
return {
'status': 500,
'error': str(e),
'path': path,
'method': method
}
else:
return {
'status': 404,
'error': 'Endpoint not found',
'path': path,
'method': method
}
class AsyncWebServer:
"""异步Web服务器模拟"""
def __init__(self):
self.handlers = {}
self.middleware = []
self.request_count = 0
def add_handler(self, path: str, handler: TornadoAsyncHandler):
"""添加处理器"""
self.handlers[path] = handler
logging.info(f"添加处理器: {path}")
def add_middleware(self, middleware_func):
"""添加中间件"""
self.middleware.append(middleware_func)
logging.info(f"添加中间件: {middleware_func.__name__}")
async def process_request(self, method: str, path: str, data: Dict[str, Any] = None) -> Dict[str, Any]:
"""处理请求"""
self.request_count += 1
request_id = f"req_{self.request_count}_{int(time.time())}"
# 应用中间件
context = {'request_id': request_id, 'method': method, 'path': path, 'data': data}
for middleware in self.middleware:
context = await middleware(context)
if context.get('stop_processing'):
return context.get('response')
# 查找处理器
handler = None
for handler_path, handler_obj in self.handlers.items():
if path.startswith(handler_path):
handler = handler_obj
break
if handler:
response = await handler.handle_request(method, path, data)
else:
response = {
'status': 404,
'error': 'Handler not found',
'path': path
}
response['request_id'] = request_id
return response
class AsyncDatabasePool:
"""异步数据库连接池模拟"""
def __init__(self, max_connections: int = 10):
self.max_connections = max_connections
self.available_connections = asyncio.Queue(maxsize=max_connections)
self.used_connections = set()
self.total_connections = 0
# 初始化连接池
for i in range(max_connections):
self.available_connections.put_nowait(f"connection_{i}")
self.total_connections += 1
async def get_connection(self) -> str:
"""获取数据库连接"""
connection = await self.available_connections.get()
self.used_connections.add(connection)
logging.debug(f"获取连接: {connection}")
return connection
async def release_connection(self, connection: str):
"""释放数据库连接"""
if connection in self.used_connections:
self.used_connections.remove(connection)
await self.available_connections.put(connection)
logging.debug(f"释放连接: {connection}")
async def execute_query(self, query: str, params: Dict[str, Any] = None) -> List[Dict[str, Any]]:
"""执行数据库查询"""
connection = await self.get_connection()
try:
# 模拟查询执行时间
await asyncio.sleep(random.uniform(0.1, 0.5))
# 模拟查询结果
if "SELECT * FROM users" in query:
result = [
{'id': 1, 'name': 'Alice', 'age': 25},
{'id': 2, 'name': 'Bob', 'age': 30},
{'id': 3, 'name': 'Charlie', 'age': 35}
]
elif "SELECT COUNT(*)" in query:
result = [{'count': random.randint(100, 1000)}]
else:
result = [{'result': f"Query executed: {query}"}]
logging.info(f"执行查询: {query} (连接: {connection})")
return result
finally:
await self.release_connection(connection)
def get_pool_stats(self) -> Dict[str, Any]:
"""获取连接池统计信息"""
return {
'total_connections': self.total_connections,
'available_connections': self.available_connections.qsize(),
'used_connections': len(self.used_connections),
'max_connections': self.max_connections
}
class AsyncCacheManager:
"""异步缓存管理器"""
def __init__(self):
self.cache = {}
self.expiry_times = {}
self.cleanup_task = None
async def start_cleanup_task(self):
"""启动清理任务"""
self.cleanup_task = asyncio.create_task(self._cleanup_expired())
logging.info("缓存清理任务已启动")
async def stop_cleanup_task(self):
"""停止清理任务"""
if self.cleanup_task:
self.cleanup_task.cancel()
try:
await self.cleanup_task
except asyncio.CancelledError:
pass
logging.info("缓存清理任务已停止")
async def _cleanup_expired(self):
"""清理过期缓存"""
while True:
try:
await asyncio.sleep(10) # 每10秒清理一次
current_time = time.time()
expired_keys = [
key for key, expiry_time in self.expiry_times.items()
if expiry_time < current_time
]
for key in expired_keys:
del self.cache[key]
del self.expiry_times[key]
logging.debug(f"清理过期缓存: {key}")
if expired_keys:
logging.info(f"清理了 {len(expired_keys)} 个过期缓存项")
except asyncio.CancelledError:
break
async def get(self, key: str) -> Optional[Any]:
"""获取缓存值"""
if key in self.cache:
expiry_time = self.expiry_times.get(key, 0)
if time.time() < expiry_time:
logging.debug(f"缓存命中: {key}")
return self.cache[key]
else:
# 缓存已过期，清理
del self.cache[key]
del self.expiry_times[key]
logging.debug(f"缓存过期: {key}")
logging.debug(f"缓存未命中: {key}")
return None
async def set(self, key: str, value: Any, ttl: int = 300) -> None:
"""设置缓存值"""
self.cache[key] = value
self.expiry_times[key] = time.time() + ttl
logging.debug(f"设置缓存: {key} (TTL: {ttl}s)")
async def delete(self, key: str) -> bool:
"""删除缓存"""
if key in self.cache:
del self.cache[key]
if key in self.expiry_times:
del self.expiry_times[key]
logging.debug(f"删除缓存: {key}")
return True
return False
def get_stats(self) -> Dict[str, Any]:
"""获取缓存统计信息"""
current_time = time.time()
expired_count = sum(
1 for expiry_time in self.expiry_times.values()
if expiry_time < current_time
)
return {
'total_items': len(self.cache),
'expired_items': expired_count,
'memory_usage': len(str(self.cache))
}
async def logging_middleware(context: Dict[str, Any]) -> Dict[str, Any]:
"""日志中间件"""
start_time = time.time()
# 模拟请求处理
await asyncio.sleep(0.01)
processing_time = time.time() - start_time
logging.info(f"请求 {context['request_id']}: {context['method']} {context['path']} "
f"耗时 {processing_time:.3f}s")
context['processing_time'] = processing_time
return context
async def auth_middleware(context: Dict[str, Any]) -> Dict[str, Any]:
"""认证中间件"""
# 模拟认证检查
await asyncio.sleep(0.005)
# 简单的认证逻辑
if context.get('data', {}).get('auth_token') == 'valid_token':
context['user_id'] = 'user_123'
logging.info(f"用户认证成功: {context['request_id']}")
else:
response = {
'status': 401,
'error': 'Unauthorized',
'request_id': context['request_id']
}
context['response'] = response
context['stop_processing'] = True
logging.warning(f"用户认证失败: {context['request_id']}")
return context
async def demonstrate_async_frameworks():
"""演示异步框架特性"""
logging.info("=== 异步框架特性演示 ===")
# 1. 模拟Tornado异步Web框架
logging.info("\n--- 1. Tornado异步Web框架 ---")
web_server = AsyncWebServer()
# 添加中间件
web_server.add_middleware(logging_middleware)
web_server.add_middleware(auth_middleware)
# 添加处理器
api_handler = TornadoAsyncHandler("api_handler")
web_server.add_handler("/api", api_handler)
# 模拟请求
requests = [
('GET', '/api/users', {'auth_token': 'valid_token'}),
('POST', '/api/users', {'auth_token': 'valid_token', 'name': 'New User', 'email': '[email protected]'}),
('GET', '/api/orders', {'auth_token': 'valid_token'}),
('GET', '/api/data', {'auth_token': 'valid_token', 'limit': 5}),
('GET', '/api/users', {'auth_token': 'invalid_token'}) # 认证失败
]
for method, path, data in requests:
response = await web_server.process_request(method, path, data)
logging.info(f"响应: {response['status']} - {response.get('error', 'Success')}")
# 2. 模拟FastAPI异步处理器
logging.info("\n--- 2. FastAPI异步处理器 ---")
fastapi_handler = FastAPIAsyncHandler()
@fastapi_handler.route("/health", ["GET"])
async def health_check(data):
return {"status": "healthy", "timestamp": datetime.now().isoformat()}
@fastapi_handler.route("/users/{user_id}", ["GET"])
async def get_user(data):
user_id = data.get('user_id', 1)
await asyncio.sleep(0.1) # 模拟数据库查询
return {
"id": user_id,
"name": f"User {user_id}",
"email": f"user{user_id}@example.com"
}
# 测试FastAPI风格的路由
test_requests = [
("GET", "/health"),
("GET", "/users/123"),
("GET", "/nonexistent")
]
for method, path in test_requests:
response = await fastapi_handler.handle_request(method, path)
logging.info(f"FastAPI {method} {path}: {response['status']}")
# 3. 异步数据库连接池
logging.info("\n--- 3. 异步数据库连接池 ---")
db_pool = AsyncDatabasePool(max_connections=5)
# 并发执行多个数据库查询
queries = [
"SELECT * FROM users",
"SELECT COUNT(*) FROM orders",
"SELECT * FROM products",
"SELECT COUNT(*) FROM users",
"SELECT * FROM categories"
]
async def execute_query_with_pool(query):
return await db_pool.execute_query(query)
start_time = time.time()
results = await asyncio.gather(*(execute_query_with_pool(q) for q in queries))
total_time = time.time() - start_time
logging.info(f"并发执行 {len(queries)} 个查询完成，总耗时: {total_time:.2f}秒")
logging.info(f"连接池统计: {db_pool.get_pool_stats()}")
# 4. 异步缓存管理器
logging.info("\n--- 4. 异步缓存管理器 ---")
cache_manager = AsyncCacheManager()
await cache_manager.start_cleanup_task()
# 测试缓存操作
await cache_manager.set("user:123", {"name": "Alice", "age": 25}, ttl=5)
await cache_manager.set("config:app", {"debug": True, "version": "1.0"}, ttl=10)
# 获取缓存
user_data = await cache_manager.get("user:123")
config_data = await cache_manager.get("config:app")
non_existent = await cache_manager.get("nonexistent")
logging.info(f"缓存命中测试: user={bool(user_data)}, config={bool(config_data)}, nonexistent={bool(non_existent)}")
logging.info(f"缓存统计: {cache_manager.get_stats()}")
# 等待一些缓存过期
await asyncio.sleep(6)
# 再次检查缓存
user_data_after = await cache_manager.get("user:123")
config_data_after = await cache_manager.get("config:app")
logging.info(f"过期检查: user={bool(user_data_after)}, config={bool(config_data_after)}")
await cache_manager.stop_cleanup_task()
async def main():
"""主函数"""
await demonstrate_async_frameworks()
if __name__ == "__main__":
# 运行异步主函数
asyncio.run(main())
---
04.协程性能优化
a.协程 vs 线程性能对比
a.功能说明
协程相比线程具有更轻量级的特性，创建和切换成本极低。通过性能对比测试可以直观了解协程在高并发场景下的优势，特别是在I/O密集型任务中的表现。
b.代码示例
---
# 协程与线程性能对比示例
import asyncio
import threading
import multiprocessing
import time
import logging
import random
import socket
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
from datetime import datetime
from typing import List, Dict, Any
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class PerformanceBenchmark:
"""性能基准测试类"""
def __init__(self):
self.results = {}
async def async_io_task(self, task_id: int, delay_range: tuple = (0.1, 0.5)) -> Dict[str, Any]:
"""异步I/O任务"""
start_time = time.time()
# 模拟I/O操作
delay = random.uniform(*delay_range)
await asyncio.sleep(delay)
# 模拟一些CPU计算
result = sum(range(100))
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'delay': delay,
'type': 'async_io'
}
def sync_io_task(self, task_id: int, delay_range: tuple = (0.1, 0.5)) -> Dict[str, Any]:
"""同步I/O任务"""
start_time = time.time()
# 模拟I/O操作
delay = random.uniform(*delay_range)
time.sleep(delay)
# 模拟一些CPU计算
result = sum(range(100))
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'delay': delay,
'type': 'sync_io'
}
def thread_io_task(self, task_id: int, delay_range: tuple = (0.1, 0.5)) -> Dict[str, Any]:
"""线程I/O任务"""
start_time = time.time()
# 模拟I/O操作
delay = random.uniform(*delay_range)
time.sleep(delay)
# 模拟一些CPU计算
result = sum(range(100))
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'delay': delay,
'type': 'thread_io'
}
async def async_cpu_task(self, task_id: int, complexity: int = 100000) -> Dict[str, Any]:
"""异步CPU任务"""
start_time = time.time()
# CPU密集型计算
result = 0
for i in range(complexity):
result += i ** 2
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'complexity': complexity,
'type': 'async_cpu'
}
def sync_cpu_task(self, task_id: int, complexity: int = 100000) -> Dict[str, Any]:
"""同步CPU任务"""
start_time = time.time()
# CPU密集型计算
result = 0
for i in range(complexity):
result += i ** 2
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'complexity': complexity,
'type': 'sync_cpu'
}
def thread_cpu_task(self, task_id: int, complexity: int = 100000) -> Dict[str, Any]:
"""线程CPU任务"""
start_time = time.time()
# CPU密集型计算
result = 0
for i in range(complexity):
result += i ** 2
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'result': result,
'complexity': complexity,
'type': 'thread_cpu'
}
async def benchmark_async_concurrent(self, task_func, num_tasks: int, **kwargs) -> Dict[str, Any]:
"""异步并发基准测试"""
logging.info(f"异步并发测试: {num_tasks} 个任务")
start_time = time.time()
# 创建任务列表
tasks = [task_func(i, **kwargs) for i in range(num_tasks)]
# 并发执行
results = await asyncio.gather(*tasks)
total_time = time.time() - start_time
return {
'method': 'async_concurrent',
'num_tasks': num_tasks,
'total_time': total_time,
'throughput': num_tasks / total_time,
'results': results,
'avg_execution_time': sum(r['execution_time'] for r in results) / len(results)
}
def benchmark_thread_concurrent(self, task_func, num_tasks: int, max_workers: int = None, **kwargs) -> Dict[str, Any]:
"""线程并发基准测试"""
logging.info(f"线程并发测试: {num_tasks} 个任务，最大线程数: {max_workers}")
start_time = time.time()
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# 提交任务
futures = [executor.submit(task_func, i, **kwargs) for i in range(num_tasks)]
# 收集结果
results = [future.result() for future in futures]
total_time = time.time() - start_time
return {
'method': 'thread_concurrent',
'num_tasks': num_tasks,
'max_workers': max_workers,
'total_time': total_time,
'throughput': num_tasks / total_time,
'results': results,
'avg_execution_time': sum(r['execution_time'] for r in results) / len(results)
}
def benchmark_process_concurrent(self, task_func, num_tasks: int, max_workers: int = None, **kwargs) -> Dict[str, Any]:
"""进程并发基准测试"""
logging.info(f"进程并发测试: {num_tasks} 个任务，最大进程数: {max_workers}")
start_time = time.time()
with ProcessPoolExecutor(max_workers=max_workers) as executor:
# 提交任务
futures = [executor.submit(task_func, i, **kwargs) for i in range(num_tasks)]
# 收集结果
results = [future.result() for future in futures]
total_time = time.time() - start_time
return {
'method': 'process_concurrent',
'num_tasks': num_tasks,
'max_workers': max_workers,
'total_time': total_time,
'throughput': num_tasks / total_time,
'results': results,
'avg_execution_time': sum(r['execution_time'] for r in results) / len(results)
}
def benchmark_sync_sequential(self, task_func, num_tasks: int, **kwargs) -> Dict[str, Any]:
"""同步顺序基准测试"""
logging.info(f"同步顺序测试: {num_tasks} 个任务")
start_time = time.time()
results = []
for i in range(num_tasks):
result = task_func(i, **kwargs)
results.append(result)
total_time = time.time() - start_time
return {
'method': 'sync_sequential',
'num_tasks': num_tasks,
'total_time': total_time,
'throughput': num_tasks / total_time,
'results': results,
'avg_execution_time': sum(r['execution_time'] for r in results) / len(results)
}
async def run_comprehensive_benchmark(self, num_tasks: int = 50):
"""运行综合性能基准测试"""
logging.info("=== 协程与线程性能对比开始 ===")
# 测试场景配置
test_scenarios = [
{
'name': 'I/O密集型任务',
'async_func': self.async_io_task,
'sync_func': self.sync_io_task,
'thread_func': self.thread_io_task,
'kwargs': {'delay_range': (0.1, 0.3)}
},
{
'name': 'CPU密集型任务',
'async_func': self.async_cpu_task,
'sync_func': self.sync_cpu_task,
'thread_func': self.thread_cpu_task,
'kwargs': {'complexity': 50000}
}
]
for scenario in test_scenarios:
logging.info(f"\n--- {scenario['name']} ---")
# 1. 同步顺序执行
sync_result = self.benchmark_sync_sequential(
scenario['sync_func'], num_tasks, **scenario['kwargs']
)
self.results[f"{scenario['name']}_sync_sequential"] = sync_result
# 2. 异步并发执行
async_result = await self.benchmark_async_concurrent(
scenario['async_func'], num_tasks, **scenario['kwargs']
)
self.results[f"{scenario['name']}_async_concurrent"] = async_result
# 3. 线程并发执行
thread_result = self.benchmark_thread_concurrent(
scenario['thread_func'], num_tasks, max_workers=10, **scenario['kwargs']
)
self.results[f"{scenario['name']}_thread_concurrent"] = thread_result
# 4. 进程并发执行（仅CPU密集型）
if 'CPU' in scenario['name']:
process_result = self.benchmark_process_concurrent(
scenario['thread_func'], min(num_tasks, 20), max_workers=4, **scenario['kwargs']
)
self.results[f"{scenario['name']}_process_concurrent"] = process_result
def analyze_results(self):
"""分析和对比测试结果"""
logging.info("\n=== 性能分析结果 ===")
# I/O密集型任务分析
io_sync = self.results.get('I/O密集型任务_sync_sequential')
io_async = self.results.get('I/O密集型任务_async_concurrent')
io_thread = self.results.get('I/O密集型任务_thread_concurrent')
if io_sync and io_async and io_thread:
logging.info("\n--- I/O密集型任务性能对比 ---")
logging.info(f"同步顺序: 吞吐量 {io_sync['throughput']:.2f} tasks/s, "
f"总耗时 {io_sync['total_time']:.2f}s")
logging.info(f"异步并发: 吞吐量 {io_async['throughput']:.2f} tasks/s, "
f"总耗时 {io_async['total_time']:.2f}s")
logging.info(f"线程并发: 吞吐量 {io_thread['throughput']:.2f} tasks/s, "
f"总耗时 {io_thread['total_time']:.2f}s")
# 计算性能提升
async_speedup = io_async['throughput'] / io_sync['throughput']
thread_speedup = io_thread['throughput'] / io_sync['throughput']
logging.info(f"异步并发提升: {async_speedup:.2f}x")
logging.info(f"线程并发提升: {thread_speedup:.2f}x")
if async_speedup > thread_speedup:
logging.info("结论: I/O密集型任务中，异步协程性能最优")
else:
logging.info("结论: I/O密集型任务中，多线程性能最优")
# CPU密集型任务分析
cpu_sync = self.results.get('CPU密集型任务_sync_sequential')
cpu_async = self.results.get('CPU密集型任务_async_concurrent')
cpu_thread = self.results.get('CPU密集型任务_thread_concurrent')
cpu_process = self.results.get('CPU密集型任务_process_concurrent')
if cpu_sync and cpu_async and cpu_thread:
logging.info("\n--- CPU密集型任务性能对比 ---")
logging.info(f"同步顺序: 吞吐量 {cpu_sync['throughput']:.2f} tasks/s, "
f"总耗时 {cpu_sync['total_time']:.2f}s")
logging.info(f"异步并发: 吞吐量 {cpu_async['throughput']:.2f} tasks/s, "
f"总耗时 {cpu_async['total_time']:.2f}s")
logging.info(f"线程并发: 吞吐量 {cpu_thread['throughput']:.2f} tasks/s, "
f"总耗时 {cpu_thread['total_time']:.2f}s")
# 计算性能提升
async_speedup = cpu_async['throughput'] / cpu_sync['throughput']
thread_speedup = cpu_thread['throughput'] / cpu_sync['throughput']
logging.info(f"异步并发提升: {async_speedup:.2f}x")
logging.info(f"线程并发提升: {thread_speedup:.2f}x")
if cpu_process:
process_speedup = cpu_process['throughput'] / cpu_sync['throughput']
logging.info(f"进程并发: 吞吐量 {cpu_process['throughput']:.2f} tasks/s, "
f"总耗时 {cpu_process['total_time']:.2f}s")
logging.info(f"进程并发提升: {process_speedup:.2f}x")
logging.info("结论: CPU密集型任务中，多进程性能最优，协程和线程受GIL限制")
# 内存使用分析
logging.info("\n--- 内存使用对比 ---")
logging.info("协程: 内存使用极少，每个协程约几KB")
logging.info("线程: 内存使用适中，每个线程约几MB")
logging.info("进程: 内存使用最多，每个进程约几十MB")
logging.info("结论: 协程在高并发场景下内存效率最优")
async def demonstrate_scalability(self):
"""演示可扩展性"""
logging.info("\n=== 可扩展性演示 ===")
task_counts = [10, 50, 100, 500]
for count in task_counts:
logging.info(f"\n--- 任务数量: {count} ---")
# 异步协程
start_time = time.time()
async_tasks = [self.async_io_task(i, (0.01, 0.05)) for i in range(count)]
await asyncio.gather(*async_tasks)
async_time = time.time() - start_time
# 多线程
start_time = time.time()
with ThreadPoolExecutor(max_workers=min(count, 50)) as executor:
futures = [executor.submit(self.thread_io_task, i, (0.01, 0.05)) for i in range(count)]
[future.result() for future in futures]
thread_time = time.time() - start_time
logging.info(f"异步协程: {async_time:.2f}s, 吞吐量 {count/async_time:.2f} tasks/s")
logging.info(f"多线程: {thread_time:.2f}s, 吞吐量 {count/thread_time:.2f} tasks/s")
if async_time < thread_time:
efficiency = (thread_time - async_time) / thread_time * 100
logging.info(f"协程优势: 效率提升 {efficiency:.1f}%")
async def main(self):
"""主函数"""
# 运行性能基准测试
await self.run_comprehensive_benchmark(num_tasks=100)
# 分析结果
self.analyze_results()
# 演示可扩展性
await self.demonstrate_scalability()
if __name__ == "__main__":
# 运行性能测试
benchmark = PerformanceBenchmark()
asyncio.run(benchmark.main())
---
b.协程数量优化
a.功能说明
虽然协程创建成本很低，但在实际应用中仍需合理控制并发协程数量。过多的协程可能导致内存压力、调度延迟和资源竞争，需要通过性能测试找到最佳并发数量。
b.代码示例
---
# 协程数量优化示例
import asyncio
import time
import logging
import random
import psutil
import socket
from typing import List, Dict, Any, Optional
from datetime import datetime
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
class ConcurrencyOptimizer:
"""并发数量优化器"""
def __init__(self):
self.system_info = self.get_system_info()
self.test_results = []
def get_system_info(self) -> Dict[str, Any]:
"""获取系统信息"""
return {
'cpu_count': psutil.cpu_count(),
'logical_cores': psutil.cpu_count(logical=True),
'physical_cores': psutil.cpu_count(logical=False),
'memory_gb': psutil.virtual_memory().total / (1024**3),
'memory_available_gb': psutil.virtual_memory().available / (1024**3)
}
async def io_intensive_task(self, task_id: int, delay_range: tuple = (0.1, 0.5)) -> Dict[str, Any]:
"""I/O密集型任务"""
start_time = time.time()
# 模拟I/O操作
delay = random.uniform(*delay_range)
await asyncio.sleep(delay)
# 模拟网络请求
try:
reader, writer = await asyncio.open_connection('httpbin.org', 80)
writer.close()
await writer.wait_closed()
except:
pass # 忽略连接错误
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'delay': delay,
'success': True
}
async def mixed_task(self, task_id: int, cpu_ops: int = 10000, io_delay: float = 0.1) -> Dict[str, Any]:
"""混合型任务"""
start_time = time.time()
# CPU计算部分
result = 0
for i in range(cpu_ops):
result += i ** 2
# I/O等待部分
await asyncio.sleep(io_delay)
execution_time = time.time() - start_time
return {
'task_id': task_id,
'execution_time': execution_time,
'cpu_ops': cpu_ops,
'io_delay': io_delay,
'result': result,
'success': True
}
async def benchmark_concurrency_level(self, task_func, num_tasks: int, concurrency_level: int) -> Dict[str, Any]:
"""测试特定并发级别"""
logging.info(f"测试并发级别: {concurrency_level}, 任务数: {num_tasks}")
# 记录系统资源使用情况
start_memory = psutil.virtual_memory().used / (1024**3)
start_cpu = psutil.cpu_percent(interval=0.1)
# 创建信号量限制并发数
semaphore = asyncio.Semaphore(concurrency_level)
async def limited_task(task_id):
async with semaphore:
return await task_func(task_id)
# 执行测试
start_time = time.time()
tasks = [limited_task(i) for i in range(num_tasks)]
results = await asyncio.gather(*tasks, return_exceptions=True)
total_time = time.time() - start_time
# 计算系统资源使用情况
end_memory = psutil.virtual_memory().used / (1024**3)
memory_usage = end_memory - start_memory
end_cpu = psutil.cpu_percent(interval=0.1)
# 统计结果
successful_tasks = sum(1 for r in results if isinstance(r, dict) and r.get('success', False))
failed_tasks = len(results) - successful_tasks
avg_execution_time = sum(r['execution_time'] for r in results if isinstance(r, dict) and r.get('success')) / successful_tasks if successful_tasks > 0 else 0
return {
'concurrency_level': concurrency_level,
'num_tasks': num_tasks,
'total_time': total_time,
'throughput': successful_tasks / total_time,
'successful_tasks': successful_tasks,
'failed_tasks': failed_tasks,
'success_rate': successful_tasks / num_tasks,
'avg_execution_time': avg_execution_time,
'memory_usage_gb': memory_usage,
'cpu_usage': (start_cpu + end_cpu) / 2
}
async def find_optimal_concurrency(self, task_func, num_tasks: int, task_type: str = 'io_intensive'):
"""寻找最优并发数量"""
logging.info(f"为 {task_type} 任务寻找最优并发数量...")
# 根据任务类型确定测试范围
if task_type == 'io_intensive':
concurrency_levels = [1, 5, 10, 20, 50, 100, 200, 500]
kwargs = {'delay_range': (0.05, 0.2)}
elif task_type == 'mixed':
concurrency_levels = [1, 5, 10, 20, 50, 100, 200]
kwargs = {'cpu_ops': 5000, 'io_delay': 0.05}
else:
concurrency_levels = [1, 5, 10, 20, 50, 100]
kwargs = {}
results = []
for level in concurrency_levels:
try:
result = await self.benchmark_concurrency_level(
lambda i: task_func(i, **kwargs), num_tasks, level
)
results.append(result)
logging.info(f"并发级别 {level}: 吞吐量 {result['throughput']:.2f} tasks/s, "
f"成功率 {result['success_rate']:.2%}, "
f"内存使用 {result['memory_usage_gb']:.2f}GB")
except Exception as e:
logging.error(f"测试并发级别 {level} 时出错: {str(e)}")
# 分析结果找出最优并发级别
if results:
# 以吞吐量为主要指标
best_throughput = max(results, key=lambda x: x['throughput'])
# 考虑内存使用效率
memory_efficient = min(results, key=lambda x: x['memory_usage_gb'])
# 考虑成功率
high_success = max(results, key=lambda x: x['success_rate'])
optimal = best_throughput # 默认选择吞吐量最优
# 如果吞吐量相近但内存使用少很多，选择内存效率高的
if (abs(best_throughput['throughput'] - memory_efficient['throughput']) / best_throughput['throughput'] < 0.05 and
memory_efficient['memory_usage_gb'] < best_throughput['memory_usage_gb'] * 0.5):
optimal = memory_efficient
# 如果成功率明显更高，选择高成功率的
if high_success['success_rate'] > optimal['success_rate'] + 0.05:
optimal = high_success
logging.info(f"\n{task_type} 任务最优并发级别分析:")
logging.info(f"最佳吞吐量: 并发 {best_throughput['concurrency_level']}, "
f"吞吐量 {best_throughput['throughput']:.2f} tasks/s")
logging.info(f"最佳内存效率: 并发 {memory_efficient['concurrency_level']}, "
f"内存使用 {memory_efficient['memory_usage_gb']:.2f}GB")
logging.info(f"最高成功率: 并发 {high_success['concurrency_level']}, "
f"成功率 {high_success['success_rate']:.2%}")
logging.info(f"推荐并发级别: {optimal['concurrency_level']} "
f"(吞吐量: {optimal['throughput']:.2f} tasks/s, "
f"内存: {optimal['memory_usage_gb']:.2f}GB)")
return optimal, results
return None, []
async def demonstrate_adaptive_concurrency(self):
"""演示自适应并发控制"""
logging.info("\n=== 自适应并发控制演示 ===")
class AdaptiveConcurrencyController:
"""自适应并发控制器"""
def __init__(self, initial_concurrency: int = 10, max_concurrency: int = 100):
self.current_concurrency = initial_concurrency
self.max_concurrency = max_concurrency
self.min_concurrency = 1
self.performance_history = []
self.adjustment_interval = 10 # 每10个任务调整一次
self.task_counter = 0
async def execute_with_adaptive_control(self, task_func, num_tasks: int):
"""使用自适应控制执行任务"""
semaphore = asyncio.Semaphore(self.current_concurrency)
completed_tasks = 0
start_time = time.time()
async def controlled_task(task_id):
async with semaphore:
result = await task_func(task_id)
# 更新性能历史
self.task_counter += 1
if self.task_counter % self.adjustment_interval == 0:
self.adjust_concurrency()
return result
# 执行任务
tasks = [controlled_task(i) for i in range(num_tasks)]
results = await asyncio.gather(*tasks, return_exceptions=True)
total_time = time.time() - start_time
successful_tasks = sum(1 for r in results if isinstance(r, dict) and r.get('success'))
return {
'total_time': total_time,
'successful_tasks': successful_tasks,
'final_concurrency': self.current_concurrency,
'performance_history': self.performance_history.copy()
}
def adjust_concurrency(self):
"""调整并发数量"""
if len(self.performance_history) < 2:
return
# 计算最近的性能趋势
recent_performance = self.performance_history[-2:]
throughput_change = (recent_performance[1]['throughput'] -
recent_performance[0]['throughput']) / recent_performance[0]['throughput']
current_memory = psutil.virtual_memory().used / (1024**3)
memory_pressure = current_memory / self.system_info['memory_gb']
old_concurrency = self.current_concurrency
# 根据性能和内存压力调整
if throughput_change > 0.1 and memory_pressure < 0.8:
# 性能提升且有内存空间，增加并发
self.current_concurrency = min(self.current_concurrency * 2, self.max_concurrency)
elif throughput_change < -0.1 or memory_pressure > 0.9:
# 性能下降或内存压力大，减少并发
self.current_concurrency = max(self.current_concurrency // 2, self.min_concurrency)
if self.current_concurrency != old_concurrency:
logging.info(f"自适应调整并发数量: {old_concurrency} -> {self.current_concurrency}")
def record_performance(self, execution_time: int, successful_tasks: int):
"""记录性能指标"""
throughput = successful_tasks / execution_time
self.performance_history.append({
'timestamp': time.time(),
'throughput': throughput,
'concurrency': self.current_concurrency
})
# 保持历史记录大小
if len(self.performance_history) > 20:
self.performance_history.pop(0)
# 创建自适应控制器
controller = AdaptiveConcurrencyController(initial_concurrency=5, max_concurrency=50)
# 模拟任务负载变化
num_tasks = 100
logging.info("开始自适应并发控制测试...")
result = await controller.execute_with_adaptive_control(
lambda i: self.io_intensive_task(i, (0.05, 0.15)), num_tasks
)
logging.info(f"自适应控制测试完成:")
logging.info(f" 最终并发级别: {result['final_concurrency']}")
logging.info(f" 成功任务数: {result['successful_tasks']}/{num_tasks}")
logging.info(f" 总执行时间: {result['total_time']:.2f}s")
async def load_balancing_demo(self):
"""负载均衡演示"""
logging.info("\n=== 负载均衡演示 ===")
class LoadBalancer:
"""简单的负载均衡器"""
def __init__(self, workers: List[str]):
self.workers = workers
self.current_worker = 0
self.worker_stats = {worker: {'tasks': 0, 'response_time': 0} for worker in workers}
def get_next_worker(self) -> str:
"""轮询获取下一个工作节点"""
worker = self.workers[self.current_worker]
self.current_worker = (self.current_worker + 1) % len(self.workers)
return worker
async def execute_task_on_worker(self, task_func, task_id: int, worker_name: str) -> Dict[str, Any]:
"""在指定工作节点执行任务"""
start_time = time.time()
# 模拟网络延迟
await asyncio.sleep(random.uniform(0.01, 0.05))
# 执行任务
result = await task_func(task_id)
execution_time = time.time() - start_time
# 更新统计信息
self.worker_stats[worker_name]['tasks'] += 1
self.worker_stats[worker_name]['response_time'] += execution_time
result['worker'] = worker_name
result['network_latency'] = execution_time - result['execution_time']
return result
# 创建负载均衡器
workers = [f"worker_{i}" for i in range(5)]
load_balancer = LoadBalancer(workers)
# 执行任务
num_tasks = 50
tasks = []
for i in range(num_tasks):
worker = load_balancer.get_next_worker()
task = load_balancer.execute_task_on_worker(
self.io_intensive_task, i, worker
)
tasks.append(task)
results = await asyncio.gather(*tasks, return_exceptions=True)
# 统计结果
worker_results = {}
for result in results:
if isinstance(result, dict) and 'worker' in result:
worker = result['worker']
if worker not in worker_results:
worker_results[worker] = []
worker_results[worker].append(result)
logging.info("负载均衡统计:")
for worker in workers:
stats = load_balancer.worker_stats[worker]
avg_response_time = stats['response_time'] / stats['tasks'] if stats['tasks'] > 0 else 0
logging.info(f" {worker}: 任务数 {stats['tasks']}, "
f"平均响应时间 {avg_response_time:.3f}s")
async def main(self):
"""主函数"""
logging.info("=== 协程数量优化测试开始 ===")
# 显示系统信息
logging.info("系统信息:")
for key, value in self.system_info.items():
logging.info(f" {key}: {value}")
# 为不同类型任务寻找最优并发级别
io_optimal, io_results = await self.find_optimal_concurrency(
self.io_intensive_task, 100, 'io_intensive'
)
mixed_optimal, mixed_results = await self.find_optimal_concurrency(
self.mixed_task, 50, 'mixed'
)
# 演示自适应并发控制
await self.demonstrate_adaptive_concurrency()
# 演示负载均衡
await self.load_balancing_demo()
# 总结建议
logging.info("\n=== 优化建议总结 ===")
if io_optimal:
logging.info(f"I/O密集型任务推荐并发数: {io_optimal['concurrency_level']}")
logging.info(" 建议根据网络延迟和系统资源动态调整")
if mixed_optimal:
logging.info(f"混合型任务推荐并发数: {mixed_optimal['concurrency_level']}")
logging.info(" 建议在CPU和I/O操作间找到平衡点")
logging.info("通用优化原则:")
logging.info("1. 监控系统资源使用情况，避免内存和CPU过载")
logging.info("2. 使用信号量控制并发数量，防止资源竞争")
logging.info("3. 根据任务特征选择合适的并发策略")
logging.info("4. 实施自适应控制，动态调整并发级别")
logging.info("5. 考虑使用负载均衡分散任务压力")
if __name__ == "__main__":
optimizer = ConcurrencyOptimizer()
asyncio.run(optimizer.main())
---
8.4 选择合适的并发方案
01.场景分析
a.任务特征分析
分析任务的性质是选择并发方案的第一步，需要考虑CPU密集程度、IO等待时间、内存使用量等因素。
b.性能要求评估
评估系统对响应时间、吞吐量、资源利用率的具体要求，确定性能指标的优先级。
c.复杂度考量
评估开发和维护的复杂度，包括代码复杂度、调试难度、错误处理复杂性等。
02.CPU密集型任务
a.多进程方案
a.适用场景
适用于计算密集型任务，如科学计算、图像处理、数据分析等，可以充分利用多核CPU。
b.代码示例
---
import multiprocessing
import time
import math
def cpu_intensive_task(n):
"""CPU密集型计算任务"""
result = 0
for i in range(n):
result += math.sqrt(i * math.sin(i))
return result
def process_with_multiprocessing():
"""使用多进程处理CPU密集型任务"""
numbers = [1000000, 2000000, 3000000, 4000000]
# 创建进程池
with multiprocessing.Pool(processes=4) as pool:
start_time = time.time()
results = pool.map(cpu_intensive_task, numbers)
end_time = time.time()
print(f"多进程处理时间: {end_time - start_time:.2f}秒")
print(f"计算结果: {results}")
return results
if __name__ == "__main__":
process_with_multiprocessing()
---
b.协程方案
a.适用场景
不适用于纯CPU密集型任务，因为协程在单线程中运行，无法利用多核优势。
b.性能对比
协程在CPU密集型任务中性能通常不如多进程，甚至可能比单线程顺序执行更慢。
03.IO密集型任务
a.协程方案
a.适用场景
适用于大量IO等待的任务，如网络请求、文件读写、数据库操作等，可以在等待时执行其他任务。
b.代码示例
---
import asyncio
import aiohttp
import time
async def fetch_url(session, url):
"""异步获取URL内容"""
try:
async with session.get(url) as response:
content = await response.text()
return len(content)
except Exception as e:
print(f"请求 {url} 失败: {e}")
return 0
async def process_with_asyncio():
"""使用协程处理IO密集型任务"""
urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2"
]
start_time = time.time()
# 创建HTTP会话
async with aiohttp.ClientSession() as session:
# 并发执行所有请求
tasks = [fetch_url(session, url) for url in urls]
results = await asyncio.gather(*tasks)
end_time = time.time()
print(f"协程处理时间: {end_time - start_time:.2f}秒")
print(f"获取内容长度: {results}")
return results
# 运行异步任务
asyncio.run(process_with_asyncio())
---
b.多线程方案
a.适用场景
适用于混合型任务，既有IO操作又有部分CPU计算，线程切换开销相对较小。
b.代码示例
---
import threading
import requests
import time
def fetch_url_thread(url, results, index):
"""线程方式获取URL内容"""
try:
response = requests.get(url)
results[index] = len(response.text)
except Exception as e:
print(f"请求 {url} 失败: {e}")
results[index] = 0
def process_with_threading():
"""使用多线程处理IO密集型任务"""
urls = [
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2",
"https://httpbin.org/delay/1",
"https://httpbin.org/delay/2"
]
threads = []
results = [0] * len(urls)
start_time = time.time()
# 创建并启动线程
for i, url in enumerate(urls):
thread = threading.Thread(
target=fetch_url_thread,
args=(url, results, i)
)
threads.append(thread)
thread.start()
# 等待所有线程完成
for thread in threads:
thread.join()
end_time = time.time()
print(f"多线程处理时间: {end_time - start_time:.2f}秒")
print(f"获取内容长度: {results}")
return results
process_with_threading()
---
04.混合型任务
a.多进程+协程组合
a.适用场景
适用于既有大量CPU计算又有大量IO操作的复杂应用，如Web爬虫、数据处理管道等。
b.架构设计
使用多进程处理CPU密集型部分，每个进程内部使用协程处理IO密集型操作。
c.代码示例
---
import multiprocessing
import asyncio
import aiohttp
import time
async def process_urls_batch(urls_batch):
"""协程处理一批URL"""
async with aiohttp.ClientSession() as session:
tasks = []
for url in urls_batch:
task = asyncio.create_task(fetch_single_url(session, url))
tasks.append(task)
results = await asyncio.gather(*tasks)
return results
async def fetch_single_url(session, url):
"""获取单个URL"""
try:
async with session.get(url) as response:
content = await response.text()
# 模拟一些CPU密集型处理
processed_content = content.upper()
return len(processed_content)
except Exception as e:
print(f"处理 {url} 失败: {e}")
return 0
def worker_process(urls_batch):
"""工作进程函数"""
# 在进程中运行异步事件循环
asyncio.set_event_loop(asyncio.new_event_loop())
return asyncio.run(process_urls_batch(urls_batch))
def hybrid_processing():
"""混合处理：多进程+协程"""
all_urls = [
f"https://httpbin.org/delay/{i%3+1}"
for i in range(20)
]
# 将URL分批
batch_size = 5
url_batches = [
all_urls[i:i+batch_size]
for i in range(0, len(all_urls), batch_size)
]
start_time = time.time()
# 使用进程池处理
with multiprocessing.Pool(processes=4) as pool:
results = pool.map(worker_process, url_batches)
# 合并结果
final_results = [item for sublist in results for item in sublist]
end_time = time.time()
print(f"混合处理时间: {end_time - start_time:.2f}秒")
print(f"处理URL总数: {len(final_results)}")
return final_results
if __name__ == "__main__":
hybrid_processing()
---
05.决策框架
a.选择流程图
a.第一步：判断任务类型
首先确定任务主要是CPU密集型、IO密集型还是混合型。
b.第二步：评估性能要求
分析系统对响应时间、吞吐量、资源利用率的具体要求。
c.第三步：考虑开发复杂度
评估团队技术水平和项目时间限制。
d.第四步：选择最终方案
综合考虑所有因素，选择最适合的并发方案。
b.性能对比表
a.CPU密集型任务
方案 响应时间 资源利用率 开发复杂度 推荐指数
多进程 短 高 中 ★★★★★
多线程 中 中 低 ★★★☆☆
协程 长 低 低 ★★☆☆☆
单线程 很长 低 很低 ★☆☆☆☆
b.IO密集型任务
方案 响应时间 资源利用率 开发复杂度 推荐指数
协程 短 高 中 ★★★★★
多线程 中 中 低 ★★★★☆
多进程 中 中 高 ★★★☆☆
单线程 长 低 很低 ★★☆☆☆
c.混合型任务
方案 响应时间 资源利用率 开发复杂度 推荐指数
多进程+协程 短 很高 很高 ★★★★★
多线程 中 中 低 ★★★☆☆
多进程 中 高 中 ★★★★☆
协程 中长 中 中 ★★★☆☆
06.最佳实践
a.渐进式优化
a.从简单开始
先用最简单的方案实现功能，确保代码正确性。
b.性能测试
使用实际的负载进行性能测试，识别瓶颈。
c.逐步优化
根据测试结果，选择合适的优化策略逐步改进。
b.监控与调优
a.性能监控
实时监控CPU使用率、内存使用、响应时间等关键指标。
b.动态调整
根据监控数据动态调整并发参数，如线程池大小、进程数量等。
c.代码示例
---
import threading
import time
import psutil
from concurrent.futures import ThreadPoolExecutor
class AdaptiveThreadPool:
"""自适应线程池"""
def __init__(self, min_workers=2, max_workers=10):
self.min_workers = min_workers
self.max_workers = max_workers
self.executor = None
self.current_workers = min_workers
self.monitoring = False
def start_monitoring(self):
"""启动监控线程"""
self.monitoring = True
monitor_thread = threading.Thread(target=self._monitor_performance)
monitor_thread.daemon = True
monitor_thread.start()
def _monitor_performance(self):
"""监控系统性能并调整线程数量"""
while self.monitoring:
cpu_percent = psutil.cpu_percent(interval=1)
memory_percent = psutil.virtual_memory().percent
# 根据CPU和内存使用情况调整线程数
if cpu_percent < 50 and memory_percent < 80:
# 资源充足，可以增加线程
if self.current_workers < self.max_workers:
self.current_workers += 1
self._adjust_workers()
elif cpu_percent > 80 or memory_percent > 90:
# 资源紧张，减少线程
if self.current_workers > self.min_workers:
self.current_workers -= 1
self._adjust_workers()
time.sleep(5) # 每5秒检查一次
def _adjust_workers(self):
"""调整线程池大小"""
if self.executor:
self.executor.shutdown(wait=False)
self.executor = ThreadPoolExecutor(
max_workers=self.current_workers
)
print(f"调整线程数量为: {self.current_workers}")
def submit_task(self, fn, *args, **kwargs):
"""提交任务到线程池"""
if not self.executor:
self.executor = ThreadPoolExecutor(
max_workers=self.current_workers
)
return self.executor.submit(fn, *args, **kwargs)
def shutdown(self):
"""关闭线程池和监控"""
self.monitoring = False
if self.executor:
self.executor.shutdown(wait=True)
# 使用示例
def cpu_task(duration):
"""模拟CPU任务"""
time.sleep(duration)
return f"任务完成，耗时{duration}秒"
# 创建自适应线程池
adaptive_pool = AdaptiveThreadPool(min_workers=2, max_workers=8)
adaptive_pool.start_monitoring()
# 提交任务
futures = []
for i in range(20):
future = adaptive_pool.submit_task(cpu_task, 1)
futures.append(future)
# 获取结果
results = [future.result() for future in futures]
print(f"处理结果: {len(results)}个任务完成")
# 关闭线程池
adaptive_pool.shutdown()
---