Fix: Python multiprocessing Not Working (freeze_support, Pickle Errors, Zombie Processes)

Q: How do I fix "Python multiprocessing Not Working (freeze_support, Pickle Errors, Zombie Processes)"?

How to fix Python multiprocessing not working — freeze_support error on Windows, pickle errors with lambdas, zombie processes, and Pool hanging indefinitely.

The Error

You use Python’s multiprocessing module and get one of these errors:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.
        ...
        if __name__ == '__main__':
            freeze_support()

Or:

_pickle.PicklingError: Can't pickle <function <lambda> at 0x...>: attribute lookup <lambda> on __main__ failed

Or the script hangs indefinitely with no output:

pool = multiprocessing.Pool(4)
results = pool.map(my_func, data)
# Hangs forever

Or processes finish but are left as zombie processes in ps aux.

Why This Happens

Python multiprocessing works by spawning new Python interpreter processes. The exact mechanism depends on the start method, which varies by platform: fork, spawn, or forkserver. The default has historically been fork on Linux and spawn on macOS (since 3.8) and Windows. Python 3.14 (October 2025) changed Linux’s default from fork to spawn to align with the other platforms and fix the long-standing fork-after-thread deadlock problem.

With spawn, the parent starts a fresh Python interpreter, re-imports your script in the child, and pickles the target function and arguments to send over a pipe. With fork, the kernel duplicates the parent’s address space and the child resumes from the same point — no re-import, no pickling. These two paths behave very differently, and code written assuming one often breaks under the other.

This has several consequences:

The if __name__ == "__main__" guard is required on Windows/macOS (and on Linux from 3.14 onward) — without it, spawned workers import the script and try to spawn more workers, causing the freeze_support error.
Only picklable objects can be passed between processes with spawn — lambdas, closures, locally defined functions, open file handles, and database connections cannot be pickled.
The Pool must be properly closed — leaving a Pool open causes zombie processes or hangs.
Shared state does not work like threads — each process has its own memory; modifying a global variable in a worker does not affect the parent.
Fork after threading is unsafe — if any thread held a lock at the moment of fork, the child inherits a locked mutex with no owner and deadlocks the first time it tries to acquire that lock.

The freeze_support error message is from Python’s process bootstrap detection: when a newly spawned child re-imports your script and reaches the Pool() call at module level, it tries to spawn yet more children, which try to spawn more, an infinite recursion that Python detects and aborts.

Platform and Environment Differences

The start method is the dominant variable in multiprocessing behavior. Two developers running the same code on different machines often see completely different failures because the default start method differs.

fork (Linux default until 3.13; opt-in from 3.14). Fast: copies the parent’s address space lazily with COW (copy-on-write). No pickling of function or arguments. Inherits all open file descriptors, network sockets, and database connections — which is exactly the problem. A psycopg2 connection forked into 4 workers gets used by all 4, corrupting the protocol state. CUDA contexts and any library using OpenMP/MKL thread pools also break after fork. Default until Python 3.13.

spawn (Windows, macOS, Linux 3.14+). Slow start: launches a new python interpreter, re-imports your module, unpickles arguments. Safe: workers start with a clean slate. Requires the if __name__ == "__main__": guard. Requires everything you pass to be picklable. This is the default for new Python code now.

forkserver (Unix only). Compromise: a server process is started once, then forks clean workers on demand. Faster than spawn (no full Python re-init) and safer than fork (no threads in the server). Useful for ML pipelines that need many short-lived workers.

Python 3.14 (October 2025) default change. On Linux, the default switched from fork to spawn. Code that worked on 3.13 may break on 3.14 with freeze_support errors or pickle errors. Either add the guard and ensure pickleability, or explicitly request the old behavior:

import multiprocessing
multiprocessing.set_start_method("fork", force=True)  # back to old Linux default

Pickling requirements for spawn. With spawn, every argument and the worker function must be picklable. That means:

Function must be importable by name from a module (not a lambda, not a local function inside another function).
Closures over local variables are not picklable — use functools.partial instead.
Class methods are picklable only if the class is defined at module level.
Database connections, file handles, and threading locks are not picklable.

macOS Python 3.8+. macOS switched default from fork to spawn in Python 3.8 because Apple’s frameworks are not fork-safe (Objective-C runtime, Grand Central Dispatch). Code that worked on macOS Python 3.7 broke on 3.8 with pickle errors. Always use the guard and avoid lambdas.

Jupyter Notebook and IPython. Workers can’t pickle functions defined in a notebook cell because the cell module is __main__, which the spawned child re-imports — but the child’s __main__ does not have your function. Workarounds:

Define the worker function in a separate .py file and import it.
Use ipyparallel instead of multiprocessing.
Use joblib with the loky backend, which serializes functions with cloudpickle and works in notebooks.

AWS Lambda cold starts. Lambda gives you one vCPU on most memory tiers (full vCPU around 1769 MB and above). ProcessPoolExecutor(4) on a 512 MB Lambda gets four workers fighting over one core — slower than serial. Also, /tmp is only 512 MB by default and is shared by all worker processes. Multiprocessing inside Lambda generally hurts more than it helps; use asyncio for I/O-bound concurrency instead. See AWS Lambda cold start timeout.

Container CPU affinity. A pod with resources.limits.cpu: "0.5" gets half a vCPU. os.cpu_count() returns the host’s CPU count, not your cgroup limit — so Pool(os.cpu_count()) over-subscribes catastrophically. Use len(os.sched_getaffinity(0)) on Linux to get the actual usable count.

PyTorch and CUDA. Always use spawn or forkserver with PyTorch DataLoader workers. Fork after torch.cuda.init() causes silent corruption or hard crashes. PyTorch’s recommended pattern is mp = torch.multiprocessing with set_start_method("spawn") at the top of __main__.

Process priority on Windows. Windows spawn uses CreateProcessW and inherits the parent’s priority class. Heavy workers can starve the parent’s event loop if it has a UI thread. On Linux, nice levels are inherited but workers compete equally with the parent in the scheduler.

Threading then forking. If you import a library that starts background threads (logging.handlers.QueueListener, prometheus_client, some HTTP clients) and then fork, the children inherit a process with running threads — except those threads are gone. Locks held by the vanished threads stay locked forever. Always switch to spawn if you use background threads.

Fix 1: Add the if name == “main” Guard

This is required on Windows and macOS when using spawn (the default start method):

Broken — no guard:

import multiprocessing

def worker(x):
    return x * x

pool = multiprocessing.Pool(4)
results = pool.map(worker, range(10))
print(results)

Fixed:

import multiprocessing

def worker(x):
    return x * x

if __name__ == "__main__":
    pool = multiprocessing.Pool(4)
    results = pool.map(worker, range(10))
    print(results)
    pool.close()
    pool.join()

On Linux (before 3.14), the default start method is fork (copies the parent process), which does not require this guard. But your code must run correctly on all platforms — always include the guard. From Python 3.14 onward, the guard is also required on Linux because the default became spawn.

Pro Tip: When Python spawns a new process on Windows, it starts a fresh interpreter and imports your script from the top. Without if __name__ == "__main__", the spawn code at module level runs again inside the worker, creating another Pool, which tries to spawn more workers — an infinite recursive spawn that crashes immediately.

Fix 2: Fix Pickle Errors — Replace Lambdas with Named Functions

multiprocessing passes arguments and functions between processes using pickle. Lambdas, closures over local variables, and locally defined classes cannot be pickled:

Broken — lambda cannot be pickled:

from multiprocessing import Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(lambda x: x * x, range(10))
        # PicklingError: Can't pickle <function <lambda>>

Fixed — use a named function defined at module level:

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(square, range(10))
    print(results)

Broken — closure captures local variable:

def make_multiplier(factor):
    def multiply(x):
        return x * factor  # Closes over `factor` — not picklable
    return multiply

if __name__ == "__main__":
    with Pool(4) as pool:
        double = make_multiplier(2)
        results = pool.map(double, range(10))  # PicklingError

Fixed — use functools.partial:

from multiprocessing import Pool
from functools import partial

def multiply(factor, x):
    return x * factor

if __name__ == "__main__":
    with Pool(4) as pool:
        double = partial(multiply, 2)  # partial is picklable
        results = pool.map(double, range(10))
    print(results)

Alternative — use pathos.multiprocessing which uses dill instead of pickle and supports lambdas:

pip install pathos

from pathos.multiprocessing import ProcessingPool as Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(lambda x: x * x, range(10))
    print(results)

Fix 3: Always Close and Join the Pool

Not closing the Pool leaves worker processes running as zombies:

Broken — Pool not closed:

from multiprocessing import Pool

if __name__ == "__main__":
    pool = Pool(4)
    results = pool.map(my_func, data)
    # Pool never closed — worker processes become zombies

Fixed — use a context manager (recommended):

from multiprocessing import Pool

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(my_func, data)
    # Pool automatically closed and joined on exit
    print(results)

Fixed — manual close/join:

from multiprocessing import Pool

if __name__ == "__main__":
    pool = Pool(4)
    try:
        results = pool.map(my_func, data)
    finally:
        pool.close()  # No more tasks will be submitted
        pool.join()   # Wait for all workers to finish

pool.close() tells the pool no more work will be submitted. pool.join() blocks until all workers finish. Both are needed — close() alone does not wait for workers.

Fix 4: Fix Pool Hanging Indefinitely

If pool.map() hangs forever, common causes are:

A worker raised an exception that was swallowed:

def bad_worker(x):
    if x == 5:
        raise ValueError("Bad input!")  # Exception in worker
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        # pool.map re-raises worker exceptions in the parent
        try:
            results = pool.map(bad_worker, range(10))
        except ValueError as e:
            print(f"Worker failed: {e}")

pool.map() re-raises exceptions from workers. Use pool.map_async() with error callbacks to handle exceptions without blocking:

def handle_error(e):
    print(f"Worker error: {e}")

if __name__ == "__main__":
    with Pool(4) as pool:
        result = pool.map_async(bad_worker, range(10), error_callback=handle_error)
        result.wait(timeout=30)  # Don't hang forever
        if result.ready():
            print(result.get())

A worker is blocked on I/O or waiting for a lock:

# Broken — worker blocks on shared queue without timeout
def worker(queue):
    item = queue.get()  # Blocks if queue is empty
    return item

# Fixed — use timeout
def worker(queue):
    try:
        item = queue.get(timeout=5)  # Give up after 5 seconds
        return item
    except Exception:
        return None

Fix 5: Fix Shared State Between Processes

Unlike threads, processes do not share memory. Modifying a global variable in a worker has no effect on the parent:

Broken — expecting shared state:

from multiprocessing import Pool

results = []

def worker(x):
    results.append(x * x)  # Modifies the worker's own copy — parent never sees it

if __name__ == "__main__":
    with Pool(4) as pool:
        pool.map(worker, range(10))
    print(results)  # Always [] — workers modified their own copy

Fixed — return values instead:

from multiprocessing import Pool

def worker(x):
    return x * x  # Return the result

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(worker, range(10))
    print(results)  # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

For genuinely shared state, use multiprocessing.Manager or Value/Array:

from multiprocessing import Pool, Manager

def worker(shared_list, x):
    shared_list.append(x * x)

if __name__ == "__main__":
    with Manager() as manager:
        shared = manager.list()
        with Pool(4) as pool:
            pool.starmap(worker, [(shared, x) for x in range(10)])
        print(list(shared))

Common Mistake: Using a global list or dict to collect results from workers. Always return results from worker functions and let pool.map() collect them.

Fix 6: Fix multiprocessing with Classes

Methods of class instances are picklable in Python 3, but only if the class is defined at module level:

Broken — class defined locally:

def main():
    class Processor:
        def process(self, x):
            return x * 2

    p = Processor()
    with Pool(4) as pool:
        results = pool.map(p.process, range(10))  # PicklingError

Fixed — define class at module level:

class Processor:
    def process(self, x):
        return x * 2

if __name__ == "__main__":
    p = Processor()
    with Pool(4) as pool:
        results = pool.map(p.process, range(10))
    print(results)

Fix 7: Choose the Right Start Method

Python 3 supports three start methods:

import multiprocessing

# Check default
print(multiprocessing.get_start_method())  # 'fork' on Linux ≤3.13, 'spawn' on Windows/macOS/Linux 3.14+

# Set explicitly (must be done before any Pool/Process creation)
if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")  # Safe, slow
    # or "fork"   — fast but unsafe with threads (default on old Linux)
    # or "forkserver" — compromise

Method	Platform	Speed	Safety
`fork`	Linux only	Fast	Unsafe with threads/CUDA
`spawn`	All	Slow	Safe
`forkserver`	Unix	Medium	Safe

For PyTorch, CUDA, or multithreaded libraries, always use spawn or forkserver — fork after threading causes deadlocks:

if __name__ == "__main__":
    multiprocessing.set_start_method("spawn")
    with Pool(4) as pool:
        results = pool.map(train_model, configs)

Still Not Working?

Check Python version differences. macOS changed the default start method from fork to spawn in Python 3.8. Linux changed it from fork to spawn in Python 3.14. Code that worked on older Python or older Linux may break on macOS 3.8+ or Linux 3.14+. Always use the if __name__ == "__main__" guard.

Check for recursive imports. If your worker function imports a module that imports __main__, it can trigger the spawn loop. Restructure imports so worker functions are in separate modules.

Use concurrent.futures.ProcessPoolExecutor as a higher-level alternative — it handles many of these edge cases more gracefully:

from concurrent.futures import ProcessPoolExecutor

def worker(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as executor:
        results = list(executor.map(worker, range(10)))
    print(results)

Check for freeze_support() in frozen executables. If you package your script with PyInstaller, cx_Freeze, or py2exe, the first line inside if __name__ == "__main__": must be multiprocessing.freeze_support(). Without it, the frozen exe re-runs from the top in each child and crashes:

if __name__ == "__main__":
    multiprocessing.freeze_support()  # Required for frozen exes only
    main()

Check that workers can find the function in __main__. This is the Jupyter trap. If you defined worker() in a cell and the child re-imports __main__, it does not see your cell-defined function. Move the function to a separate .py file and import it, or switch to joblib with the loky backend (it uses cloudpickle and handles this).

Check for stale shared semaphores. A crashed parent can leak posix_ipc semaphores in /dev/shm. List them with ls /dev/shm/ and remove sem.mp-* entries that belong to your user. Repeated leaks suggest a worker is being killed by the OOM killer or by Kubernetes — see Kubernetes OOMKilled.

Check CPU count under cgroups. If os.cpu_count() returns 16 but your container only has 2 vCPU, you are creating 16 workers competing for 2 cores. Use len(os.sched_getaffinity(0)) on Linux for the cgroup-aware count. On older Python (<3.13), this is also a known cause of severe over-subscription in CI runners.

Use multiprocessing.shared_memory for large data (Python 3.8+). If you are passing large NumPy arrays between processes, pickling them is wasteful. shared_memory.SharedMemory lets the parent and child see the same memory region without copying.

For async multiprocessing with asyncio, use loop.run_in_executor() as shown in Fix: Python asyncio not running. For the underlying GIL behavior that motivates multiprocessing in the first place, see Python threading not running in parallel (GIL).