Current Location: Home> Latest Articles> How to combine curl_multi_exec() and curl_share_init() to achieve concurrent request optimization

How to combine curl_multi_exec() and curl_share_init() to achieve concurrent request optimization

M66 2025-05-18

In modern web development, handling a large number of HTTP requests is a common requirement, such as batch crawling interface data, parallel downloading resources, checking link status, etc. The traditional serial request method (i.e., one request ends and then initiates the next one) is inefficient. To this end, PHP provides curl_multi_* series functions to implement concurrent requests, while curl_share_* series can share resources (such as DNS cache) between multiple cURL handles to further optimize performance. This article will introduce how to combine curl_multi_exec() and curl_share_init() to achieve efficient concurrent request optimization.

1?? Why use curl_multi_*?

The curl_multi_* series of functions allow us to send multiple HTTP requests at the same time instead of waiting in line one by one. Its core workflow is:

  1. Initialize a multi handle ( curl_multi_init ).

  2. Add multiple easy handles ( curl_init ) to the multi handle.

  3. Use curl_multi_exec() to drive concurrent requests.

  4. Use curl_multi_select() to wait for the active handle (avoid CPU idle).

  5. Collect results and close resources.

This method is especially suitable for scenarios where a large number of requests are required to process at the same time, such as crawlers, interface aggregation, CDN preheating, etc.

2?? What is curl_share_*? Why use it in combination?

Although curl_multi_* can send concurrent requests, each easy handle is independent by default, for example, each needs to resolve DNS separately, maintain its own cookies, etc. The curl_share_* series of functions allow multiple easy handles to share certain internal resources (such as DNS cache, SSL sessions, cookies, etc.), which can avoid duplicate work, improve performance and reduce memory overhead.

Core steps include:

  1. Use curl_share_init() to create a shared object.

  2. Use curl_share_setopt() to set the shared content (such as CURLSHOPT_SHARE).

  3. Use curl_setopt($ch, CURLOPT_SHARE, $share) to attach a shared object to an easy handle.

  4. Release the shared object after the request is completed.

3?? Full example: Concurrent request + Shared DNS cache

Here is an example of PHP code that is actually runnable:

 <?php

// Concurrent requests are requiredURLList
$urls = [
    'https://m66.net/api/endpoint1',
    'https://m66.net/api/endpoint2',
    'https://m66.net/api/endpoint3',
    'https://m66.net/api/endpoint4'
];

// initialization multi handle
$multiHandle = curl_multi_init();

// Create a shared object
$shareHandle = curl_share_init();
curl_share_setopt($shareHandle, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);

// Store each easy handle
$curlHandles = [];

foreach ($urls as $url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SHARE, $shareHandle);
    curl_multi_add_handle($multiHandle, $ch);
    $curlHandles[$url] = $ch;
}

// Perform concurrent requests
$active = null;
do {
    $mrc = curl_multi_exec($multiHandle, $active);
    if ($mrc == CURLM_CALL_MULTI_PERFORM) {
        continue;
    }
    curl_multi_select($multiHandle);
} while ($active && $mrc == CURLM_OK);

// Collect results
$responses = [];
foreach ($curlHandles as $url => $ch) {
    $responses[$url] = curl_multi_getcontent($ch);
    curl_multi_remove_handle($multiHandle, $ch);
    curl_close($ch);
}

// closure multi handle and share handle
curl_multi_close($multiHandle);
curl_share_close($shareHandle);

// Print results
foreach ($responses as $url => $content) {
    echo "Response from $url:\n";
    echo $content . "\n\n";
}

4?? Summary of key points of optimization

? Use curl_multi_ to improve concurrency performance *: initiate and process multiple requests simultaneously, significantly reducing the total time-consuming.
? Use curl_share_ to save duplicate overhead *: By sharing resources such as DNS cache, avoid duplicate parsing and improve overall efficiency.
? Note resource release : After the request is completed, remember to release each easy handle, multi handle and share handle, otherwise it may cause memory leakage.
? Combined with curl_multi_select() : Wait for network activity to avoid CPU occupancy of 100%.

5?? Applicable scenarios

  • High concurrent interface aggregation requests.

  • Bulk crawling of large numbers of pages or files.

  • Website monitoring and health checks.

  • CDN or cache warm-up.

Using curl_multi_exec() and curl_share_init() can not only achieve true concurrency, but also reduce duplicate work through resource sharing, making PHP more efficient and stable when handling highly concurrent network requests.

If you encounter optimization bottlenecks in actual applications, you can further combine connection pooling, task scheduling, exception handling and other means to improve the overall solution.

Related