How to use curl_multi_* series functions to achieve high-performance concurrent requests

In PHP, we often need to make network requests with external APIs, websites, or services. Although the regular curl_exec() single request is simple, in high concurrency scenarios, its performance bottleneck is very obvious: each request is executed sequentially, waiting for the previous request to complete before the next one, resulting in a linear increase in overall time.

To improve network request efficiency, PHP provides two very powerful tools: curl_share_init() and curl_multi_* series functions. This article will provide an in-depth look at how to use them to achieve high-performance concurrent requests.

1?? The role of curl_share_init()

curl_share_init() is used to share data between multiple cURL handles, such as DNS cache, cookies, SSL sessions, etc.

For example: If you need to request multiple addresses of the same domain name at the same time, enabling shared DNS cache can significantly reduce DNS query time.

 $sh = curl_share_init();
curl_share_setopt($sh, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);

This code initializes a shared handle and is set to share DNS data. Next we will bind this shared handle to each curl request.

2?? curl_multi_* function of series functions

The curl_multi_* series of functions allow us to initiate multiple HTTP requests at the same time, rather than sending them one by one like curl_exec() .

The main functions include:

• curl_multi_init() : Initializes a multi handle.

• curl_multi_add_handle() : Adds a single curl handle to a multi handle.

• curl_multi_exec() : Runs all requests in the multi handle.

• curl_multi_select() : Blocks the connection waiting for active.

• curl_multi_getcontent() : Gets the returned content of each request.

• curl_multi_remove_handle() : Removes a single handle when finished.

• curl_multi_close() : Close the multi handle.

3?? Practical example: Concurrent requests for multiple URLs

Here is a complete example showing how to use curl_share_init() and curl_multi_* series functions to request multiple addresses concurrently.

 <?php

// To request URL List
$urls = [
    'https://m66.net/api/endpoint1',
    'https://m66.net/api/endpoint2',
    'https://m66.net/api/endpoint3'
];

// Initialize the shared handle
$sh = curl_share_init();
curl_share_setopt($sh, CURLSHOPT_SHARE, CURL_LOCK_DATA_DNS);

// initialization multi Handle
$mh = curl_multi_init();
$curl_handles = [];

// For each URL Create a separate curl Handle
foreach ($urls as $i => $url) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SHARE, $sh); // 绑定共享Handle
    curl_multi_add_handle($mh, $ch);
    $curl_handles[$i] = $ch;
}

// Perform all requests
$running = null;
do {
    $status = curl_multi_exec($mh, $running);
    if ($status > 0) {
        echo "cURL error: " . curl_multi_strerror($status) . "\n";
    }
    // Waiting for an active connection
    curl_multi_select($mh);
} while ($running > 0);

// Get results
$responses = [];
foreach ($curl_handles as $i => $ch) {
    $responses[$i] = curl_multi_getcontent($ch);
    curl_multi_remove_handle($mh, $ch);
    curl_close($ch);
}

// closure multi and share Handle
curl_multi_close($mh);
curl_share_close($sh);

// Output result
foreach ($responses as $i => $response) {
    echo "Response from URL $i:\n$response\n\n";
}
?>

4?? Performance improvement analysis

With curl_multi_* , all requests will be executed concurrently as much as possible, and the overall time is close to the longest single request, rather than the sum.

Using curl_share_init() can reduce the overhead of duplicate DNS queries, SSL handshakes, etc., and further optimize performance.

In scenarios where high concurrency and frequent requests for the same domain name API (such as crawling multiple interfaces at the same time, crawling web pages in batches, synchronizing multiple microservice data, etc.), this combination can significantly improve efficiency.

5?? Notes

• DNS sharing does not always speed up significantly : it mainly takes effect under the same domain name and multiple requests.

• Reasonably set timeout : a stuck request in concurrent may affect the overall situation.

• Controlling the number of concurrency : Too many requests may cause the server or local resources to be exhausted and can be distributed in semaphores or batches.

• Error handling : In the example, the single request failed to be processed in detail, and the actual application needs to be supplemented.