apify/proxy-chain

Programmable HTTP proxy server for Node.js

A programmable proxy server (think Squid) with support for SSL/TLS, authentication, upstream proxy chaining, SOCKS4/5 protocol, custom HTTP responses, and traffic statistics. The authentication and proxy chaining configuration is defined in code and can be fully dynamic, giving you a high level of customization for your use case.

For example, the proxy-chain package is useful if you need to use headless Chrome web browser and proxies with authentication, because Chrome doesn't support proxy URLs with password, such as http://username:[email protected]:8080. With this package, you can set up a local proxy server without any password that will forward requests to the upstream proxy with password. For details, read How to make headless Chrome and Puppeteer use a proxy server with authentication.

The proxy-chain package is developed by Apify, the full-stack web scraping and data extraction platform, to support their Apify Proxy product, which provides an easy access to a large pool of datacenter and residential IP addresses all around the world. The proxy-chain package is also used by Crawlee, the world's most popular web craling library for Node.js.

The proxy-chain package currently supports HTTP/SOCKS forwarding and HTTP CONNECT tunneling to forward arbitrary protocols such as HTTPS or FTP (learn more). The HTTP CONNECT tunneling also supports the SOCKS protocol. Also, proxy-chain only supports the Basic Proxy-Authorization.

Run a simple HTTP/HTTPS proxy server

import { Server } from 'proxy-chain';

const server = new Server({ port: 8000 });

await server.listen();
console.log(`Proxy server is listening on port ${server.port}`);

Run a HTTP/HTTPS proxy server with credentials and upstream proxy

import { Server } from 'proxy-chain';

const server = new Server({
    // Port where the server will listen. By default 8000.
    port: 8000,

    // Optional host where the proxy server will listen.
    // If not specified, the sever listens on an unspecified IP address (0.0.0.0 in IPv4, :: in IPv6)
    // You can use this option to limit the access to the proxy server.
    host: 'localhost',

    // Enables verbose logging
    verbose: true,

    // Custom user-defined function to authenticate incoming proxy requests,
    // and optionally provide the URL to chained upstream proxy.
    // The function must return an object (or promise resolving to the object) with the following signature:
    // { requestAuthentication: boolean, upstreamProxyUrl: string, failMsg?: string, customTag?: unknown }
    // If the function is not defined or is null, the server runs in simple mode.
    // Note that the function takes a single argument with the following properties:
    // * request      - An instance of http.IncomingMessage class with information about the client request
    //                  (which is either HTTP CONNECT for SSL protocol, or other HTTP request)
    // * username     - Username parsed from the Proxy-Authorization header. Might be empty string.
    // * password     - Password parsed from the Proxy-Authorization header. Might be empty string.
    // * hostname     - Hostname of the target server
    // * port         - Port of the target server
    // * isHttp       - If true, this is a HTTP request, otherwise it's a HTTP CONNECT tunnel for SSL
    //                  or other protocols
    // * connectionId - Unique ID of the HTTP connection. It can be used to obtain traffic statistics.
    prepareRequestFunction: ({ request, username, password, hostname, port, isHttp, connectionId }) => {
        return {
            // If set to true, the client is sent HTTP 407 resposne with the Proxy-Authenticate header set,
            // requiring Basic authentication. Here you can verify user credentials.
            requestAuthentication: username !== 'bob' || password !== 'TopSecret',

            // Sets up an upstream HTTP/HTTPS/SOCKS proxy to which all the requests are forwarded.
            // If null, the proxy works in direct mode, i.e. the connection is forwarded directly
            // to the target server. This field is ignored if "requestAuthentication" is true.
            // The username and password must be URI-encoded.
            upstreamProxyUrl: `http://username:[email protected]:3128`,
            // Or use SOCKS4/5 proxy, e.g.
            // upstreamProxyUrl: `socks://username:[email protected]:1080`,

            // Applies to HTTPS upstream proxy. If set to true, requests made to the proxy will
            // ignore certificate errors. Useful when upstream proxy uses self-signed certificate. By default "false".
            ignoreUpstreamProxyCertificate: true

            // If "requestAuthentication" is true, you can use the following property
            // to define a custom error message to return to the client instead of the default "Proxy credentials required"
            failMsg: 'Bad username or password, please try again.',

            // Optional custom tag that will be passed back via
            // `tunnelConnectResponded` or `tunnelConnectFailed` events
            // Can be used to pass information between proxy-chain
            // and any external code or application using it
            customTag: { userId: '123' },
        };
    },
});

// Emitted when HTTP connection is closed
server.on('connectionClosed', ({ connectionId, stats }) => {
  console.log(`Connection ${connectionId} closed`);
  console.dir(stats);
});

// Emitted when HTTP request fails
server.on('requestFailed', ({ request, error }) => {
  console.log(`Request ${request.url} failed`);
  console.error(error);
});

await server.listen();
console.log(`Proxy server is listening on port ${server.port}`);

Run a simple HTTPS proxy server

This example demonstrates how to create an HTTPS proxy server with a self-signed certificate. The HTTPS proxy server works identically to the HTTP version but with TLS encryption.

import { Server } from 'proxy-chain';
import fs from 'node:fs';
import path from 'node:path';

(async () => {
    // TODO: update these lines to use your own key and cert
    const sslKey = fs.readFileSync(path.join(import.meta.dirname, 'ssl.key'));
    const sslCrt = fs.readFileSync(path.join(import.meta.dirname, 'ssl.crt'));

    const server = new Server({
        // Main difference between 'http' and 'https' is additional event listening:
        //
        // http
        // -> listen for 'connection' events to track raw TCP sockets
        //
        // https:
        // -> listen for 'securedConnection' events (instead of 'connection') to track only post-TLS-handshake sockets
        // -> additionally listen for 'tlsError' events to handle TLS handshake errors
        //
        // Default value is 'http'
        serverType: 'https',

        // Provide the TLS certificate and private key
        httpsOptions: {
            key: sslKey,
            cert: sslCrt,
        },

        // Port where the server will listen
        port: 8443,

        // Enable verbose logging to see what's happening
        verbose: true,

        // Optional: Add authentication and upstream proxy configuration
        prepareRequestFunction: ({ username, hostname, port }) => {
            console.log(`Request to ${hostname}:${port} from user: ${username || 'anonymous'}`);

            // Allow the request
            return {};
        },
    });

    // Handle failed HTTP/HTTPS requests
    server.on('requestFailed', ({ request, error }) => {
        console.log(`Request ${request.url} failed`);
        console.error(error);
    });

    // Handle TLS handshake errors
    server.on('tlsError', ({ error, socket }) => {
        console.error(`TLS error from ${socket.remoteAddress}: ${error.message}`);
    });

    // Emitted when HTTP/HTTPS connection is closed
    server.on('connectionClosed', ({ connectionId, stats }) => {
        console.log(`Connection ${connectionId} closed`);
        console.dir(stats);
    });

    // Start the server
    await server.listen();

    // Handle graceful shutdown
    process.on('SIGINT', async () => {
        console.log('\nShutting down server...');
        await server.close(true);
        console.log('Server closed.');
        process.exit(0);
    });

    // Keep the server running
    await new Promise(() => { });
})();

Run server:

node https_proxy_server.js

Send request via proxy:

curl --proxy-insecure -x https://localhost:8443 -k https://example.com

Use custom HTTP agents for connection pooling

You can provide custom HTTP/HTTPS agents to enable connection pooling and reuse with upstream proxies. This is particularly useful for maintaining sticky IP addresses or reducing connection overhead:

import http from 'node:http';
import https from 'node:https';
import { Server } from 'proxy-chain';

// Create agents with keepAlive to enable connection pooling
const httpAgent = new http.Agent({
    keepAlive: true,
    maxSockets: 10,
});

const httpsAgent = new https.Agent({
    keepAlive: true,
    maxSockets: 10,
});

const server = new Server({
    port: 8000,
    prepareRequestFunction: ({ request }) => {
        return {
            upstreamProxyUrl: 'http://proxy.example.com:8080',
            // Or for HTTPS upstream proxy: 'https://proxy.example.com:8080'

            // Agents enable connection pooling to upstream proxy
            httpAgent,    // Used for HTTP upstream proxies
            httpsAgent,   // Used for HTTPS upstream proxies
        };
    },
});

await server.listen();
console.log(`Proxy server is listening on port ${server.port}`);

Note: Custom agents are only supported for HTTP and HTTPS upstream proxies. SOCKS upstream proxies use direct socket connections and do not support custom agents.

SOCKS support

SOCKS protocol is supported for versions 4 and 5, specifically: ['socks', 'socks4', 'socks4a', 'socks5', 'socks5h'], where socks will default to version 5.

You can use an upstreamProxyUrl like socks://username:[email protected]:1080.

Error status codes

The 502 Bad Gateway HTTP status code is not comprehensive enough. Therefore, the server may respond with 590-599 instead:

590 Non Successful

Upstream responded with non-200 status code.

591 RESERVED

This status code is reserved for further use.

592 Status Code Out Of Range

Upstream respondend with status code different than 100-999.

593 Not Found

DNS lookup failed - EAI_NODATA or EAI_NONAME.

594 Connection Refused

Upstream refused connection.

595 Connection Reset

Connection reset due to loss of connection or timeout.

596 Broken Pipe

Trying to write on a closed socket.

597 Auth Failed

Incorrect upstream credentials.

598 RESERVED

This status code is reserved for further use.

599 Upstream Error

Generic upstream error.

---

590 and 592 indicate an issue on the upstream side.
593 indicates an incorrect proxy-chain configuration.
594, 595 and 596 may occur due to connection loss.
597 indicates incorrect upstream credentials.
599 is a generic error, where the above is not applicable.

Custom error responses

To return a custom HTTP response to indicate an error to the client, you can throw the RequestError from inside of the prepareRequestFunction function. The class constructor has the following parameters: RequestError(body, statusCode, headers). By default, the response will have Content-Type: text/plain; charset=utf-8.

import { Server, RequestError } from 'proxy-chain';

const server = new Server({
    prepareRequestFunction: ({ request, username, password, hostname, port, isHttp, connectionId }) => {
        if (username !== 'bob') {
           throw new RequestError('Only Bob can use this proxy!', 400);
        }
    },
});

Measuring traffic statistics

To get traffic statistics for a certain HTTP connection, you can use:

const stats = server.getConnectionStats(connectionId);
console.dir(stats);

The resulting object looks like:

{
    // Number of bytes sent to client
    srcTxBytes: Number,
    // Number of bytes received from client
    srcRxBytes: Number,
    // Number of bytes sent to target server (proxy or website)
    trgTxBytes: Number,
    // Number of bytes received from target server (proxy or website)
    trgRxBytes: Number,
}

If the underlying sockets were closed, the corresponding values will be null, rather than 0.

Custom responses

Custom responses allow you to override the response to a HTTP requests to the proxy, without contacting any target host. For example, this is useful if you want to provide a HTTP proxy-style interface to an external API or respond with some custom page to certain requests. Note that this feature is only available for HTTP connections. That's because HTTPS connections cannot be intercepted without access to the target host's private key.

To provide a custom response, the result of the prepareRequestFunction function must define the customResponseFunction property, which contains a function that generates the custom response. The function is passed no parameters and it must return an object (or a promise resolving to an object) with the following properties:

{
  // Optional HTTP status code of the response. By default it is 200.
  statusCode: 200,

  // Optional HTTP headers of the response
  headers: {
    'X-My-Header': 'bla bla',
  }

  // Optional string with the body of the HTTP response
  body: 'My custom response',

  // Optional encoding of the body. If not provided, defaults to 'UTF-8'
  encoding: 'UTF-8',
}

Here is a simple example:

import { Server } from 'proxy-chain';

const server = new Server({
    port: 8000,
    prepareRequestFunction: ({ request, username, password, hostname, port, isHttp }) => {
        return {
            customResponseFunction: () => {
                return {
                    statusCode: 200,
                    body: `My custom response to ${request.url}`,
                };
            },
        };
    },
});

await server.listen();
console.log(`Proxy server is listening on port ${server.port}`);

Routing CONNECT to another HTTP server

While customResponseFunction enables custom handling methods such as GET and POST, many HTTP clients rely on CONNECT tunnels. It's possible to route those requests differently using the customConnectServer option. It accepts an instance of Node.js HTTP server.

import http from 'node:http';
import { Server } from 'proxy-chain';

const exampleServer = http.createServer((request, response) => {
    response.end('Hello from a custom server!');
});

const server = new Server({
    port: 8000,
    prepareRequestFunction: ({ request, username, password, hostname, port, isHttp }) => {
        if (request.url.toLowerCase() === 'example.com:80') {
            return {
                customConnectServer: exampleServer,
            };
        }

        return {};
    },
});

await server.listen();
console.log(`Proxy server is listening on port ${server.port}`);

In the example above, all CONNECT tunnels to example.com are overridden. This is an unsecure server, so it accepts only http: requests.

In order to intercept https: requests, https.createServer should be used instead, along with a self signed certificate.

import https from 'node:https';
import fs from 'node:fs';

const key = fs.readFileSync('./test/ssl.key');
const cert = fs.readFileSync('./test/ssl.crt');

const exampleServer = https.createServer({
    key,
    cert,
}, (request, response) => {
    response.end('Hello from a custom server!');
});

Closing the server

To shut down the proxy server, call the close([destroyConnections]) function. For example:

await server.close(true);
console.log('Proxy server was closed.');

The closeConnections parameter indicates whether pending proxy connections should be forcibly closed. If it's false, the function will wait until all connections are closed, which can take a long time. The function returns a promise.

Accessing the CONNECT response headers for proxy tunneling

Some upstream proxy providers might include valuable debugging information in the CONNECT response headers when establishing the proxy tunnel, for they may not modify future data in the tunneled connection.

The proxy server would emit a tunnelConnectResponded event for exposing such information, where the parameter types of the event callback are described in Node.js's documentation. Example:

server.on('tunnelConnectResponded', ({ proxyChainId, response, socket, head, customTag }) => {
    console.log(`CONNECT response headers received: ${response.headers}`);
});

Alternatively a helper function may be used:

listenConnectAnonymizedProxy(anonymizedProxyUrl, ({ response, socket, head }) => {
    console.log(`CONNECT response headers received: ${response.headers}`);
});

You can also listen to CONNECT requests that receive response with status code different from 200. The proxy server would emit a tunnelConnectFailed event.

server.on('tunnelConnectFailed', ({ proxyChainId, response, socket, head, customTag }) => {
    console.log(`CONNECT response failed with status code: ${response.statusCode}`);
});

Helper functions

The package also provides several utility functions.

anonymizeProxy({ url, port })

Parses and validates a HTTP/HTTPS proxy URL. If the proxy requires authentication, then the function starts an open local proxy server that forwards to the proxy. The port (on which the local proxy server will start) can be set via the port property of the first argument, if not provided, it will be chosen randomly.

For HTTPS proxy with self-signed certificate, set ignoreProxyCertificate property of the first argument to true to ignore certificate errors in proxy requests.

The function returns a promise that resolves to a String with the anonymous proxy URL, or the original URL if it was already anonymous.

The following example shows how you can use a proxy with authentication from headless Chrome and Puppeteer. For details, read this blog post.

import puppeteer from 'puppeteer';
import { anonymizeProxy, closeAnonymizedProxy } from 'proxy-chain';

(async() => {
    const oldProxyUrl = 'http://bob:[email protected]:8000';
    const newProxyUrl = await anonymizeProxy(oldProxyUrl);

    // Prints something like "http://127.0.0.1:45678"
    console.log(newProxyUrl);

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${newProxyUrl}`],
    });

    // Do your magic here...
    const page = await browser.newPage();
    await page.goto('https://www.example.com');
    await page.screenshot({ path: 'example.png' });
    await browser.close();

    // Clean up
    await closeAnonymizedProxy(newProxyUrl, true);
})();

closeAnonymizedProxy(anonymizedProxyUrl, closeConnections)

Closes anonymous proxy previously started by anonymizeProxy(). If proxy was not found or was already closed, the function has no effect and its result is false. Otherwise the result is true.

The closeConnections parameter indicates whether pending proxy connections are forcibly closed. If it's false, the function will wait until all connections are closed, which can take a long time.

The function returns a promise that resolves to a Boolean.

createTunnel(proxyUrl, targetHost, options)

Creates a TCP tunnel to targetHost that goes through a HTTP/HTTPS proxy server specified by the proxyUrl parameter.

The optional options parameter is an object with the following properties:

• port: Number - Enables specifying the local port to listen at. By default 0, which means a random port will be selected.
• hostname: String - Local hostname to listen at. By default localhost.
• ignoreProxyCertificate - For HTTPS proxy, ignore certificate errors in proxy requests. Useful for proxy with self-signed certificate. By default false.
• verbose: Boolean - If true, the functions logs a lot. By default false.

The result of the function is a local endpoint in a form of hostname:port. All TCP connections made to the local endpoint will be tunneled through the proxy to the target host and port. For example, this is useful if you want to access a certain service from a specific IP address.

The tunnel should be eventually closed by calling the closeTunnel() function.

The createTunnel() function returns a promise that resolves to a String with the path to the local endpoint.

For more information, read this blog post.

Example:

const host = await createTunnel('http://bob:[email protected]:8000', 'service.example.com:356');
// Prints something like "localhost:56836"
console.log(host);

closeTunnel(tunnelString, closeConnections)

Closes tunnel previously started by createTunnel(). The result value is false if the tunnel was not found or was already closed, otherwise it is true.

The closeConnections parameter indicates whether pending connections are forcibly closed. If it's false, the function will wait until all connections are closed, which can take a long time.

The function returns a promise that resolves to a Boolean.

listenConnectAnonymizedProxy(anonymizedProxyUrl, tunnelConnectRespondedCallback)

Allows to configure a callback on the anonymized proxy URL for the CONNECT response headers. See the above section Accessing the CONNECT response headers for proxy tunneling for details.

redactUrl(url, passwordReplacement)

Takes a URL and hides the password from it. For example:

// Prints 'http://bob:<redacted>@example.com'
console.log(redactUrl('http://bob:[email protected]'));