DpExchange.Core.HttpClient (DpExchangeCore v0.3.120)

Copy Markdown View Source

The shared HTTP request pipeline: rate limiting, retry, logging, error shaping and the generic authentication schemes.

What it deliberately does not know

No venue appears in this module. It once carried a Coinbase JWT builder and a case provider do table mapping "coinbase" and "gemini" to their own rate-limit header parsers. Both moved into their venue packages, because a venue fact living in shared code is a second place that can be wrong about that venue — and it was: every provider except Coinbase once fell through to Coinbase's socket codec, so one venue's socket spoke another's protocol at its own endpoint and delivered nothing for as long as that stood.

What is left is generic: :hmac_sha256, :basic and :bearer auth, the request pipeline, query-string building, response parsing, retry, and a rate-limit header parser that reads only the conventional x-ratelimit-* shape. A venue whose scheme or headers differ supplies its own — see build_auth_headers/5 and parse_rate_limit_headers/1.

A response body is bounded as it is received

:max_body_bytes (default 16 MiB) caps what is held for one response. Req is given into: fun, and the function counts bytes as they arrive and halts the transfer past the limit, so a venue, proxy or CDN sending an unbounded body cannot exhaust the calling node's memory. Streaming into a function skips only Req's decompression, and this pipeline never enables that (compressed is off, so no accept-encoding is sent and a body is never inflated), so below the limit the body, the JSON decoding, decode_body: false and non-JSON handling are what they were. A transport that ignores :into is checked after the fact instead; that path cannot prevent the allocation.

An error text never quotes a credential the request carried

The error tuple, the retry warning and the [:dp_exchange, :request, :exception] metadata are built from what the transport and the venue answered, and either can quote the request back: a 401 body that names the key it was sent, an exception message built from a header value. The values of credential-named headers (and the token after a scheme, so Bearer abc also yields abc), a URL's userinfo, and the value of a credential-named query parameter are therefore replaced with [redacted] in those texts. Values shorter than six bytes are not hunted for. The log_requests line names the endpoint (Telemetry.endpoint/1), never the URL, a header or a body.

The limiter is resolved at call time

Never at compile time. Application.compile_env/3 would freeze whichever limiter the consumer configured when this dependency was compiled, so a consumer changing it later would either have to recompile or get a boot-time mismatch. Resolution goes through DpExchange.Core.Config, so a consumer's async: true test can swap the limiter for its own process tree without configuring it for every test beside it.

Neither acquire nor check fills the bucket

They answer "is there capacity" and, for acquire, wait until there is. Something has to report what actually left, or the bucket stays empty and every check passes. This module records on behalf of every request it makes; a venue package issuing its own HTTP calls must record for itself. One that did not acquired before every request and recorded none, so its ceiling bound nothing: 395 calls per 60s against a documented 300, while the budget panel read 83/240.

This module itself repeated the same mechanism in a smaller way, found 2026-09-05. record/3 was called only from the {:ok, response} branch of the request pipeline — a retried 5xx and a venue 429 both genuinely reached the wire and genuinely consumed the venue's quota, and neither was recorded. The 312 calls missing from the 83/240 incident above were not a mystery once this was found: they were exactly the retried and rate-limited requests, the very shapes this branch skipped. Every outcome of make_http_request/5 — success, retry, 429, or a permanent 4xx — is now counted once, by the reservation each attempt makes before it is sent (an acquire/3, with a zero timeout on the non-blocking path). See check_rate_limit/2.

Summary

Types

How to authenticate: one of the generic schemes, or a venue's own builder.

A response as it comes back from the pipeline.

The :provider option, echoed back as the tag of {:exchange_error, provider, reason}. Every venue passes its atom (:gemini, :gemini_private); a string is accepted too.

Why a request did not produce a response.

Headers, or a function producing them that is called again for every attempt. Use the function form for a signature that expires or carries a one-time nonce, so a retry is signed afresh rather than replayed. See request/5.

Functions

Build authentication headers for API requests.

Convenience function for GET requests with query parameters.

Parses rate-limit information from response headers, for the conventional x-ratelimit-limit / x-ratelimit-remaining / x-ratelimit-reset shape only.

Make an HTTP request with provider-specific and account-aware rate limiting.

Types

account_id()

@type account_id() :: String.t()

auth_scheme()

@type auth_scheme() ::
  :hmac_sha256
  | :basic
  | :bearer
  | (http_method(), String.t(), body(), map() -> headers())

How to authenticate: one of the generic schemes, or a venue's own builder.

The function form is the hook that replaced a per-venue branch. The spec named only atom() for a while after the hook was added, so a venue passing a builder — the whole point of the hook — was told by dialyzer that the call "breaks the contract". Adding a capability without widening its spec makes the tool argue against the feature.

body()

@type body() :: String.t() | nil

headers()

@type headers() :: [{String.t(), String.t()}]

http_method()

@type http_method() :: :get | :post | :put | :delete

http_response()

@type http_response() :: %{
  status: integer(),
  headers: headers() | map(),
  body: term()
}

A response as it comes back from the pipeline.

body is whatever the transport decoded, not a string. Req decodes JSON into a map or a list before this module sees it, and only leaves a binary when it could not.

It was declared String.t(), and that was not a harmless inaccuracy: dialyzer then concluded that any consumer matching a decoded body was matching something impossible, so every function reachable only through that branch was reported as unreachable dead code. A venue package's mix dialyzer failed with Function decimal/1 will never be called about a function called on every price it parses.

The application this pipeline came from hit the same class of problem from the other end and said so in a comment: a missing clause "poisoned dialyzer's success typing for every HttpClient.request/5 caller — the whole chain got narrowed to {:error, _} only." A wrong type is worse than a missing one; it makes the tool confidently wrong, and the reader believes it.

options()

@type options() :: keyword()

provider()

@type provider() :: atom() | String.t()

The :provider option, echoed back as the tag of {:exchange_error, provider, reason}. Every venue passes its atom (:gemini, :gemini_private); a string is accepted too.

rate_limited_request_options()

@type rate_limited_request_options() :: [
  {:provider, provider()}
  | {:rate_limit_key, provider()}
  | {:account_id, account_id()}
  | {:user_id, user_id()}
  | {:operation, String.t()}
  | {atom(), term()}
]

request_error()

@type request_error() ::
  String.t()
  | {:response_too_large, pos_integer()}
  | {:exchange_error, provider(), term()}

Why a request did not produce a response.

Three shapes, and the spec used to name only the first — which made dialyzer tell every consumer that its handling of the others was unreachable dead code.

  • String.t() — a plain message, when no :provider was given.
  • {:response_too_large, limit} — the body passed :max_body_bytes, when no :provider was given.
  • {:exchange_error, provider, reason} — the same message (or {:response_too_large, limit}) tagged with the venue, which is what a caller gets whenever it passes :provider, i.e. almost always.

Rate limiting arrives as a two-element error, not a three-element one

This spec used to advertise {:error, :rate_limited, retry_after: seconds} as a third return shape. request/5 never returns it. Both rate-limit paths convert to a two-element error before returning, deliberately and for recorded reasons:

  • a venue 429 becomes "Rate limited by the venue — retry after Ns", because a three-element tuple reaching a case written for two-element ones crashed 152 collector tasks in one night;
  • our own limiter refusing becomes "Throttled by our own rate limiter (not the venue)", because the two used to share wording and a self-inflicted refusal was read as a flaky venue for weeks.

Both keep the retry interval in the message. The spec is corrected here rather than the behaviour: a spec that names a shape the function cannot return sends dialyzer after every caller that handles it, reporting correct code as unreachable — which is exactly what it did to the Gemini package's rate-limit clause. Found 2026-08-28.

signed_headers()

@type signed_headers() ::
  headers() | (-> headers() | {:ok, headers()} | {:error, term()})

Headers, or a function producing them that is called again for every attempt. Use the function form for a signature that expires or carries a one-time nonce, so a retry is signed afresh rather than replayed. See request/5.

user_id()

@type user_id() :: String.t()

Functions

build_auth_headers(method, path, body, credentials, auth_type)

@spec build_auth_headers(http_method(), String.t(), body(), map(), auth_scheme()) ::
  headers()

Build authentication headers for API requests.

Parameters

  • method: HTTP method
  • path: API endpoint path
  • body: Request body
  • credentials: API credentials
  • auth_type: Authentication type: :hmac_sha256, :basic, :bearer, or a 4-arity function (method, path, body, credentials -> headers) for a venue's own scheme

Returns

  • List of authentication headers. An auth_type that is none of the above raises ArgumentError. It used to yield [], so a typo'd scheme (:hmac_sha_256) sent the request unauthenticated — the opposite of failing closed.

get(url, params \\ [], opts \\ [])

@spec get(String.t(), keyword() | map(), rate_limited_request_options()) ::
  {:ok, any()} | {:error, request_error()}

Convenience function for GET requests with query parameters.

Parameters

  • url: Base URL for the request
  • params: Query parameters as keyword list or map. nil values are dropped. Keys and values are form-encoded. A list, tuple or plain-map value raises ArgumentError: it has no single spelling, and the venue decides between repeated keys and a comma join.
  • opts: Additional options

Returns

  • {:ok, parsed_json} on success. With decode_body: false the bytes are decoded here with Jason's floats: :decimals, so a JSON number is a Decimal and keeps its digits.
  • {:error, reason} on failure

parse_rate_limit_headers(headers)

@spec parse_rate_limit_headers(headers() | map()) :: map() | nil

Parses rate-limit information from response headers, for the conventional x-ratelimit-limit / x-ratelimit-remaining / x-ratelimit-reset shape only.

Returns nil when the headers do not carry it. nil means "this response did not say", never "there is no limit" — a caller must not read an absent header as headroom.

This used to take a provider name and dispatch on it, with branches for two venues' bespoke headers. Those branches are venue knowledge and moved into the venue packages (D-C); a venue whose headers differ parses them itself, and does not need this function to have heard of it.

request(method, url, headers \\ [], body \\ nil, opts \\ [])

@spec request(
  http_method(),
  String.t(),
  signed_headers(),
  body(),
  rate_limited_request_options()
) ::
  {:ok, http_response()} | {:error, request_error()}

Make an HTTP request with provider-specific and account-aware rate limiting.

Parameters

  • method: HTTP method (:get, :post, :put, :delete)
  • url: Full URL for the request
  • headers: List of HTTP headers, or a zero-arity function returning them (or {:ok, list} / {:error, reason}) that is called again before every attempt, so a signed request is re-signed on retry rather than replayed
  • body: Request body (for POST/PUT requests). A :get with a non-empty body raises ArgumentError before anything is sent, because Req 0.7.0-0.7.2 send it as a POST
  • opts: Additional options including rate limiting context

Options

  • :timeout - Request timeout in milliseconds, or :infinity (default: 30_000). It bounds the whole response, not each chunk, and it applies to each attempt when :retry_attempts retries.
  • :retry_attempts - Number of attempts in all; 0 sends once, like 1 (default: 3 for :get, 1 for :post, :put and :delete). A transport failure or 5xx after a write may mean the write took effect, and a retry repeats it. Whether that is safe depends on whether the venue deduplicates the request, which only the venue package knows. So a write is never retried unless the venue says so: one with an idempotency key, such as a client_order_id, passes an explicit retry_attempts. A write without one needs nothing.
  • :retry_delay - Base delay between retries in milliseconds (default: 1000). The wait before retry n is retry_delay * n plus up to a quarter of that again at random, so it is never shorter than the configured delay. A delay whose longest wait over the attempts (jitter included) passes 4_294_967_295 ms, the most a process can sleep, is an ArgumentError before anything is sent. What is retried: a transport failure, a 5xx and an unexpected status. What is not: a 4xx (permanent), a 3xx (redirect is off, so the same request gets the same answer), a 429 (the venue's Retry-After holds the bucket and the error returns at once), a body over :max_body_bytes, and a request that could not be signed.
  • :log_requests - Whether to log requests (default: true)
  • :provider - The venue's identity: echoed in {:exchange_error, provider, _} and in every [:dp_exchange, :request | :rate_limit, _] event's provider:. Also the limiter bucket unless :rate_limit_key names another (no :provider and no :rate_limit_key means no rate limiting). An atom or a non-blank string, or ArgumentError before anything is sent (:rate_limit_key too): both are printed into log lines and bucket names.
  • :rate_limit_key - The limiter bucket this request is metered in (default: :provider). Selects ONLY the bucket: a venue that meters order writes apart from its reads passes provider: :venue, rate_limit_key: :venue_orders, and errors and telemetry still say :venue. The limiter is handed the key and, as :telemetry_provider, the identity. The ceiling for the key is configured under the key (DefaultRateLimiter :limits).
  • :plug, :adapter, :req_adapter - The transport seam, passed to Req (:req_adapter is this package's name for :adapter and loses to an explicit one). For a transport that never leaves the process, such as a test stub.
  • :headers - get/3 only: request headers (default: []).
  • :rate_limit_per_endpoint - Also meter this request in a bucket of its own endpoint, keyed "<provider> <path>", beside the provider's (default: false). For a venue that states a per-endpoint limit as well as a global one. The endpoint bucket takes the limiter's :default limits unless the consumer names it.
  • :rate_limit_blocking - Wait for the limiter instead of failing when the bucket is empty (default: false). The wait is bounded by :timeout.
  • :weight - Tokens this request costs (default: 1)
  • :max_retry_after_s - Ceiling, in seconds, on a venue's Retry-After that is honoured (default: 600). A non-negative integer of at most 4_294_967 (the seconds whose milliseconds are the longest hold a limiter can serve), or ArgumentError before anything is sent.
  • :account_id, :user_id, :operation - Forwarded to the limiter, which may use them (optional, no defaults). DefaultRateLimiter ignores all three: the ceiling is per venue, never per key.
  • :max_body_bytes - The most of a response body that is received (default: 16 MiB, 16_777_216). Counted as the bytes arrive; the transfer is halted the moment the total passes the limit, so an oversized or endless body is never held in memory. The result is {:error, {:response_too_large, limit}} (wrapped in {:exchange_error, provider, _} when :provider is given), is never retried, and is reported on [:dp_exchange, :request, :stop] with result: :error, reason: :response_too_large. A body of exactly the limit is accepted. A positive integer, or ArgumentError. The count is of the bytes on the wire: this pipeline never asks for compression (Req's :compressed is off), so it is also the decoded size.
  • :raw_status - Return {:ok, response} for a 4xx instead of an error string (default: false). A venue package needs this to tell a refusal from an error: the contract makes {:refused, reason} permanent and {:error, reason} possibly transient, and the venue states which in the 4xx status and body. Without it that evidence is flattened into a message and the venue has to string-match its way back to it. 5xx is unaffected — a server error is not a venue's considered answer. 429 is unaffected too: it is the venue's rate limit, handled by this pipeline (Retry-After, the held bucket, telemetry), and is always the "Rate limited by the venue" error, whether or not :raw_status is set.
  • :decode_body - Passed to Req (default: Req's, which decodes JSON with floats). false returns the body as bytes, for a venue that types money as a JSON number and must decode it without float rounding (Jason.decode(body, floats: :decimals)).

Returns

  • {:ok, http_response()} on success, and on a 4xx when raw_status: true
  • {:error, reason} on failure