# `DpExchange.Core.HttpClient`
[🔗](https://github.com/DistortionPoint/dp-exchange-core/blob/v0.3.120/lib/dp_exchange/core/http_client.ex#L1)

The shared HTTP request pipeline: rate limiting, retry, logging, error shaping and
the generic authentication schemes.

## What it deliberately does not know

**No venue appears in this module.** It once carried a Coinbase JWT builder and a
`case provider do` table mapping `"coinbase"` and `"gemini"` to their own rate-limit
header parsers. Both moved into their venue packages, because a venue fact living in
shared code is a second place that can be wrong about that venue — and it was: every
provider except Coinbase once fell through to Coinbase's socket codec, so one venue's
socket spoke another's protocol at its own endpoint and delivered nothing for as long
as that stood.

What is left is generic: `:hmac_sha256`, `:basic` and `:bearer` auth, the request
pipeline, query-string building, response parsing, retry, and a rate-limit header
parser that reads only the conventional `x-ratelimit-*` shape. A venue whose scheme or
headers differ supplies its own — see `build_auth_headers/5` and
`parse_rate_limit_headers/1`.

## A response body is bounded as it is received

`:max_body_bytes` (default 16 MiB) caps what is held for one response. Req is given
`into: fun`, and the function counts bytes as they arrive and halts the transfer past the
limit, so a venue, proxy or CDN sending an unbounded body cannot exhaust the calling node's
memory. Streaming into a function skips only Req's decompression, and this pipeline never
enables that (`compressed` is off, so no `accept-encoding` is sent and a body is never
inflated), so below the limit the body, the JSON decoding, `decode_body: false` and
non-JSON handling are what they were. A transport that ignores `:into` is checked after
the fact instead; that path cannot prevent the allocation.

## An error text never quotes a credential the request carried

The error tuple, the retry warning and the `[:dp_exchange, :request, :exception]` metadata
are built from what the transport and the venue answered, and either can quote the request
back: a 401 body that names the key it was sent, an exception message built from a header
value. The values of credential-named headers (and the token after a scheme, so `Bearer abc`
also yields `abc`), a URL's userinfo, and the value of a credential-named query parameter
are therefore replaced with `[redacted]` in those texts. Values shorter than six bytes are
not hunted for. The `log_requests` line names the endpoint (`Telemetry.endpoint/1`), never
the URL, a header or a body.

## The limiter is resolved at call time

Never at compile time. `Application.compile_env/3` would freeze whichever limiter the
*consumer* configured when this dependency was compiled, so a consumer changing it
later would either have to recompile or get a boot-time mismatch. Resolution goes
through `DpExchange.Core.Config`, so a consumer's `async: true` test can swap the
limiter for its own process tree without configuring it for every test beside it.

## Neither acquire nor check fills the bucket

They answer "is there capacity" and, for acquire, wait until there is. Something has to
report what actually left, or the bucket stays empty and every check passes. This
module records on behalf of every request it makes; a venue package issuing its own
HTTP calls must record for itself. One that did not acquired before every request and
recorded none, so its ceiling bound nothing: 395 calls per 60s against a documented
300, while the budget panel read 83/240.

**This module itself repeated the same mechanism in a smaller way, found 2026-09-05.**
`record/3` was called only from the `{:ok, response}` branch of the request pipeline — a
retried 5xx and a venue 429 both genuinely reached the wire and genuinely consumed the
venue's quota, and neither was recorded. The 312 calls missing from the 83/240 incident
above were not a mystery once this was found: they were exactly the retried and
rate-limited requests, the very shapes this branch skipped. Every outcome of
`make_http_request/5` — success, retry, 429, or a permanent 4xx — is now counted once,
by the reservation each attempt makes before it is sent (an `acquire/3`, with a zero
timeout on the non-blocking path). See `check_rate_limit/2`.

# `account_id`

```elixir
@type account_id() :: String.t()
```

# `auth_scheme`

```elixir
@type auth_scheme() ::
  :hmac_sha256
  | :basic
  | :bearer
  | (http_method(), String.t(), body(), map() -> headers())
```

How to authenticate: one of the generic schemes, or a venue's own builder.

The function form is the hook that replaced a per-venue branch. The spec named only
`atom()` for a while after the hook was added, so a venue passing a builder — the whole
point of the hook — was told by dialyzer that the call "breaks the contract". Adding a
capability without widening its spec makes the tool argue against the feature.

# `body`

```elixir
@type body() :: String.t() | nil
```

# `headers`

```elixir
@type headers() :: [{String.t(), String.t()}]
```

# `http_method`

```elixir
@type http_method() :: :get | :post | :put | :delete
```

# `http_response`

```elixir
@type http_response() :: %{
  status: integer(),
  headers: headers() | map(),
  body: term()
}
```

A response as it comes back from the pipeline.

`body` is **whatever the transport decoded**, not a string. Req decodes JSON into a map
or a list before this module sees it, and only leaves a binary when it could not.

It was declared `String.t()`, and that was not a harmless inaccuracy: dialyzer then
concluded that any consumer matching a decoded body was matching something impossible,
so every function reachable only through that branch was reported as **unreachable
dead code**. A venue package's `mix dialyzer` failed with `Function decimal/1 will
never be called` about a function called on every price it parses.

The application this pipeline came from hit the same class of problem from the other
end and said so in a comment: a missing clause "poisoned dialyzer's success typing for
every `HttpClient.request/5` caller — the whole chain got narrowed to `{:error, _}`
only." A wrong type is worse than a missing one; it makes the tool confidently wrong,
and the reader believes it.

# `options`

```elixir
@type options() :: keyword()
```

# `provider`

```elixir
@type provider() :: atom() | String.t()
```

The `:provider` option, echoed back as the tag of `{:exchange_error, provider, reason}`.
Every venue passes its atom (`:gemini`, `:gemini_private`); a string is accepted too.

# `rate_limited_request_options`

```elixir
@type rate_limited_request_options() :: [
  {:provider, provider()}
  | {:rate_limit_key, provider()}
  | {:account_id, account_id()}
  | {:user_id, user_id()}
  | {:operation, String.t()}
  | {atom(), term()}
]
```

# `request_error`

```elixir
@type request_error() ::
  String.t()
  | {:response_too_large, pos_integer()}
  | {:exchange_error, provider(), term()}
```

Why a request did not produce a response.

Three shapes, and the spec used to name only the first — which made dialyzer tell every
consumer that its handling of the others was unreachable dead code.

  * `String.t()` — a plain message, when no `:provider` was given.
  * `{:response_too_large, limit}` — the body passed `:max_body_bytes`, when no `:provider`
    was given.
  * `{:exchange_error, provider, reason}` — the same message (or `{:response_too_large,
    limit}`) tagged with the venue, which is what a caller gets whenever it passes
    `:provider`, i.e. almost always.

## Rate limiting arrives as a two-element error, not a three-element one

This spec used to advertise `{:error, :rate_limited, retry_after: seconds}` as a third
return shape. **`request/5` never returns it.** Both rate-limit paths convert to a
two-element error before returning, deliberately and for recorded reasons:

  * a **venue 429** becomes `"Rate limited by the venue — retry after Ns"`, because a
    three-element tuple reaching a `case` written for two-element ones crashed 152
    collector tasks in one night;
  * **our own limiter** refusing becomes `"Throttled by our own rate limiter (not the
    venue)"`, because the two used to share wording and a self-inflicted refusal was
    read as a flaky venue for weeks.

Both keep the retry interval in the message. The spec is corrected here rather than the
behaviour: a spec that names a shape the function cannot return sends dialyzer after
every caller that handles it, reporting correct code as unreachable — which is exactly
what it did to the Gemini package's rate-limit clause. Found 2026-08-28.

# `signed_headers`

```elixir
@type signed_headers() ::
  headers() | (-> headers() | {:ok, headers()} | {:error, term()})
```

Headers, or a function producing them that is called again for every attempt. Use the
function form for a signature that expires or carries a one-time nonce, so a retry is
signed afresh rather than replayed. See `request/5`.

# `user_id`

```elixir
@type user_id() :: String.t()
```

# `build_auth_headers`

```elixir
@spec build_auth_headers(http_method(), String.t(), body(), map(), auth_scheme()) ::
  headers()
```

Build authentication headers for API requests.

## Parameters
- `method`: HTTP method
- `path`: API endpoint path
- `body`: Request body
- `credentials`: API credentials
- `auth_type`: Authentication type: `:hmac_sha256`, `:basic`, `:bearer`, or a 4-arity
  function `(method, path, body, credentials -> headers)` for a venue's own scheme

## Returns
- List of authentication headers. An `auth_type` that is none of the above raises
  `ArgumentError`. It used to yield `[]`, so a typo'd scheme (`:hmac_sha_256`) sent the
  request **unauthenticated** — the opposite of failing closed.

# `get`

```elixir
@spec get(String.t(), keyword() | map(), rate_limited_request_options()) ::
  {:ok, any()} | {:error, request_error()}
```

Convenience function for GET requests with query parameters.

## Parameters
- `url`: Base URL for the request
- `params`: Query parameters as keyword list or map. `nil` values are dropped. Keys and
  values are form-encoded. A list, tuple or plain-map value raises `ArgumentError`: it has
  no single spelling, and the venue decides between repeated keys and a comma join.
- `opts`: Additional options

## Returns
- `{:ok, parsed_json}` on success. With `decode_body: false` the bytes are decoded here
  with `Jason`'s `floats: :decimals`, so a JSON number is a `Decimal` and keeps its digits.
- `{:error, reason}` on failure

# `parse_rate_limit_headers`

```elixir
@spec parse_rate_limit_headers(headers() | map()) :: map() | nil
```

Parses rate-limit information from response headers, for the conventional
`x-ratelimit-limit` / `x-ratelimit-remaining` / `x-ratelimit-reset` shape only.

Returns `nil` when the headers do not carry it. **`nil` means "this response did not
say", never "there is no limit"** — a caller must not read an absent header as
headroom.

This used to take a provider name and dispatch on it, with branches for two venues'
bespoke headers. Those branches are venue knowledge and moved into the venue packages
(D-C); a venue whose headers differ parses them itself, and does not need this
function to have heard of it.

# `request`

```elixir
@spec request(
  http_method(),
  String.t(),
  signed_headers(),
  body(),
  rate_limited_request_options()
) ::
  {:ok, http_response()} | {:error, request_error()}
```

Make an HTTP request with provider-specific and account-aware rate limiting.

## Parameters
- `method`: HTTP method (:get, :post, :put, :delete)
- `url`: Full URL for the request
- `headers`: List of HTTP headers, or a zero-arity function returning them (or
  `{:ok, list}` / `{:error, reason}`) that is called again before every attempt, so a
  signed request is re-signed on retry rather than replayed
- `body`: Request body (for POST/PUT requests). A `:get` with a non-empty body raises
  `ArgumentError` before anything is sent, because Req 0.7.0-0.7.2 send it as a POST
- `opts`: Additional options including rate limiting context

## Options
- `:timeout` - Request timeout in milliseconds, or `:infinity` (default: 30_000). It bounds the whole
  response, not each chunk, and it applies to each attempt when `:retry_attempts` retries.
- `:retry_attempts` - Number of attempts in all; `0` sends once, like `1` (default: 3 for
  `:get`, **1 for `:post`,
  `:put` and `:delete`**). A transport failure or 5xx after a write may mean the write
  took effect, and a retry repeats it. Whether that is safe depends on whether the venue
  deduplicates the request, which only the venue package knows. So a write is never retried
  unless the venue says so: one with an idempotency key, such as a `client_order_id`, passes
  an explicit `retry_attempts`. A write without one needs nothing.
- `:retry_delay` - Base delay between retries in milliseconds (default: 1000). The wait
  before retry `n` is `retry_delay * n` plus up to a quarter of that again at random, so it is
  never shorter than the configured delay. A delay whose longest wait over the attempts
  (jitter included) passes 4_294_967_295 ms, the most a process can sleep, is an
  `ArgumentError` before anything is sent. What is retried: a transport failure, a 5xx and an
  unexpected status. What is not: a 4xx (permanent), a 3xx (`redirect` is off, so the same
  request gets the same answer), a 429 (the venue's `Retry-After` holds the bucket and the
  error returns at once), a body over `:max_body_bytes`, and a request that could not be
  signed.
- `:log_requests` - Whether to log requests (default: true)
- `:provider` - The venue's identity: echoed in `{:exchange_error, provider, _}` and in
  every `[:dp_exchange, :request | :rate_limit, _]` event's `provider:`. Also the limiter
  bucket unless `:rate_limit_key` names another (no `:provider` and no `:rate_limit_key`
  means no rate limiting). An atom or a non-blank string, or `ArgumentError` before anything
  is sent (`:rate_limit_key` too): both are printed into log lines and bucket names.
- `:rate_limit_key` - The limiter bucket this request is metered in (default: `:provider`).
  Selects ONLY the bucket: a venue that meters order writes apart from its reads passes
  `provider: :venue, rate_limit_key: :venue_orders`, and errors and telemetry still say
  `:venue`. The limiter is handed the key and, as `:telemetry_provider`, the identity.
  The ceiling for the key is configured under the key (`DefaultRateLimiter` `:limits`).
- `:plug`, `:adapter`, `:req_adapter` - The transport seam, passed to `Req` (`:req_adapter`
  is this package's name for `:adapter` and loses to an explicit one). For a transport that
  never leaves the process, such as a test stub.
- `:headers` - `get/3` only: request headers (default: `[]`).
- `:rate_limit_per_endpoint` - Also meter this request in a bucket of its own endpoint,
  keyed `"<provider> <path>"`, beside the provider's (default: `false`). For a venue that
  states a per-endpoint limit as well as a global one. The endpoint bucket takes the
  limiter's `:default` limits unless the consumer names it.
- `:rate_limit_blocking` - Wait for the limiter instead of failing when the bucket is
  empty (default: `false`). The wait is bounded by `:timeout`.
- `:weight` - Tokens this request costs (default: 1)
- `:max_retry_after_s` - Ceiling, in seconds, on a venue's `Retry-After` that is
  honoured (default: 600). A non-negative integer of at most 4_294_967 (the seconds whose
  milliseconds are the longest hold a limiter can serve), or `ArgumentError` before anything
  is sent.
- `:account_id`, `:user_id`, `:operation` - Forwarded to the limiter, which may use them
  (optional, no defaults). `DefaultRateLimiter` ignores all three: the ceiling is per
  venue, never per key.
- `:max_body_bytes` - The most of a response body that is received (default: 16 MiB,
  `16_777_216`). Counted as the bytes arrive; the transfer is halted the moment the total
  passes the limit, so an oversized or endless body is never held in memory. The result is
  `{:error, {:response_too_large, limit}}` (wrapped in `{:exchange_error, provider, _}` when
  `:provider` is given), is never retried, and is reported on `[:dp_exchange, :request,
  :stop]` with `result: :error, reason: :response_too_large`. A body of exactly the limit is
  accepted. A positive integer, or `ArgumentError`. The count is of the bytes on the wire:
  this pipeline never asks for compression (`Req`'s `:compressed` is off), so it is also the
  decoded size.
- `:raw_status` - Return `{:ok, response}` for a 4xx instead of an error string
  (default: `false`). A venue package needs this to tell a **refusal** from an
  **error**: the contract makes `{:refused, reason}` permanent and `{:error, reason}`
  possibly transient, and the venue states which in the 4xx status and body. Without
  it that evidence is flattened into a message and the venue has to string-match its
  way back to it. 5xx is unaffected — a server error is not a venue's considered answer.
  **429 is unaffected too**: it is the venue's rate limit, handled by this pipeline
  (`Retry-After`, the held bucket, telemetry), and is always the "Rate limited by the venue"
  error, whether or not `:raw_status` is set.
- `:decode_body` - Passed to `Req` (default: Req's, which decodes JSON with floats).
  `false` returns the body as bytes, for a venue that types money as a JSON number and
  must decode it without float rounding (`Jason.decode(body, floats: :decimals)`).

## Returns
- `{:ok, http_response()}` on success, and on a 4xx when `raw_status: true`
- `{:error, reason}` on failure

---

*Consult [api-reference.md](api-reference.md) for complete listing*