The CORS Error That Was Really a Memory Problem

One day, a user tried to do something simple: transfer a CSV file with 645,000 rows (answers.csv). But the browser console showed this:

Access to XMLHttpRequest at '.../api/steps/24186/run' blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
POST .../run net::ERR_FAILED

This looks like a classic CORS error. So you go and fix the origin setting in the backend, right? I didn't. Because the CORS config was already this:

origins "*"

Everything is allowed. So this could not be a real CORS problem. The header was missing because the server crashed before rack-cors could add it. The browser can't see that. It only sees "no CORS header in the response" and points you in the wrong direction.

Lesson 1: If you see a CORS error while origins "*" is set, the problem is not CORS. The server can't respond. Go read the logs.

The logs told the truth

08:55:29 heroku[web.1]: Error R14 (Memory quota exceeded)

R14. Out of memory. The web dyno crashed while handling the /run request. But why would a "start the job" request use so much memory?

Fake chunking

The code split large CSVs into chunks of 10,000 rows. That sounds safe. But the chunking worked like this:

file = read_csv_file(...)            # loads ALL 645k rows into RAM
file.drop(offset).take(chunk_size)   # then takes 10k of them

So every chunk loaded the whole file into memory before doing its work. The chunking never limited memory. It only sliced the data after the full load. 645,000 CSV::Row objects means gigabytes. It was a false sense of safety.

Lesson 2: "We chunk it" does not mean "memory is bounded." Look at the order — where does it load, and where does it slice?

Layers, like an onion

I thought I fixed the problem in one place, but it showed up in another. There were three separate full-load points:

1. Worker read — every chunk job parsed all 645k rows again.
2. Web count — /run needed to know the row count to plan the chunks... so it loaded the whole file again, just to call .count.
3. Raw download — at the very bottom, read_as_utf8 downloaded the 183 MB file into a single String (plus one more copy if it needed re-encoding).

I changed the first two to streaming: read row by row (csv.shift), count row by row (each_csv_row). Only one row in RAM at a time.

To learn how many pages a book has, you don't put every page on the table. You turn them one by one and count.

Don't fix without measuring

After two pull requests, it was still on the edge. Instead of guessing, I measured — I ran the real production file on a one-off dyno:

raw = 183 MB
streaming count = 21.6 s
peak RSS = 558 MB

Here is the important number: 558 MB, even with streaming. Because the bottom layer, read_as_utf8, still pulled the full 183 MB into a String. The web dyno had 1 GB. A 558 MB spike + the app's base memory + other requests at the same time = too close to both the 1 GB limit and the 30-second router timeout.

Lesson 3: Streaming removes the object pile (CSV::Table), but it does not remove the cost of downloading the raw data. You can't know which layer is expensive until you measure.

The end: quick patch vs. real fix

The user was blocked, so I was pragmatic: I resized the web dyno to Performance-M (2.5 GB). The transfer finished. But this is a patch, not a solution.

The real fix is in the design: don't do heavy work inside a web request. Counting a 183 MB file synchronously is not the web dyno's job. The codebase already had the right pattern (another count endpoint did this off-request, with a Sidekiq job + Redis cache + polling). The /run path just wasn't using it.

Lesson 4: If a user request takes 30 seconds, it is running in the wrong place. Put it on a queue, return right away, and let the client poll.

Summary

• origins "*" + a CORS error = the server is crashing, not CORS.
• "Chunking" does not automatically mean "memory-safe."
• Bugs come in layers; when you close one full-load, the next one appears.
• Measure before you fix. I didn't know I was on the edge until I saw 558 MB.
• Heavy work belongs in the background, not in a web request.