Delimited Generators – A more natural API for JS generators

http://axisofeval.blogspot.com/2024/05/delimited-generators.html

I have been studying ways to work around the horrors issues of JavaScript's async APIs for years. I have even built a series of increasingly elaborate continuation-based Lisp interpreters (here's the latest one, it's quite good, if I may say so).

But recently I finally came to the point where I understood JS generators well enough to realize that generators already solve the problem! With a small constant syntactic overhead (having to use function* to define generators, having to use yield* to call them, and using next() to call them from non-generator functions), one can program asynchronous code in a quasi-direct style.

But the plain generator interface is rather low-level, and not very intuitive to use. So I built delimgen, a thin layer on top of plain generators, that mimics delimited control. Delimited control is initially hard to understand, but once you grok it it's a very natural approach (previous post).

Here's how a simple delimited generator looks like. See it in action here.

You can easily spawn multiple independent generators. See it in action here.

You can also do blocking event loops easily. See it in action here.

I'm not claiming any novelty here. You also cannot do anything with this library that you couldn't do with plain generators, but for me, seeing that you can write quasi-blocking code in JS with some small overhead was a real eye opener.

Update: Thanks to mmastrac on HN for pointing out that this leaks stack - back to the drawing board!

Update 2: Turns out the supposed stack leak was just an artefact of the "async call stack" feature of browser devtools, which keeps stacks for promises around for debugging, even though there isn't any real stack growth when running normally (outside of devtools).

(See also: Lobsters, with an alternative formulation using async/await by easrng)

{
  "by": "jchanimal",
  "descendants": 33,
  "id": 40236923,
  "kids": [
    40240858,
    40241428,
    40246837,
    40268868,
    40241313,
    40241068,
    40240785,
    40242285,
    40240905,
    40267009
  ],
  "score": 20,
  "time": 1714661237,
  "title": "Delimited Generators – A more natural API for JS generators",
  "type": "story",
  "url": "http://axisofeval.blogspot.com/2024/05/delimited-generators.html"
}

{
  "author": "Manuel Simoni",
  "date": "2024-12-21T11:23:00.000Z",
  "description": "I have been studying ways to work around the horrors issues of JavaScript’s async APIs for years. I have even built a series of increasing…",
  "image": "https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKEf33j5WZSVgQ_xMLeRI5zhinU-jDXY5SaF-Kxb9E4rdhc3_HOlnikE6-oYinv5VvN3q7versNsFdFOt39WzDTcrF3zYWdQcbCJMsKV8Le1VcRp_Dur5xub02rWE5meb5jkEIlMRvLSs/s1600/aoe.png",
  "logo": null,
  "publisher": "A more natural API for JS generators",
  "title": "Delimited Generators - A more natural API for JS generators",
  "url": "http://axisofeval.blogspot.com/2024/05/delimited-generators.html"
}

{
  "url": "http://axisofeval.blogspot.com/2024/05/delimited-generators.html",
  "title": "Delimited Generators - A more natural API for JS generators",
  "description": "I have been studying ways to work around the horrors issues of JavaScript's async APIs for years. I have even built a series of increasingly elaborate continuation-based Lisp interpreters (here's the latest...",
  "links": [
    "http://axisofeval.blogspot.com/2024/05/delimited-generators.html"
  ],
  "image": "",
  "content": "<div>\n<p>I have been studying ways to work around <b>the horrors issues of JavaScript's async APIs</b> for years. I have even built a series of increasingly elaborate continuation-based Lisp interpreters (here's <a target=\"_blank\" href=\"https://github.com/lispx/lispx/\">the latest one</a>, it's quite good, if I may say so).</p><p>But recently I finally came to the point where I understood JS generators well enough to realize that <b>generators already solve the problem!</b> With a small constant syntactic overhead (having to use <b>function*</b> to define generators, having to use <b>yield*</b> to call them, and using <b>next() </b>to call them from non-generator functions), one can program asynchronous code in a <b>quasi-direct style</b>.</p><p>But the plain generator interface is rather low-level, and not very intuitive to use. So<b> I built <a target=\"_blank\" href=\"https://github.com/manuel/delimgen\">delimgen</a>, a thin layer on top of plain generators</b>, that mimics delimited control. Delimited control is initially hard to understand, but once you grok it it's a very natural approach (<a target=\"_blank\" href=\"http://axisofeval.blogspot.com/2011/08/notes-on-delimited-continuations.html\">previous post</a>).</p><p>Here's how a simple delimited generator looks like. <a target=\"_blank\" href=\"https://manuel.github.io/delimgen/demo/single-sleeper.html\">See it in action here.</a></p>\n<p>You can easily spawn multiple independent generators. <a target=\"_blank\" href=\"https://manuel.github.io/delimgen/demo/multiple-sleepers.html\">See it in action here.</a></p>\n<p>You can also do blocking event loops easily. <a target=\"_blank\" href=\"https://manuel.github.io/delimgen/demo/balls.html\">See it in action here.</a></p>\n<p>I'm not claiming any novelty here.  You also cannot do anything with this library that you couldn't do with plain generators, but for me, seeing that you can write quasi-blocking code in JS with some small overhead was a real eye opener.</p>\n<p><b>Update:</b> Thanks to mmastrac on <a target=\"_blank\" href=\"https://news.ycombinator.com/item?id=40236923\">HN</a> for pointing out that this leaks stack - back to the drawing board!</p>\n<p><b>Update 2:</b> <a target=\"_blank\" href=\"https://news.ycombinator.com/item?id=40245182\">Turns out</a> the supposed stack leak was just an artefact of the \"async call stack\" feature of browser devtools, which keeps stacks for promises around for debugging, even though there isn't any real stack growth when running normally (outside of devtools).</p><p>\n(See also: <a target=\"_blank\" href=\"https://lobste.rs/s/tr9lkd/delimited_generators_more_natural_api\">Lobsters</a>, with an alternative formulation using async/await by easrng)\n</p>\n</div>",
  "author": "",
  "favicon": "http://axisofeval.blogspot.com/favicon.ico",
  "source": "axisofeval.blogspot.com",
  "published": "",
  "ttr": 61,
  "type": ""
}