Go Module Bloat

https://flak.tedunangst.com/post/go-module-bloat

For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.

 317140 May  6  2019 humungus-0.9.6.tgz
3094567 Apr 30 22:25 humungus-0.9.7.tgz

There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?

I poked around the vendor directory for a bit for some large files. One of them is hsluv-snapshot-rev4.json, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long.

I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and go-colorful was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running go vendor and everything still works. That reduced the size of the final release tarball by 500K.

2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.

Ah, there it is. tcell nonblock_bsd.go imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.

I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on go.mod to make sure it doesn’t explode. But I’ve been lax in running du on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.

Posted 01 May 2024 18:46 by tedu Updated: 01 May 2024 22:38
Tagged: go programming

{
  "by": "mfrw",
  "descendants": 1,
  "id": 40232725,
  "kids": [
    40237111,
    40232902
  ],
  "score": 12,
  "time": 1714624405,
  "title": "Go Module Bloat",
  "type": "story",
  "url": "https://flak.tedunangst.com/post/go-module-bloat"
}

{
  "author": null,
  "date": null,
  "description": null,
  "image": null,
  "logo": null,
  "publisher": null,
  "title": "go module bloat",
  "url": "https://flak.tedunangst.com/post/go-module-bloat"
}

{
  "url": "https://flak.tedunangst.com/post/go-module-bloat",
  "title": "go module bloat",
  "description": "For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five...",
  "links": [
    "https://flak.tedunangst.com/post/go-module-bloat"
  ],
  "image": "",
  "content": "<div>\n<p>For some of the software I write, I try to make occasional releases, and for the go software I vendor the dependencies so it’s all there. I was just reminded that I hadn’t made a release of something in five years, so I reran my release script, and ended up with a tarball that was 10x bigger than the previous one. Some terrible choices have been made.</p><pre><code> 317140 May  6  2019 humungus-0.9.6.tgz\n3094567 Apr 30 22:25 humungus-0.9.7.tgz</code></pre><p>There’s been five years of development, but that’s really not that much. I most certainly have not 10xed the functionality. Where did this all come from?</p><p>I poked around the vendor directory for a bit for some large files. One of them is <a target=\"_blank\" href=\"https://github.com/lucasb-eyer/go-colorful/blob/master/hsluv-snapshot-rev4.json\">hsluv-snapshot-rev4.json</a>, a 1.5MB json file. I wouldn’t recommend clicking on that link, unless your browser likes displaying single lines of text 1.5MB bytes long. </p><p>I have no idea what this file does. The only color I need is for a little green menu in the terminal admin interface. I checked, and <i>go-colorful</i> was a dependency in previous releases, but it didn’t have this file at the time. Apparently I still don’t need it. I just deleted it after running <i>go vendor</i> and everything still works. That reduced the size of the final release tarball by 500K.</p><p>2MB (post compression) still to go, or thereabouts. I was in a hurry, so I didn’t actually trim anything more down. The majority of the code is the golang/sys/unix repository. I still have to find out what’s dragging this in, and then delete or rewrite it.</p><p>Ah, there it is. <a target=\"_blank\" href=\"https://github.com/gdamore/tcell/blob/main/nonblock_bsd.go\">tcell nonblock_bsd.go</a> imports sys/unix. 9.5MB of vendored source code to set a flag on a file descriptor.</p><p>I have tried to be mindful of the number of dependencies I’m adding. I keep an eye on <i>go.mod</i> to make sure it doesn’t explode. But I’ve been lax in running <i>du</i> on the vendor directory. Even limited sets of dependencies are subject to extreme bloat.\n</p></div><p>\nPosted 01 May 2024 18:46 by tedu Updated: 01 May 2024 22:38 \n<br />Tagged: <a target=\"_blank\" href=\"https://flak.tedunangst.com/t/go\">go</a> <a target=\"_blank\" href=\"https://flak.tedunangst.com/t/programming\">programming</a>\n</p>",
  "author": "",
  "favicon": "https://flak.tedunangst.com/icon.png",
  "source": "flak.tedunangst.com",
  "published": "",
  "ttr": 68,
  "type": ""
}