Editable Image Elements for Controllable Synthesis

{
  "by": "jasondavies",
  "descendants": 0,
  "id": 40248847,
  "score": 2,
  "time": 1714750395,
  "title": "Editable Image Elements for Controllable Synthesis",
  "type": "story",
  "url": "https://jitengmu.github.io/Editable_Image_Elements/"
}

{
  "author": "Jiteng Mu",
  "date": null,
  "description": "Editable Image Elements for Controllable Synthesis.",
  "image": "https://jitengmu.github.io/Editable_Image_Elements/images/teaser.jpg",
  "logo": null,
  "publisher": null,
  "title": "Editable Image Elements for Controllable Synthesis",
  "url": "https://jitengmu.github.io/Editable_Image_Elements"
}

{
  "url": "https://jitengmu.github.io/Editable_Image_Elements",
  "title": "Editable Image Elements for Controllable Synthesis",
  "description": "(ECCV 2024)          1University of California San Diego,            2Adobe Research                                      We propose editable image elements, that can faithfully reconstruct an input...",
  "links": [
    "https://jitengmu.github.io/Editable_Image_Elements",
    "https://jitengmu.github.io/Editable_Image_Elements/"
  ],
  "image": "https://jitengmu.github.io/Editable_Image_Elements/images/teaser.jpg",
  "content": "<div>\n          <h3>(ECCV 2024)</h3>\n          <p><span><sup>1</sup>University of California San Diego,</span>\n            <span><sup>2</sup>Adobe Research</span>\n          </p>\n        </div>\n<div>\n    <h2>\n       <p> \n        We propose editable image elements, that can faithfully reconstruct an input image, while enabling various spatial editing operations.\n       </p>\n    </h2>\n    </div>\n<div>\n      <video autoplay muted loop height=\"100%\">\n        <source src=\"https://jitengmu.github.io/static/images/teaser_car.mp4\" type=\"video/mp4\"></source>\n      </video>\n    <h2>\n       <p> \n        The user simply selects interesting image elements and edits their locations and sizes. Our model automatically decoded the edited elements into a realistic image. \n        The example shows the sequential editing results achieved with our method.\n       </p>\n    </h2>\n    </div>\n<div>\n    <div>\n        <h2>Abstract</h2>\n        <p>\n    Diffusion models have made significant advances in text-guided synthesis tasks. \n    However, editing user-provided images remains challenging, as the high dimensional noise input space of diffusion models is not naturally suited for image inversion or spatial editing. \n    In this work, we propose an image representation that promotes spatial editing of input images using a diffusion model. Concretely, we learn to encode an input into `image elements' that can faithfully reconstruct an input image. \n    These elements can be intuitively edited by a user, and are decoded by a diffusion model into realistic images. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition.\n          </p>\n      </div>\n    <div>\n        <h2>Video</h2>\n        <p>\n          <iframe src=\"https://youtube.com/embed/vAhy-sNzsTg?rel=0&amp;showinfo=0\" frameborder=\"0\" allow=\"autoplay; encrypted-media\"></iframe>\n        </p>\n      </div>\n  </div>\n<div>\n    <h2>Approach</h2>\n    <div>\n      <p> \n          To encode the image, we extract features from <a target=\"_blank\" href=\"https://segment-anything.com/\"> Segment Anything Model </a> with equally spaced query points and perform simple clustering to obtain grouping of object parts \n          with comparable sizes, resembling superpixels. Each element is individually encoded with our convolutional encoder and is associated with its centroid and size parameters \n          to form image elements. The user can directly modify the image elements, such as moving, resizing, or removing. We pass the modified image elements to our diffusion-based \n          decoder along with a text description of the overall scene to synthesize a realistic image that respects the modified elements.\n        </p>\n    </div>\n  </div>\n<div>\n    <h2>Spatial Editing</h2>\n    <h2>\n       <p> \n        The user can directly edit the image elements with simple selection, dragging, resizing, and deletion operations. \n        The selected and edited elements are highlighted with red and green dots at the centroid of each element.\n       </p>\n    </h2>\n    </div>\n<div>\n    <h2>Object Removal</h2>\n    <h2>\n       <p> \n        The user selects elements to delete (shown in blue), and provide a text prompt pertaining to the background. Our diffusion decoder can generate content\n        in the missing region (in black).\n       </p>\n    </h2>\n    </div>\n<div>\n    <h2>Object Variations</h2>\n    <h2>\n       <p> \n        Our method supports object variations by deleting some image elements (shown in black in Target Edit), \n        and performing inpainting guided by text prompt and the remaining image elements, such as the ``beak'' element in the bird example.\n       </p>\n    </h2>\n    </div>\n<div>\n    <h2>BibTeX</h2>\n    <pre><code>@article{mu2024editable,\n         author = {Mu, Jiteng and Gharbi, Michaël and Zhang, Richard and Shechtman, Eli \n                          and Vasconcelos, Nuno and Wang, Xiaolong and Park, Taesung},\n         title = {Editable Image Elements for Controllable Synthesis},\n         journal={arXiv preprint arXiv:2404.16029},\n         year={2024}}</code>\n     </pre>\n  </div>",
  "author": "",
  "favicon": "",
  "source": "jitengmu.github.io",
  "published": "",
  "ttr": 84,
  "type": ""
}