Making many improvements on the user friendliness of the colorization notebooks

ImageColorizer.ipynb

@@ -0,0 +1,180 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ['CUDA_VISIBLE_DEVICES']='1' "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import print_function\n",
+    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
+    "import ipywidgets as widgets"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from fasterai.visualize import *\n",
+    "plt.style.use('dark_background')\n",
+    "torch.backends.cudnn.benchmark=True"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "NOTE:  Set artistic to False if you're having trouble getting a good render.  Chances are it will work with the Stable model. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "colorizer = get_image_colorizer(artistic=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Instructions\n",
+    "\n",
+    "### source_url\n",
+    "Type in a url to a direct link of an image.  Usually that means they'll end in .png, .jpg, etc.  NOTE: If you want to use your own image, you can set source_url to None and just upload the image to /test_images/ in Jupyter.  Just make sure that the source_path parameter matches the file you uploaded.\n",
+    "\n",
+    "### source_path\n",
+    "Name this whatever sensible image path (plus extension of jpg/png/ext) you want!  Sensible means the path exists and the file exists if source_url=None.\n",
+    "\n",
+    "### render_factor\n",
+    "The default value of 35 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which the color portion of the image is rendered. Lower resolution will render faster, and colors also tend to look more vibrant. Older and lower quality images in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality images, but the colors may get slightly washed out. \n",
+    "\n",
+    "### result_path\n",
+    "Ditto- don't change.\n",
+    "\n",
+    "### How to Download a Copy\n",
+    "Simply shift+right click on the displayed image and click \"Save Image As...\"!\n",
+    "\n",
+    "\n",
+    "## Pro Tips\n",
+    "1. You can evaluate how well the image is rendered at each render_factor by using the code at the bottom (that cell under \"See how well render_factor values perform on a frame here\"). \n",
+    "2. Keep in mind again that you can go up top and set artistic to False for the colorizer to use the 'Stable' model instead.  This will often tend to do better on portraits, and natural landscapes.  \n",
+    "\n",
+    "\n",
+    "## Troubleshooting\n",
+    "If you get a 'CUDA out of memory' error, you probably have the render_factor too high.  The max is 45 on 11GB video cards."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Colorize!!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#NOTE:  Max is 45 with 11GB video cards. 35 is a good default\n",
+    "render_factor=35\n",
+    "#NOTE:  Make source_url None to just read from file at ./video/source/[file_name] directly without modification\n",
+    "source_url='https://i.imgur.com/NajrX6Z.jpg'\n",
+    "source_path = 'test_images/image.png'\n",
+    "result_path = None\n",
+    "\n",
+    "if source_url is not None:\n",
+    "    result_path = colorizer.plot_transformed_image_from_url(url=source_url, path=source_path, render_factor=render_factor)\n",
+    "else:\n",
+    "    result_path = colorizer.plot_transformed_image(path=source_path, render_factor=render_factor)\n",
+    "\n",
+    "show_image_in_notebook(result_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## See how well render_factor values perform on the image here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i in range(5,46):\n",
+    "    colorizer.plot_transformed_image(source_path, render_factor=i, display_render_factor=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.0"
+  },
+  "toc": {
+   "colors": {
+    "hover_highlight": "#DAA520",
+    "navigate_num": "#000000",
+    "navigate_text": "#333333",
+    "running_highlight": "#FF0000",
+    "selected_highlight": "#FFD700",
+    "sidebar_border": "#EEEEEE",
+    "wrapper_background": "#FFFFFF"
+   },
+   "moveMenuLeft": true,
+   "nav_menu": {
+    "height": "67px",
+    "width": "252px"
+   },
+   "navigate_menu": true,
+   "number_sections": true,
+   "sideBar": true,
+   "threshold": 4,
+   "toc_cell": false,
+   "toc_section_display": "block",
+   "toc_window_display": false,
+   "widenNotebook": false
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

ImageColorizerColab.ipynb

@@ -158,31 +158,37 @@
     "colorizer = get_image_colorizer(artistic=True)"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "colab_type": "text",
-    "id": "sUQrbSYipiJn"
-   },
-   "source": [
-    "#◢ Colorize!!"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Instructions\n",
+    "# Instructions\n",
     "\n",
-    "#### source_url\n",
-    "Any direct link to an image should do (will end with extension .jpg, .png, etc). NOTE: If you want to use your own image, upload it first to a site like Imgur. \n",
+    "### source_url\n",
+    "Type in a url to a direct link of an image.  Usually that means they'll end in .png, .jpg, etc. NOTE: If you want to use your own image, upload it first to a site like Imgur. \n",
     "\n",
-    "\n",
-    "#### render_factor\n",
-    "The default value of 35 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which the photo is rendered. Lower resolution will render faster, and colors also tend to look more vibrant.  Older and lower quality photos in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality photos.\n",
+    "### render_factor\n",
+    "The default value of 35 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which the color portion of the image is rendered. Lower resolution will render faster, and colors also tend to look more vibrant. Older and lower quality images in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality images, but the colors may get slightly washed out. \n",
     "\n",
     "#### How to Download a Copy\n",
-    "Simply right click on the displayed image and click \"Save image as...\"!"
+    "Simply right click on the displayed image and click \"Save image as...\"!\n",
+    "\n",
+    "## Pro Tips\n",
+    "1. You can evaluate how well the image is rendered at each render_factor by using the code at the bottom (that cell under \"See how well render_factor values perform on a frame here\"). \n",
+    "2. Keep in mind again that you can go up top and set artistic to False for the colorizer to use the 'Stable' model instead.  This will often tend to do better on portraits, and natural landscapes.  \n",
+    "\n",
+    "## Troubleshooting\n",
+    "If you get a 'CUDA out of memory' error, you probably have the render_factor too high.  The max is 45 on 11GB video cards."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "sUQrbSYipiJn"
+   },
+   "source": [
+    "#◢ Colorize!!"
    ]
   },
   {
@@ -192,7 +198,7 @@
    "outputs": [],
    "source": [
     "source_url = '' #@param {type:\"string\"}\n",
-    "render_factor = 35  #@param {type: \"slider\", min: 7, max: 46}\n",
+    "render_factor = 35  #@param {type: \"slider\", min: 7, max: 45}\n",
     "\n",
     "if source_url is not None and source_url !='':\n",
     "    image_path = colorizer.plot_transformed_image_from_url(url=source_url, render_factor=render_factor)\n",
@@ -201,6 +207,23 @@
     "    print('Provide an image url and try again.')"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## See how well render_factor values perform on the image here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i in range(5,46):\n",
+    "    colorizer.plot_transformed_image(source_path, render_factor=i, display_render_factor=True)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {

VideoColorizer.ipynb

@@ -27,33 +27,58 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "#Adjust render_factor (int) if image doesn't look quite right (max 45 on 11GB GPU).  \n",
-    "#Lower render factors (as low as 12-15) tend to work well for old and low quality videos.\n",
-    "#High render factors (25-45) tend to work well for higher quality and more recent videos\n",
+    "colorizer = get_video_colorizer()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Instructions\n",
     "\n",
-    "#Not satisfied with color saturation?  Lower the render factor.  \n",
-    "#Unacceptable object flicker?  Increase the render factor.\n",
+    "### source_url\n",
+    "Type in a url hosting a video from YouTube, Imgur, Twitter, Reddit, Vimeo, etc.  Many sources work!  GIFs also work.  Full list here: https://ytdl-org.github.io/youtube-dl/supportedsites.html NOTE: If you want to use your own video, you can set source_url to None and just upload the file to video/source/ in Jupyter.  Just make sure that the file_name parameter matches the file you uploaded.\n",
     "\n",
-    "#It literally just is a number multiplied by 16 to get the square render resolution.  \n",
-    "#Note that this doesn't affect the resolution of the final output- the output is the same resolution as the input.\n",
-    "#Example:  render_factor=21 => color is rendered at 16x21 = 336x336 px.  \n",
-    "render_factor=21\n",
-    "#Specify media_url. Many sources will work (YouTube, Imgur, Twitter, Reddit, etc). \n",
-    "#Complete list here: https://rg3.github.io/youtube-dl/supportedsites.html. \n",
-    "#NOTE:  Make source_url None to just read from file at ./video/source/[file_name] directly without modification\n",
-    "#source_url= 'https://twitter.com/silentmoviegifs/status/1112256563182489600'\n",
-    "#source_url='https://archive.org/details/impact'\n",
-    "source_url='https://twitter.com/silentmoviegifs/status/1116751583386034176'\n",
-    "file_name = 'DogBath.mp4'"
+    "\n",
+    "### file_name\n",
+    "Name this whatever sensible file name you want (minus extension)! It should actually exist in video/source if source_url=None\n",
+    "\n",
+    "\n",
+    "### render_factor\n",
+    "The default value of 21 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which the color portion of the video is rendered. Lower resolution will render faster, and colors also tend to look more vibrant. Older and lower quality film in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality videos and inconsistencies (flashy render) will generally be reduced, but the colors may get slightly washed out. \n",
+    "\n",
+    "\n",
+    "### file_name_ext\n",
+    "There's no reason to changes this.\n",
+    "\n",
+    "\n",
+    "### result_path\n",
+    "Ditto- don't change.\n",
+    "\n",
+    "\n",
+    "### How to Download a Copy\n",
+    "Simply shift+right click on the displayed video and click \"Save video as...\"!\n",
+    "\n",
+    "\n",
+    "## Pro Tips\n",
+    "1. If a video takes a long time to render and you're wondering how well the frames will actually be colorized, you can preview how well the frames will be rendered at each render_factor by using the code at the bottom. Just stop the video rendering by hitting the stop button on the cell, then run that bottom cell under \"See how well render_factor values perform on a frame here\". It's not perfect and you may still need to experiment a bit especially when it comes to figuring out how to reduce frame inconsistency.  But it'll go a long way in narrowing down what actually works.\n",
+    "\n",
+    "\n",
+    "## Troubleshooting\n",
+    "The video player may wind up not showing up, in which case- make sure to wait for the Jupyter cell to complete processing first (the play button will stop spinning).  Then follow these alternative download instructions\n",
+    "\n",
+    "1. In the menu to the left, click Home icon.\n",
+    "2. By default, rendered video will be in /video/result/\n",
+    "\n",
+    "If a video you downloaded doesn't play, it's probably because the cell didn't complete processing and the video is in a half-finished state.\n",
+    "If you get a 'CUDA out of memory' error, you probably have the render_factor too high.  The max is 44 on 11GB video cards."
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "colorizer = get_video_colorizer()"
+    "## Colorize!!"
    ]
   },
   {
@@ -62,12 +87,27 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "#NOTE:  Max is 44 with 11GB video cards.  21 is a good default\n",
+    "render_factor=21\n",
+    "#NOTE:  Make source_url None to just read from file at ./video/source/[file_name] directly without modification\n",
+    "source_url='https://twitter.com/silentmoviegifs/status/1116751583386034176'\n",
+    "file_name = 'DogShy1926'\n",
+    "file_name_ext = file_name + '.mp4'\n",
+    "result_path = None\n",
+    "\n",
     "if source_url is not None:\n",
-    "    video_path = colorizer.colorize_from_url(source_url, file_name, render_factor=render_factor)\n",
-    "    show_video_in_notebook(video_path)\n",
+    "    result_path = colorizer.colorize_from_url(source_url, file_name_ext, render_factor=render_factor)\n",
     "else:\n",
-    "    video_path = colorizer.colorize_from_file_name(file_name)\n",
-    "    show_video_in_notebook(video_path)"
+    "    result_path = colorizer.colorize_from_file_name(file_name_ext)\n",
+    "\n",
+    "show_video_in_notebook(result_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## See how well render_factor values perform on a frame here"
    ]
   },
   {
@@ -75,7 +115,10 @@
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "for i in range(5,44):\n",
+    "    colorizer.vis.plot_transformed_image('video/bwframes/' + file_name + '/00001.jpg', render_factor=i, display_render_factor=True)"
+   ]
   }
  ],
  "metadata": {

VideoColorizerColab.ipynb

@@ -161,43 +161,45 @@
     "colorizer = get_video_colorizer()"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "colab_type": "text",
-    "id": "sUQrbSYipiJn"
-   },
-   "source": [
-    "#◢ Colorize!!"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Instructions\n",
+    "# Instructions\n",
     "\n",
-    "#### source_url\n",
-    "YouTube, Imgur, Twitter, Reddit, Vimeo, etc.  Many sources work!  GIFs also work.  Full list here: https://ytdl-org.github.io/youtube-dl/supportedsites.html NOTE: If you want to use your own video, upload it first to a site like YouTube. \n",
+    "### source_url\n",
+    "Type in a url hosting a video from YouTube, Imgur, Twitter, Reddit, Vimeo, etc.  Many sources work!  GIFs also work.  Full list here: https://ytdl-org.github.io/youtube-dl/supportedsites.html NOTE: If you want to use your own video, upload it first to a site like YouTube. \n",
     "\n",
-    "#### render_factor\n",
-    "The default value of 21 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which video is rendered. Lower resolution will render faster, and colors also tend to look more vibrant. Older and lower quality film in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality videos and inconsistencies (flashy render) will generally be reduced, but the colors may get slightly washed out. \n",
+    "### render_factor\n",
+    "The default value of 21 has been carefully chosen and should work -ok- for most scenarios (but probably won't be the -best-). This determines resolution at which the color portion of the video is rendered. Lower resolution will render faster, and colors also tend to look more vibrant. Older and lower quality film in particular will generally benefit by lowering the render factor. Higher render factors are often better for higher quality videos and inconsistencies (flashy render) will generally be reduced, but the colors may get slightly washed out. \n",
     "\n",
-    "#### How to Download a Copy\n",
+    "### How to Download a Copy\n",
     "Simply right click on the displayed video and click \"Save video as...\"!\n",
     "\n",
-    "#### Pro Tips\n",
-    "1. If a video takes a long time to render and you're wondering how well the frames are being colorized, you can preview these colorized frames by going to the menu on the left, clicking \"Files\", and go to /DeOldify/video/colorframes/video. You can then simply double click on the images and get a preview in the browser.\n",
+    "## Pro Tips\n",
+    "1. If a video takes a long time to render and you're wondering how well the frames will actually be colorized, you can preview how well the frames will be rendered at each render_factor by using the code at the bottom. Just stop the video rendering by hitting the stop button on the cell, then run that bottom cell under \"See how well render_factor values perform on a frame here\". It's not perfect and you may still need to experiment a bit especially when it comes to figuring out how to reduce frame inconsistency.  But it'll go a long way in narrowing down what actually works.\n",
     "2. If videos are taking way too much time for your liking, running the Jupyter notebook VideoColorizer.ipynb on your own machine (with DeOldify installed) will generally be much faster (as long as you have the hardware for it).   \n",
     "\n",
-    "#### Troubleshooting\n",
+    "## Troubleshooting\n",
     "The video player may wind up not showing up, in which case- make sure to wait for the Jupyter cell to complete processing first (the play button will stop spinning).  Then follow these alternative download instructions\n",
     "\n",
     "1. In the menu to the left, click Files\n",
     "2. If you don't see the 'DeOldify' folder, click \"Refresh\"\n",
     "3. By default, rendered video will be in /DeOldify/video/result/\n",
     "\n",
-    "If a video you downloaded doesn't play, it's probably because the cell didn't complete processing and the video is in a half-finished state."
+    "If a video you downloaded doesn't play, it's probably because the cell didn't complete processing and the video is in a half-finished state.\n",
+    "\n",
+    "If you get a 'CUDA out of memory' error, you probably have the render_factor too high. The max is 44 on 11GB video cards."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "colab_type": "text",
+    "id": "sUQrbSYipiJn"
+   },
+   "source": [
+    "#◢ Colorize!!"
    ]
   },
   {
@@ -207,7 +209,7 @@
    "outputs": [],
    "source": [
     "source_url = '' #@param {type:\"string\"}\n",
-    "render_factor = 21  #@param {type: \"slider\", min: 7, max: 46}\n",
+    "render_factor = 21  #@param {type: \"slider\", min: 5, max: 44}\n",
     "\n",
     "if source_url is not None and source_url !='':\n",
     "    video_path = colorizer.colorize_from_url(source_url, 'video.mp4', render_factor)\n",
@@ -216,6 +218,23 @@
     "    print('Provide a video url and try again.')"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## See how well render_factor values perform on a frame here"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for i in range(5,44):\n",
+    "    colorizer.vis.plot_transformed_image('video/bwframes/' + file_name + '/00001.jpg', render_factor=i, display_render_factor=True)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {

fasterai/visualize.py

@@ -34,21 +34,23 @@ class ModelImageVisualizer():
 
     def _get_image_from_url(self, url:str)->Image:
         response = requests.get(url)
-        img = Image.open(BytesIO(response.content)).convert('RGB')
+        img = PIL.Image.open(BytesIO(response.content)).convert('RGB')
         return img
 
-    def plot_transformed_image_from_url(self, url:str, path:str='test_images/image.png', figsize:(int,int)=(20,20), render_factor:int=None)->Path:
+    def plot_transformed_image_from_url(self, url:str, path:str='test_images/image.png', figsize:(int,int)=(20,20), 
+            render_factor:int=None, display_render_factor:bool=False)->Path:
         img = self._get_image_from_url(url)
         img.save(path)
-        return self.plot_transformed_image(path=path, figsize=figsize, render_factor=render_factor)
+        return self.plot_transformed_image(path=path, figsize=figsize, render_factor=render_factor, 
+                                            display_render_factor=display_render_factor)
 
-    def plot_transformed_image(self, path:str, figsize:(int,int)=(20,20), render_factor:int=None)->Path:
+    def plot_transformed_image(self, path:str, figsize:(int,int)=(20,20), render_factor:int=None, display_render_factor:bool=False)->Path:
         path = Path(path)
         result = self.get_transformed_image(path, render_factor)
         orig = self._open_pil_image(path)
         fig,axes = plt.subplots(1, 2, figsize=figsize)
-        self._plot_image(orig, axes=axes[0], figsize=figsize)
-        self._plot_image(result, axes=axes[1], figsize=figsize)
+        self._plot_image(orig, axes=axes[0], figsize=figsize, render_factor=render_factor, display_render_factor=False)
+        self._plot_image(result, axes=axes[1], figsize=figsize, render_factor=render_factor, display_render_factor=display_render_factor)
         return self._save_result_image(path, result)
 
     def _save_result_image(self, source_path:Path, image:Image)->Path:
@@ -62,11 +64,13 @@ class ModelImageVisualizer():
         filtered_image = self.filter.filter(orig_image, orig_image, render_factor=render_factor)
         return filtered_image
 
-    def _plot_image(self, image:Image, axes:Axes=None, figsize=(20,20)):
+    def _plot_image(self, image:Image, render_factor:int, axes:Axes=None, figsize=(20,20), display_render_factor:bool=False):
         if axes is None: 
             _,axes = plt.subplots(figsize=figsize)
         axes.imshow(np.asarray(image)/255)
         axes.axis('off')
+        if render_factor is not None and display_render_factor:
+            plt.text(10,10,'render_factor: ' + str(render_factor), backgroundcolor='black')
 
     def _get_num_rows_columns(self, num_images:int, max_columns:int)->(int,int):
         columns = min(num_images, max_columns)