Thief: The Dark Project intro briefing video, upscaled to 1080p via machine learning - by goldwell
goldwell on 1/6/2019 at 20:52
[video=youtube;Zs4CYxB0PGI]https://www.youtube.com/watch?v=Zs4CYxB0PGI[/video]
I have seen quite a few older games have their cinematics and textures upscaled to more modern resolutions using machine learning powered applications. I decided to give one of the Thief videos a shot and this was the result. Took just under 2 hours to process.
goldwell on 1/6/2019 at 21:34
And just for fun I decided to do the success.avi file too!
[video=youtube;R-QeMjSxtLI]https://www.youtube.com/watch?v=R-QeMjSxtLI&feature=youtu.be[/video]
Mat99 on 2/6/2019 at 00:47
Quote Posted by goldwell
Took just under 2 hours to process.
Terrible waste of electricity ;P
Aspect ratio has been broken, details has been broken, structure-texture has been broken, pattern gradients has been broken, so... the movies has been broken at all :P :)
(
https://www.dropbox.com/s/c46evvuiqr3q5rc/Image4.png?dl=0)
Not long ago Akven has done it much better (all T1 & T2 movies). The only good things in Machine Learning version are initial subtitles - the aggressive filter worked better than in Akven's ones.
N00bThief on 2/6/2019 at 02:36
Well this is a good start!
Like Mat99 said the ratio is broken and sometimes when something is rotating or getting closer (like the ring scene) it looks kinda odd (Probably this is because of the original 4:3 ratio?).
Keep up with your work i'm curious too see how this improves and especially how it will handle the more "animated" Movies.
If you improve this a little bit this could be really great!
Akven on 2/6/2019 at 05:19
My attempt on machine learning with ESRGAN for Intro from Thief Gold
1080p@30fps. Was based upon my video refiltering mod, not on vanilla 320x240@15fps videos
[video=youtube;-BgskHuTECI]https://www.youtube.com/watch?v=-BgskHuTECI[/video]
Something like this was used for the initial filtering and frame interpolation. I'm not sure it is the exact script, but it is very close.
#avstp.dll
#RemovegrainSSE2.dll
#RepairSSE2.dll
#mt_masktools-26.dll
#nnedi3.dll
#SmoothAdjust.dll
#FluxSmooth.dll
#mvtools2mod.dll
#AddGrainC.dll
#GradFun2DB.dll
#sbr.avs
#MinMapBlur.avs
#GradFun2DBmod.avs
#nnedi3.dll
#splineresize.dll
#svpflow2.dll
#svpflow1.dll
#AvsFilterNet.dll
#InterFrame2.avsi
#flash3kyuu_deband.dll
#maa2.avsi
#fturn-26.dll
#SangNom2.dll
# setmemorymax(768)
RemoveGrain(0) # No-Op filter, just to do frame request
ChangeFPS(last,last,true) # initiate a small forward buffer
/*
o = last
bob1 = o.nnedi3(field=-2)
bob2 = o.sharpen(0,1).nnedi3(field=-2)
max = bob1.mt_logic(bob1.selectevery(1,-1),"max",U=3,V=3).mt_logic(bob1.selectevery(1,1),"max",U=3,V=3)
min = bob1.mt_logic(bob1.selectevery(1,-1),"min",U=3,V=3).mt_logic(bob1.selectevery(1,1),"min",U=3,V=3)
clamped = bob2.mt_clamp(max,min,0,0,U=3,V=3)
clamped.selectevery(2,-1).trim(1,0)
*/
source = last
x1 = source.fluxsmootht(3)
x2 = source.removegrain(11,-1)
x22 = source.mt_makediff(mt_makediff(x2,x2.removegrain(20,-1))).MinMapBlur()
enhD = mt_lutxy(x22,x22.removegrain(4,-1).sbr(),"128 x y - abs 2 / 1 1.6 / ^ 2.51 * x y - x y - abs 0.1 + / * +",U=2,V=2)
enh = source.mt_adddiff(enhD,U=2,V=2)
tr = 3
dct = 0
thSAD = 320
chroma = true
planes = chroma ? 4 : 0
sup1 = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, chroma=chroma)
sup2 = enh.MSuper(hpad=8, vpad=8, pel=4, levels=1, sharp=1, chroma=chroma)
rsup = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, levels=1, chroma=chroma)
vmulti = sup1.MAnalyse(multi=True, overlap=4, blksize=8, search=4, truemotion=True, delta=tr, trymany=True, searchparam=8, pelsearch=4, dct=dct, levels=0, divide=2, badrange=-24)
vmulti = rsup.MRecalculate(vmulti, overlap= 2, blksize= 4, thsad=thsad/2, truemotion=True, tr=tr, search=4, searchparam=8, dct=dct, smooth=1, divide=0)
MDegrainN (sup2, vmulti, tr, thSAD=thSAD, thSCD1=350, thSCD2=104, thSAD2=150, limit=135, plane=planes)
nnedi3_rpow2(rfactor=2,cshift="spline144resize",fwidth=640,fheight=480)
maa2() # подключить по необходимости
InterFrame (GPU=false, Preset="Medium", Tuning ="Smooth", OverrideAlgo=23, InputType="2D", Cores=1, FrameDouble=true)
AssumeFps(30)
GradFun2DBmod(thr=1.6,thrC=2.0,mode=2,str=1.0,strC=0.0,temp=50,adapt=64)
# SmoothLevels(0,1.0,255,0,255, useopt=0, HQ=true, useMT=1) # 0 -automatic choice, 1- SSE2 CPU, 2 - Sandy Bridge CPUAnd I used some filter after the ESRGAN upscale for noise.
This is link to 1080p mkv file without Youtube compression.
(
https://mega.nz/#!69VmACTb!OcFuYJBZ6csvwv5itb0vJrHLZsk5HWb7go2Wb5V3AKY)
P.S. If you want to do AI upscale for movies, I suggest you do some video pre-filtering before upscaling. Via Avisynth or some jpeg artefacts remover ESRGAN model. Or using my movies mod as a base. In your video a lot of aliasing glitches on lines. And jaggy framerate. There was some Neural learning project for fps converting, but I can't say without google if they were released to public.
goldwell on 2/6/2019 at 08:32
Quote Posted by Mat99
Terrible waste of electricity ;P
Aspect ratio has been broken, details has been broken, structure-texture has been broken, pattern gradients has been broken, so... the movies has been broken at all :P :)
(
https://www.dropbox.com/s/c46evvuiqr3q5rc/Image4.png?dl=0)
Not long ago Akven has done it much better (all T1 & T2 movies). The only good things in Machine Learning version are initial subtitles - the aggressive filter worked better than in Akven's ones.
I'm fortunate enough to live in a country where electricity isn't a concern ;P
The aspect ratio was purposefully changed from 4:3 to 16:9. I watched it again and it doesn't look like a problem to me. Things might be a little wider stretched in some aspects but otherwise looks good to me.
The only area that looks really off to me is the scene with the keepers ring, but I think that's something else going on there rather than aspect ratio.
Quote Posted by N00bThief
Well this is a good start!
Like Mat99 said the ratio is broken and sometimes when something is rotating or getting closer (like the ring scene) it looks kinda odd (Probably this is because of the original 4:3 ratio?).
Keep up with your work i'm curious too see how this improves and especially how it will handle the more "animated" Movies.
If you improve this a little bit this could be really great!
This was a one off deal, I don't plan on doing anymore at this time. I looked around for any other Thief cinematics that had been upscaled but couldn't find any (until seeing Akven's response below). Seems like there are a few different methods to doing it.
The ratio isn't causing the weird scene issue in the keeper ring scene, it almost looks wavy to me. Not sure what that is.
Quote Posted by Akven
My attempt on machine learning with ESRGAN for Intro from Thief Gold
1080p@30fps. Was based upon my video refiltering mod, not on vanilla 320x240@15fps videos
[video=youtube;-BgskHuTECI]https://www.youtube.com/watch?v=-BgskHuTECI[/video]
Something like this was used for the initial filtering and frame interpolation. I'm not sure it is the exact script, but it is very close.
#avstp.dll
#RemovegrainSSE2.dll
#RepairSSE2.dll
#mt_masktools-26.dll
#nnedi3.dll
#SmoothAdjust.dll
#FluxSmooth.dll
#mvtools2mod.dll
#AddGrainC.dll
#GradFun2DB.dll
#sbr.avs
#MinMapBlur.avs
#GradFun2DBmod.avs
#nnedi3.dll
#splineresize.dll
#svpflow2.dll
#svpflow1.dll
#AvsFilterNet.dll
#InterFrame2.avsi
#flash3kyuu_deband.dll
#maa2.avsi
#fturn-26.dll
#SangNom2.dll
# setmemorymax(768)
RemoveGrain(0) # No-Op filter, just to do frame request
ChangeFPS(last,last,true) # initiate a small forward buffer
/*
o = last
bob1 = o.nnedi3(field=-2)
bob2 = o.sharpen(0,1).nnedi3(field=-2)
max = bob1.mt_logic(bob1.selectevery(1,-1),"max",U=3,V=3).mt_logic(bob1.selectevery(1,1),"max",U=3,V=3)
min = bob1.mt_logic(bob1.selectevery(1,-1),"min",U=3,V=3).mt_logic(bob1.selectevery(1,1),"min",U=3,V=3)
clamped = bob2.mt_clamp(max,min,0,0,U=3,V=3)
clamped.selectevery(2,-1).trim(1,0)
*/
source = last
x1 = source.fluxsmootht(3)
x2 = source.removegrain(11,-1)
x22 = source.mt_makediff(mt_makediff(x2,x2.removegrain(20,-1))).MinMapBlur()
enhD = mt_lutxy(x22,x22.removegrain(4,-1).sbr(),"128 x y - abs 2 / 1 1.6 / ^ 2.51 * x y - x y - abs 0.1 + / * +",U=2,V=2)
enh = source.mt_adddiff(enhD,U=2,V=2)
tr = 3
dct = 0
thSAD = 320
chroma = true
planes = chroma ? 4 : 0
sup1 = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, chroma=chroma)
sup2 = enh.MSuper(hpad=8, vpad=8, pel=4, levels=1, sharp=1, chroma=chroma)
rsup = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, levels=1, chroma=chroma)
vmulti = sup1.MAnalyse(multi=True, overlap=4, blksize=8, search=4, truemotion=True, delta=tr, trymany=True, searchparam=8, pelsearch=4, dct=dct, levels=0, divide=2, badrange=-24)
vmulti = rsup.MRecalculate(vmulti, overlap= 2, blksize= 4, thsad=thsad/2, truemotion=True, tr=tr, search=4, searchparam=8, dct=dct, smooth=1, divide=0)
MDegrainN (sup2, vmulti, tr, thSAD=thSAD, thSCD1=350, thSCD2=104, thSAD2=150, limit=135, plane=planes)
nnedi3_rpow2(rfactor=2,cshift="spline144resize",fwidth=640,fheight=480)
maa2() # подключить по необходимости
InterFrame (GPU=false, Preset="Medium", Tuning ="Smooth", OverrideAlgo=23, InputType="2D", Cores=1, FrameDouble=true)
AssumeFps(30)
GradFun2DBmod(thr=1.6,thrC=2.0,mode=2,str=1.0,strC=0.0,temp=50,adapt=64)
# SmoothLevels(0,1.0,255,0,255, useopt=0, HQ=true, useMT=1) # 0 -automatic choice, 1- SSE2 CPU, 2 - Sandy Bridge CPUAnd I used some filter after the ESRGAN upscale for noise.
This is link to 1080p mkv file without Youtube compression.
(
https://mega.nz/#!69VmACTb!OcFuYJBZ6csvwv5itb0vJrHLZsk5HWb7go2Wb5V3AKY)
P.S. If you want to do AI upscale for movies, I suggest you do some video pre-filtering before upscaling. Via Avisynth or some jpeg artefacts remover ESRGAN model. Or using my movies mod as a base. In your video a lot of aliasing glitches on lines. And jaggy framerate. There was some Neural learning project for fps converting, but I can't say without google if they were released to public.
That looks good! I only planned on doing this as a one off test, just to see how it would look. Considering it took a couple of hours just to produce this, I don't feel like investing much more time beyond that. Seems like there are quite a few different methods to it, and obviously effects you can add after the fact to improve it.
All in all it was an interesting experiment, i'm quite happy with the results too. I get it could be tweaked further but i'll leave that to others to handle.
N00bThief on 2/6/2019 at 13:23
Quote Posted by Akven
My attempt on machine learning with ESRGAN for Intro from Thief Gold
1080p@30fps. Was based upon my video refiltering mod, not on vanilla 320x240@15fps videos
[video=youtube;-BgskHuTECI]https://www.youtube.com/watch?v=-BgskHuTECI[/video]
Something like this was used for the initial filtering and frame interpolation. I'm not sure it is the exact script, but it is very close.
#avstp.dll
#RemovegrainSSE2.dll
#RepairSSE2.dll
#mt_masktools-26.dll
#nnedi3.dll
#SmoothAdjust.dll
#FluxSmooth.dll
#mvtools2mod.dll
#AddGrainC.dll
#GradFun2DB.dll
#sbr.avs
#MinMapBlur.avs
#GradFun2DBmod.avs
#nnedi3.dll
#splineresize.dll
#svpflow2.dll
#svpflow1.dll
#AvsFilterNet.dll
#InterFrame2.avsi
#flash3kyuu_deband.dll
#maa2.avsi
#fturn-26.dll
#SangNom2.dll
# setmemorymax(768)
RemoveGrain(0) # No-Op filter, just to do frame request
ChangeFPS(last,last,true) # initiate a small forward buffer
/*
o = last
bob1 = o.nnedi3(field=-2)
bob2 = o.sharpen(0,1).nnedi3(field=-2)
max = bob1.mt_logic(bob1.selectevery(1,-1),"max",U=3,V=3).mt_logic(bob1.selectevery(1,1),"max",U=3,V=3)
min = bob1.mt_logic(bob1.selectevery(1,-1),"min",U=3,V=3).mt_logic(bob1.selectevery(1,1),"min",U=3,V=3)
clamped = bob2.mt_clamp(max,min,0,0,U=3,V=3)
clamped.selectevery(2,-1).trim(1,0)
*/
source = last
x1 = source.fluxsmootht(3)
x2 = source.removegrain(11,-1)
x22 = source.mt_makediff(mt_makediff(x2,x2.removegrain(20,-1))).MinMapBlur()
enhD = mt_lutxy(x22,x22.removegrain(4,-1).sbr(),"128 x y - abs 2 / 1 1.6 / ^ 2.51 * x y - x y - abs 0.1 + / * +",U=2,V=2)
enh = source.mt_adddiff(enhD,U=2,V=2)
tr = 3
dct = 0
thSAD = 320
chroma = true
planes = chroma ? 4 : 0
sup1 = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, chroma=chroma)
sup2 = enh.MSuper(hpad=8, vpad=8, pel=4, levels=1, sharp=1, chroma=chroma)
rsup = x1.removegrain(11).MSuper(hpad=8, vpad=8, pel=4, sharp=0, levels=1, chroma=chroma)
vmulti = sup1.MAnalyse(multi=True, overlap=4, blksize=8, search=4, truemotion=True, delta=tr, trymany=True, searchparam=8, pelsearch=4, dct=dct, levels=0, divide=2, badrange=-24)
vmulti = rsup.MRecalculate(vmulti, overlap= 2, blksize= 4, thsad=thsad/2, truemotion=True, tr=tr, search=4, searchparam=8, dct=dct, smooth=1, divide=0)
MDegrainN (sup2, vmulti, tr, thSAD=thSAD, thSCD1=350, thSCD2=104, thSAD2=150, limit=135, plane=planes)
nnedi3_rpow2(rfactor=2,cshift="spline144resize",fwidth=640,fheight=480)
maa2() # подключить по необходимости
InterFrame (GPU=false, Preset="Medium", Tuning ="Smooth", OverrideAlgo=23, InputType="2D", Cores=1, FrameDouble=true)
AssumeFps(30)
GradFun2DBmod(thr=1.6,thrC=2.0,mode=2,str=1.0,strC=0.0,temp=50,adapt=64)
# SmoothLevels(0,1.0,255,0,255, useopt=0, HQ=true, useMT=1) # 0 -automatic choice, 1- SSE2 CPU, 2 - Sandy Bridge CPUAnd I used some filter after the ESRGAN upscale for noise.
This is link to 1080p mkv file without Youtube compression.
(
https://mega.nz/#!69VmACTb!OcFuYJBZ6csvwv5itb0vJrHLZsk5HWb7go2Wb5V3AKY)
P.S. If you want to do AI upscale for movies, I suggest you do some video pre-filtering before upscaling. Via Avisynth or some jpeg artefacts remover ESRGAN model. Or using my movies mod as a base. In your video a lot of aliasing glitches on lines. And jaggy framerate. There was some Neural learning project for fps converting, but I can't say without google if they were released to public.
Well, this looks really good!
Any plans on doing all the Movies, so we can use them in the game?
Mat99 on 2/6/2019 at 14:15
Quote Posted by goldwell
I'm fortunate enough to live in a country where electricity isn't a concern ;P
Me too ;P but it's not the point. The resources had been wasted but could have been used for better purpose ;P
i.e.: two hours more to polish and refine Your new ACT3 (or other AS ALWAYS fabulous) mission ;P
Quote Posted by goldwell
The aspect ratio was purposefully changed from 4:3 to 16:9.
Such thing as Aspect Ratio should NEVER be changed purposefully, unpurposefully, consciously, unconsciously or whatever/otherwise... NEVER :)
At least 16:10 (means 1728x1080) could be less painful than 16:9, but 4:3 is really what should stay.
Quote Posted by goldwell
I watched it again and it doesn't look like a problem to me. Things might be a little wider stretched in some aspects but otherwise looks good to me.
You have very strange definition of what is "looking good" ;P ;) The "inflatable (pumped) screen" never looks good and always will be looking unnatural (by proportions). Moreover - proportions which change progressively during the rotating screen-move.
Vertical things look too stretched and during rotation they shrink progressively in realtime to become too thin in horizontal position :) This is not looking good at all :) (i.e.: the buildings in 0:35-0:40)
Quote Posted by goldwell
The ratio isn't causing the weird scene issue in the keeper ring scene, it almost looks wavy to me. Not sure what that is.
That problem is in original movie already, but You BOOSTED it by adding X-axis-pixels that shouldn't even exist :) And we have horizontal scene-move there ([X-axis] from right to left). Anti-aliased pixels (i.e.: keyhole in Keeper's ring) were treated by the algorithm as edge data. Contrast on them has been increased - so what should stay as artifact (blury edge) become new, NON-existent earlier
current edge and other filters are feed on them :) Additionally every ONE-X-pixel is treated now (virtually) as NON-INTEGER 1,3-pixel (additional 480px in x-axis). After calculate it and averaging values on neighboring pixels boost/strengthening of the shitty wavy effect appeared immediately :)
Quote Posted by goldwell
This was a one off deal, I don't plan on doing anymore
Thank to the Builder ;))
Quote Posted by goldwell
I looked around for any other Thief cinematics that had been upscaled but couldn't find any
Quote Posted by N00bThief
Any plans on doing all the Movies, so we can use them in the game?
Here they are - Akven's versions from his thread(s):
Thief Gold Refiltered Movies
(
https://mega.nz/#!j0dAkQAT!vS5gKjD4W...UjvSwz00dJAx_A)
Thief 2 Refiltered Movies
(
https://mega.nz/#!PxFkxCzK!F5zNDI-fo...UUpHr37a3hs3NE)
Akven on 2/6/2019 at 15:20
Quote Posted by N00bThief
Well, this looks really good!
Any plans on doing all the Movies, so we can use them in the game?
Unfortunately, you can't use HD movies ingame. NewDark currently only supports 640x480 resulution for video and interface.
And 640x480@30fps refiltered movies already done (see previous comment).
I hope I'll do proper HD (1080p) version of all the movies in time. Right now I'm completing ESRGAN Texture pack.
marbleman on 2/6/2019 at 18:55
Quote Posted by Mat99
Me too ;P but it's not the point. The resources had been wasted but could have been used for better purpose ;P
i.e.: two hours more to polish and refine Your new ACT3 (or other AS ALWAYS fabulous) mission ;P
You could just let people decide on what to do with their time ;P Just a thought ;PP