原文连接:https://bartwronski.com/2014/03/15/temporal-supersampling-and-antialiasing/html
Before I address temporal supersampling, just a quick reminder on what aliasing is.程序员
在我讲述时间超级采样以前,对锯齿是什么作一个快速的提示。算法
Aliasing is a problem that is very well defined in signal theory. According to the general sampling theorem we need to have our signal spectrum containing only frequencies lower than Nyquist frequency. If we don’t (and when rasterizing triangles we always will as triangle edge is infinite frequency spectrum, step-like response) we will have some frequencies appearing in the final signal (reconstructed from samples) that were not in the original signal. Visual aliasing can have different appearance, it can appear as regular patterns (so-called moire), noise or flickering.缓存
锯齿是一个在信号理论中意义明确的问题。根据广泛的采样原理,咱们须要让咱们的信号频谱只包含比Nyquist频率低的频率。若是咱们不这么作(而且当光栅化三角形时咱们老是会这样由于三角形边是无限频率的频谱,梯状的反应)咱们将有一些频率表如今最终的信号(采样重构)不是原始的信号。可见的锯齿会有不一样的表现,他可能以规律的模式(称为云纹),噪波或闪烁表现出来。sass
Classic supersampling is a technique that is extremely widely used by the CGI industry. Per every target image fragment we perform sampling multiple times at much higher frequencies (for example by tracing multiple rays per simply pixel or shading fragments multiple times at various positions that cover the same on-screen pixel) and then performing the signal downsampling/filtering – for example by averaging. There are various approaches to even easiest supersampling (I talked about this in one of my previous blog posts), but the main problem with it is the associated cost – N times supersampling means usually N times the basic shading cost (at least for some pipeline stages) and sometimes additionally N times the basic memory cost. Even simple, hardware-accelerated techniques like MSAA that do estimate only some parts of the pipeline (pixel coverage) in higher frequency and don’t provide as good results, have quite big cost on consoles.session
经典的超级采样是一个在CGI行业普遍使用的技术。对每个目标图像像素咱们用更高的频率执行屡次采样(例如,在每一个简单像素跟踪多个射线或在覆盖相同屏幕像素的不一样位置上着色屡次片断)而且以后执行信号下采样/过滤——例如经过均值。有不一样的方式去实现甚至最先还有超级采样(我在我以前的博客中有讨论过)。可是最主要的问题是它相关的性能消耗,N次的超级采样意味着一般N倍的基本着色消耗(至少对于一些渲染管线阶段)。而且一些额外的N倍的基本内存消耗。甚至简单的,硬件加速技术像MSAA只在部分管线阶段用更高的频率,而且提供不了那么好的效果,在游戏机有至关高的消耗。app
But even if supersampling is often unpractical technique, it’s temporal variation can be applied with almost zero cost.less
即便超级采样是一个难以应用的技术,可是他的时间变体几乎0消耗应用。ide
So what is the temporal supersampling? Temporal supersampling techniques base on a simple observation – from frame to frame most of the on-screen screen content do not change. Even with complex animations we see that multiple fragments just change their position, but apart from this they usually correspond to at least some other fragments in previous and future frames.wordpress
因此时间超级采样是什么?时间超级采样技术基于一个简单的观察——帧与帧之间屏幕上的屏幕内容大多数都没有改变。甚至一些有复杂的动画咱们看到多个像素只是改变了他们的位置,可是除此以外,他们一般对应至少一些其余的片断在以前和将来的帧中。
Based on this observation, if we know the precise texel position in previous frame (and we often do! Using motion vectors that are used for per-object motion blur for instance), we can distribute the multiple fragment evaluation component of supersampling between multiple frames.
基于这个观察,若是咱们知道以前帧精确的像素位置(而且咱们常常这么作!使用运动向量用于逐对象的运动模糊实例),咱们能够分布超级采样的多个像素求值份量到多个帧之间。
What is even more exciting is that this technique can be applied to any pass – to your final image, to AO, screen-space reflections and others – to either filter the signal or increase the number of samples taken. I will first describe how it can be used to supersample final image and achieve much better AA and then example of using it to double or triple number of samples and quality of effects like SSAO.
更让人惊喜的是这个技术能够应用到任何Pass中——最终的图像,AO,SSR以及其余——甚至过滤信号或增长采样的数量。我将第一次描述它若是用与超级采样最终的图像而且实现更佳的AA,而且以后使用到双倍或三倍数量的采样和效果质量像SSAO的例子
I have no idea which game was the first to use the temporal supersampling AA, but Tiago Sousa from Crytek had a great presentation on Siggraph 2011 on that topic and its usage in Crysis 2 [1]. Crytek proposed using a sub pixel jitter to the final MVP transformation matrix that alternates every frame – and combine two frames in post-effect style pass. This way they were able to increase the sampling resolution twice at almost no cost!
我不清楚哪一款游戏是第一个使用时间超级采样抗锯齿的,可是Crytek的Tiago Sousa在Siggraph2011有一个很棒的主题展现也包含了它在Crysis2的使用。Cryteck推荐每一帧交替的对最终的MVP变换矩阵使用一个次像素抖动——而且在后处理风格Pass中将两帧合并。这个方式能够在几乎0消耗的状况下提高两倍的采样分辨率。
Too good to be true?
好的让人难以相信?
Yes, the result of such simple implementation looks perfect on still screenshots (and you can implement it in just couple hours!***), but breaks in motion. Previous frame pixels that correspond to current frame were in different positions. This one can be easily fixed by using motion vectors, but sometimes the information you are looking for was occluded or had. To address that, you cannot rely on depth (as the whole point of this technique is having extra coverage and edge information from the samples missing in current frame!), so Crytek proposed relying on comparison of motion vector magnitudes to reject mismatching pixels.
是的,这样简单的实现结果在静止的截屏上看起来完美(而且你能够实现它在短短的几个小时——我不能够),可是在运动的时候会露馅。以前的帧像素对应的当前帧像素在不一样的位置。这个问题可使用motion vectors简单的修复。可是有的时候,你查找的信息被阻挡的。为了处理这个问题,你不能依靠深度(由于这个技术的总体要点是有额外的覆盖和来自当前帧的采样丢失边缘信息!),因此Crytek 推荐依靠motion vector放大来对比拒毫不匹配的像素。
***yeah, I really mean maximum one working day if you have a 3D developer friendly engine. Multiply your MVP matrix with a simple translation matrix that jitters in (-0.5 / w, -0.5 / h) and (0.5 / w, 0.5 / h) every other frame plus write a separate pass that combines frame(n) and frame(n-1) together and outputs the result.
***是的,若是你有一个3D开发友好的引擎,个人意思是最多一个工做日。将你的MVP矩阵乘一个简单的(-0.5 / w, -0.5 / h) 和(0.5 / w, 0.5 / h) 抖动过渡矩阵每隔一帧+写一个单独的pass将第n帧和第n-1帧混合在一块儿而且输出结果。
For a long time we relied on FXAA (aided by depth-based edge detection) as a simple AA technique during our game development. This simple technique usually works “ok” with static image and improves its quality, but breaks in motion – as edge estimations and blurring factors change from frame to frame. While our motion blur (simple and efficient implementation that used actual motion vectors for every skinned and moving objects) helped to smooth edge look for objects moving quite fast (small motion vector dilation helped even more), it didn’t do anything with calm animations and subpixel detail. And our game was full of them – just look at all the ropes tied to sails, nicely tessellated wooden planks and dense foliage in jungles!
很长的一段时间,在咱们的游戏开发期间咱们依靠FXAA(受助于经过基于深度的边缘检测)做为简单的抗锯齿技术。这个简单的技术在一些静帧图像和提高它的质量上一般工做的还能够,可是运动的时候会出错——由于边缘判断和模糊因素逐帧改变。然而咱们的运动模糊(用于每一个蒙皮且运动的对象使用真实的运动向量的简单和高效的实现)有助于为运动的至关快的对象的平滑边缘(微小的运动向量更有帮助)。它对任何平静动画和次像素细节的没有做用。而且咱们的游戏充满了它们——只须要看看全部的系着帆的绳索,在丛林中完美嵌合的木板和茂密的植物。
Unfortunately motion blur did nothing to help the antialiasing of such slowly moving objects and FXAA added some nasty noise during movement, especially on grass. We didn’t really have time to try so-called “wire AA” and MSAA was out of our budgets so we decided to try using temporal antialiasing techniques.
不幸的是运动模糊对运动缓慢的对象的抗锯齿没有帮助,而且FXAA在运动的时候增长了一些使人不爽的噪声,尤为在草地上。咱们真的没有时间去尝试所谓的“wire抗锯齿”,而且MSAA超出了咱们的性能预算,因此咱们决定尝试使用时间抗锯齿技术。
I would like to thank here especially Benjamin Goldstein, our Technical Lead with whom I had a great pleasure to work on trying and prototyping various temporal AA techniques very late in the production.
这里我要感谢Benjamin Goldstein,咱们的技术Lead,我很愉快在开发后期尝试研究不一样时间抗锯齿技术。
As a first iteration, we started with single-frame variation of morphological SMAA by Jimenez et al. [2] In its even most basic settings it showed definitely better-quality alternative to FXAA (at a bit higher cost, but thanks to much bigger computing power of next-gen consoles it stayed in almost same budget compared to FXAA on current-gen consoles). There was less noise and artifacts and much better morphological edge reconstruction , but obviously it wasn’t able do anything to reconstruct all this subpixel detail.
做为第一次迭代,咱们以Jimenez et al的形态学SMAA 单帧变体开始。在它的甚至大多数的基础设置上清楚的展现了比起FXAA更佳的质量的选择(略高一点的性能消耗,可是得益于次世代游戏机更强的计算能力,比起本世代游戏机的FXAA来讲基本上是同样的预算消耗)。他有更少的噪声和瑕疵和更佳的形态学边缘重构,可是显然他不可以重建全部的次像素细节。
So the next step was to try to plug in temporal AA component. Couple hours of work and voila – we had much better AA. Just look at the following pictures.
因此下一步是尝试插入时间抗锯齿部分。数小时的工做后,瞧——咱们有个更佳的抗锯齿表现。就像下面图片展现的那样。
Pretty amazing, huh?
确实使人惊讶?
Sure, but this was at first the result only for static image – and this is where your AA problems start (not end!).
不过这只是第一步静帧图像的结果——而且这是你的抗锯齿问题的开始。
Ok, so we had some subtle and we thought “precise” motion blur, so getting motion vectors to allow proper reprojection for moving objects should be easy?
好的,因此咱们有一些精巧的而且咱们认为“精确”运动模糊,那么对于运动物体获得运动向量去容许正确的重映射应该很容易?
Well, it wasn’t. We were doing it right for most of the objects and motion blur was ok – you can’t really notice lack of motion blur or slightly wrong motion blur on some specific objects. However for temporal AA you need to have them proper and pixel-perfect for all of your objects!
然而并非这样。咱们对大多数的对象这样作是正确的而且运动模糊是能够接受的——在一些特定的对象你几乎不会注意到运动模糊缺失或轻微的运动模糊错误。然而对于时间抗锯齿,你的全部对象你都须要它们有正确且像素级别的完美。
Other way you will get huge ghosting. If you try to mask out this objects and not apply temporal AA on them at all, you will get visible jittering and shaking from sub-pixel camera position changes.
否则的话,你会获得巨大的重影。若是你试着去遮掩掉这个对象,并不该用时间抗锯齿在上面,你会获得来自次像素摄像机位置改变的明显的抖动和震动。
Let me list all the problems with motion vectors we have faced and some comments of whether we solved them or not:
让我列一下咱们面对过的运动向量的全部问题和一些咱们是否解决他们的体会:
Ok, we spend 1-2 weeks on fixing our motion vectors (and motion blur also got much better!😊 ), but in the meanwhile realized that the approach proposed by Crytek and used in SMAA for motion rejection is definitely far from perfect. I would divide problems into two categories.
咱们花费了1-2周在修复咱们的运动向量(同时咱们的运动模糊效果也更好了!😊),可是同时意识到Crytek推荐的方法而且在SMAA中使用的运动拒绝确定离完美还远得很。我把问题分红了两类。
It was something we didn’t really expect, but temporal AA can break if menu pops up quickly, you pause the game, you exit to console dashboard (but game remains visible), camera teleports or some post-effect immediately kicks in. You will see some weird transition frame. We had to address each case separately – by disabling the jitter and frame combination on such frame. Add another week or two to your original plan of enabling temporal AA to find, test and fix all such issues…
这是咱们没有预料到的事情,若是菜单弹出太快可能时间抗锯齿会出错,你暂停游戏,退出到控制台仪表盘(可是游戏保持可见),相机瞬移或一些后处理特效马上关闭。你会看到一些奇怪的过渡帧。咱们必须去单独处理每一种状况——经过禁用抖动和在这样的帧进行帧合并。花费额外的一个或两个星期在你本来的启用时间抗锯齿计划去查找,测试和修复这些全部的问题。
This is my actual biggest problem with naive SMAA-like way of rejecting blending by comparing movement of objects.
经过对比对象的运动,拒绝混合的幼稚的相似SMAA的方式,是我真正的最大的问题。
First of all, we a had very hard time to adjust the “magic value” for the rejection threshold and 8-bit motion vectors didn’t help it. Objects were either ghosting or shaking.
首先,对于拒绝阈值,咱们有段艰难的时间来调整这个“魔幻值”,并且8位的移动向量对此也没有什么帮助。对象依然重影或抖动。
Secondly, there were huge problems on for example ground and shadows – the shadow itself was ghosting – well, there is no motion vector for shadow or any other animated texture, right? 😊 It was the same with explosions, particles, slowly falling leaves (that we simulated as particle systems).
第二,例如地面和阴影有巨大的问题——阴影自己就是重影——那好,对于阴影或者任何运动的纹理没有移动向量,对吧?😊爆炸,粒子,缓慢下落的树叶(咱们用粒子系统模拟)也是同样的方式。
For both of those issues, we came up with simple workaround – we were not only comparing similarity of motion of objects, but on top of it added a threshold value – if object moved faster than around ~2 pixels per frame in current or previous frame, do not blend them at all! We found such value much easier to tweak and to work with. It solved the issue of shadows and visible ghosting.
对于这些全部的问题,咱们想出了简单的变通方案——咱们不只仅对比了对象的运动类似性,可是在这之上还添加了一个阈值——若是物体在当前或以前的帧中移动超过约2个像素,不要对他们进行混合!咱们发现这样的值更容易调整和运用。解决了阴影和明显的重影问题。
We also increased motion blur to reduce any potential visible shaking.
咱们也增长了运动模糊来减小可能存在的明显的震动。
Unfortunately, it didn’t do anything for transparent or animated texture changes over time, they were blended and over-blurred – but as a cool side effect we got free rain drops and rain ripples antialiasing and our art director preferred such soft, “dreamy” result.
不幸地是,这对于透明或随着时间改变的动画纹理没有做用,他们被混合且过分模糊——可是做为一个很酷的反作用,咱们获得了免费的雨滴和雨涟漪的抗锯齿,重要的是咱们的艺术指导更喜欢这样柔软,“梦幻”的结果。
Recently Tiago Souse in his Siggraph 2013 talk proposed to address this issue by changing metric to color-based and we will investigate it in the near future [3].
最近Tiago Souse在他的siggraph2013演讲中提出了经过改变基于颜色的度量标准来解决这个问题,咱们在不久的未来研究它。
I wanted to mention another use of temporal supersampling that got into final game on the next-gen consoles and that I really liked. I got inspired by Matt Swoboda’s presentation [4] and mention of distributing AO calculation sampling patterns between multiple frames. For our SSAO we were having 3 different sampling patterns (spiral-based) that changed (rotated) every frame and we combined them just before blurring the SSAO results. This way we effectively increased number of samples 3 times, needed less blur and got much much better AO quality and performance for cost of storing just two additional history textures.Unfortunately I do not have screenshots to prove that and you have to take my word for it, but I will try to update my post later.
我想要谈谈时间超级采样在次世代的游戏机中已经进入了决胜局的其余应用,而且我真的的喜欢。我受Matt Swoboda的演讲启发,而且说起在多帧之间分布AO计算采样模式。对于咱们的SSAO,咱们有三种不一样的每帧改变(旋转)的采样模式(基于螺旋的),而且咱们在模糊SSAO的结果前就将它们组合在一块儿。这个方法有效的提升了三倍的采样数量,须要更少的模糊且获得好不少的AO质量,并且只有两张额外的历史纹理的存储性能消耗。不幸地是,我没有截图来证实它,你必须相信个人话,但我以后会更新个人文章。
For rejection technique I was relying on a simple depth comparison – we do not really care about SSAO on geometric foreground object edges and depth discontinuities as by AO definition, there should be almost none. Only visible problem was when SSAO caster moved very fast along static SSAO receiver – there was visible trail lagging in time – but this situation was more artificial problem I have investigated, not a serious in-game problem/situation. Unlike the temporal antialiasing, putting this in game (after having proper motion vectors) and testing took under a day, there were no real problems, so I really recommend using such techniques – for SSAO, screen-space reflections and many more.
对于拒绝技术,我依靠简单的深度对比——正如AO的定义同样,咱们真的不在乎几何前景边缘和深度间断处的SSAO,那里应该几乎没有AO。只有可见的问题是,当投射SSAO的对象沿着静态的SSAO接收对象移动的很是快的时候——那里有明显的时间滞后的痕迹——可是这种状况是我研究过的更虚假的问题,不是一个严重的游戏中问题/情形。SSAO不像时间抗锯齿,我把它加入游戏中(在有正确的移动向量后)而且只用了一天测试,没有什么真正的问题,因此我强烈建议使用这个技术——对于SSAO,SSR还有更多的应用。
Temporal supersampling is a great technique that will increase final look and feel of your game a lot, but don’t expect that you can do it in just couple days. Don’t wait till the end of the project, “because it is only a post-effect, should be simple to add” – it is not! Take weeks or even months to put it in, have testers report all the problematic cases and then properly and iteratively fix all the issues. Have proper and optimal motion vectors, think how to write them for artist-authored materials, how to batch your objects in passes to avoid using extra MRT if you don’t need to write them (static objects and camera-only motion vector). Look at differences in quality between 16bit and 8bit motion vectors (or maybe R11G11B10 format and some other G-Buffer property in B channel?), test all the cases and simply take your time to do it all properly and early in production, while for example changing a bit skeleton calculation or caching vertex skinning information (having “vertex history”) is still an acceptable option.
时间超级采样是一个提高你的游戏最终的表现和感觉的绝佳的技术,可是不要妄想你能够在短短的几天实现它。不要等到项目的后期,“由于它只是一个后处理,应该很容易添加”——并非!须要花费数周乃至数月来投入开发,让测试人员报告全部问题状况,而后正确地,迭代地修复全部的问题。拥有正确和最优化的移动向量,考虑如何为艺术家制做的材质写入它们,若是你不须要写入它们(静态对象和仅摄像机的移动向量),如何在多个Pass中分批处理你的对象以免使用额外的MRT。观察16位和8位移动向量质量上的差别(或者多是R11G11B10 格式和B通道中一些其余的G-Buffer属性?),测试全部的状况,花点时间在开发上作得很好,例如当改变一点骨架计算或者缓存顶点蒙皮信息(存在“顶点历史”)的时候依然是一个可接受的选择。
[1] http://iryoku.com/aacourse/
[2] http://www.iryoku.com/smaa/
[3] http://advances.realtimerendering.com/s2013/index.html
[4] http://directtovideo.wordpress.com/2012/03/15/get-my-slides-from-gdc2012/